That didn't seem to be about first-class macros, except for a very narrow special case (granted, it's about modules, which you seem most interested in). If you think about it, macro-expanding the car of a list won't really first-class anything -- you'd still need to do
because macro-expanding the car above doesn't get rid of the macro literals push and pull, which will cause an error on application. vincenz tried to make ac recognize macro literals, but it still amounted to evaling (at least of macros), which he didn't get working. http://arclanguage.org/item?id=804
The most obvious way to handle this is to straight-out interpret. As the most obvious way, I'd guess that's why first-class macros have the reputation of being difficult to compile, since there will be some cases that need run-time expansion.
Yes, that was the discussion I was talking about, and I suppose you're correct; it wouldn't really help in most cases.
"As I said, to the best of my knowledge it's still an unsolved problem."
Then let's see if we can't solve it ;)
Personally, I think that the advantages in terseness of being able to use first class macros would probably outweigh the slight decrease in performance for edge cases.
Couldn't you use the current system, but if the car is a list, eval it, since you don't know what it is yet? That would only add a performance penalty for complex cars.
Maybe the complex car structures could start with a symbol that signifies that it returns a macro, and otherwise it isn't evaled and the performance penalty isn't incurred.
I.e. add a special form that transforms a function call into a macro call, such as (callmac fn). Then functions and macros could be the same, but you just call them differently. ssyntax could then be used to make it look nicer.
They're what first-class functions are to functions. That is, the ability to create anonymous macros and pass them around as first-class objects. Imagine you could do something like
(let m (macro (x) `(foo ,x))
...)
or
(map (macro (x) `(= ,x 'blah)) xs)
It's like the difference between having fn and just having def. Right now we only have mac and no way to reference macros as their own entity.
arc> ((annotate 'mac (fn (x) x)) 'x)
Error: "Function call on inappropriate object #(tagged mac #<procedure>) (x)"
Whenever the topic resurfaces, the word seems to be the same: they still make compilation difficult (the most recent research only seems to be from 2000 or so http://lambda-the-ultimate.org/node/2987) and pg has yet to comment on them. So to implement first-class macros currently, you'd probably effectively turn ac into an interpreter so that macros could expand at run-time.
That example specifically? It was nonsense. My point was that you could, say, map macros (anonymous or not). Those used to inline functions, for instance: you want to use a macro like a function, but it's a macro for efficiency. Currently, you have to wrap it in a function, which is kludgey.
Silly example:
; instead of
(map complement list-of-fns)
; you have to do
(map [complement _] list-of-fns)
Again, macros being first-class intuitively works like functions being first-class. So, with first-class macros you could have expressions return macros.
(mac m ...)
((traced m) ...) ; traced is a higher-order function
; since m is a macro, (traced m) returns a macro
More popularly, Bawden's paper (http://people.csail.mit.edu/alan/mtt/) demonstrates writing a module system with first-class macros in which the modules are also first-class.
I think the example was supposed to set all of the variable names contained in the list xs to contain the value 'blah.
The example wasn't supposed to do anything. I was just foo-bar-baz-ing something. I don't think the example would actually do that anyways, for the same reason the following doesn't work.
arc> (mac m (x) `(= ,x 'blah))
#(tagged mac #<procedure: m>)
arc> (m 'a)
Error: "reference to undefined identifier: _quote"
I guess it would've alleviated confusion had I said
Well, first class macros don't work right now anyway, so I wasn't expecting your example to work. I was just trying to figure out the idea behind the example.
And I think that first class macros would be required to set an arbitrary list of variables to contain an arbitrary list of values via map. Obviously with or let would work in a local scope, but if you want global variables you'd need something like that, or at least a function that wraps the macro.
Oh, do you mean you want access to the lexical environment at run time? That sounds like to me more like you want the lexical environment to be a first class object.
The lexical environment is (mostly) accessible at run time. The problem is when you're returning a macro from a function, instead of just renaming it.
Suppose you implement packages as tables of symbols. Suppose also that you have a macro stored in said package. There are two ways to use the macro
1.
(= newmac package!newmac)
(newmac args)
2.
(package!newmac args)
I think that most people would like to use the second form, because it doesn't require importing the macro into the current namespace. Unfortunately, the second version requires first class macros.
Obviously, since the compiler can't easily determine that an expression is going to be a macro, it won't have the advantage of compile time optimization. I'm not sure how important that is. Given that arc is already a "slow" language, why not give it more expressive power at the cost of slightly lower speed? Especially since the user can choose whether or not to use them.
So, as I see it first class macros give you several features:
1. easy to make a package system that can contain macros
2. macros fit in better with functions; no wrapping of macros in functions
3. anonymous macros created for mapping across expressions, etc.
evaluation of expr2 and expr3 can be delayed until after expr1 is evaluated. If expr1 turns out to evaluate to a macro, then the literal expr2 and expr3 can be passed to the macro.
The real issue here is any use of nil. There's a lot of mangling in ac.scm to make sure there are nils using things like ar-nil-terminate.
ssyntax: [], :, .
Actually, [] syntax is already defined by reader macros in brackets.scm.
convenient dot notation for rest args
This is Scheme syntax, too.
> (define (f . args) (car args))
> (f 'a 'b 'c)
a
Scheme doesn't have Arc's (o ...) optional parameters, though.
What can't we do with just macros in PLT scheme?
Do you mean adding things on top of PLT Scheme (e.g., with macros) to turn it into Arc (plus whatever else PLT Scheme defines) so that there's no separate "compilation" step? I.e., Start an mzscheme REPL, punch in these definitions, and then start coding like you're in Arc?
Or do you mean more like "what can Arc do that Scheme can't"?
I assume the former, given your title, but I'm unsure to what end this would be.
For one thing, you'd be failing to separate the two languages. It's already pretty muddled: Arc inherits things like read from Scheme, so
arc> #t
#t
arc> (if #t 'a 'b)
a
arc> (if #f 'a 'b)
b
As mingled as the Arc-side and Scheme-side are now, there are some restrictions; e.g., Arc is vector-agnostic (probably because it uses vectors to implement type-tagging) and only knows about certain Scheme functions, but not others (cf. xdef in ac.scm).
arc> #()
Error: "Bad object in expression #()"
arc> first ; Scheme synonym for car
Error: "reference to undefined identifier: _first"
arc> car
#<procedure:car>
If instead "everything is Scheme", you wouldn't have those sorts of "hey, I'll whine if you use these" restrictions. Or is that the aim?
Other difficulties that come to mind are where Arc features step on Scheme's toes directly. For example, atstrings
arc> (declare 'atstrings t)
t
arc> "@(+ 1 2)"
"3"
or re-kajiggering if to accept multiple clauses.
In the limit, you could probably do everything in PLT Scheme (Turing-complete, yadda yadda), it's just a matter of bending over backwards enough: reader macros for atstrings, (define-syntax if ...) the right way, etc.
Or, y'know, just figure out how to make mzscheme's evaluator pass everything through ac first. Ideas for implementation: tl2 in ac.scm... ;)
"Do you mean adding things on top of PLT Scheme to turn it into arc, so that there's no separate "compilation" step?"
Yes.
"I'm unsure to what end this would be."
The reason I ask: I'm running into performance issues with readwarp. I can't have more than 3 users at a time before I start seeing the occasional timeout. At 5 users timeouts become common.
PLT scheme has a GIL and can only run one thread at a time. I think I'm running into that limitation this quickly because arc is a factor of n slower from the rewriting. Caching isn't an option like for HN because it's literally picking one story at a time to render.
So I may have to move reluctantly to a different scheme, or even to common lisp. I'm wondering how much of my house I can carry on my back to the ghetto :)
Even in the context of PLT scheme it seems a valid question to ask why we need this compilation step. Is it buying us enough to justify itself? So far it seems like what we get is nil-termination, optional args and parallel/destructuring assignment. I think a library of macros without those features would still seem mostly like arc to me, and it would be significantly faster.
You should try and see what subset of the language you actually use, and figure out how hard that would be to implement directly on top of scheme.
I don't really know, but the main thing that I see ac providing is the ability to do arbitrary transformations on the first element of a list before it's passed to scheme. This allows functional position definitions for all kinds of objects, and could allow for lots of different kinds of function dispatch (such as dispatch on second arg, which I think pg was planning)
Also, ac (and arc in general) give us a succinctly defined language whose source fits in only two files of less than 1k lines each. This means that the language is easy to learn about and from, and easy to modify. I don't know how many times I've thought "gee, I wish arc could do something cool, like be able to print the source code for global functions" and just gone to the source and added it.
Mind you, that doesn't preclude macros, and in fact arc already is in some ways defined as a set of macros over scheme. It's just that one of those macros is recursive, and visits all of the elements of the source tree ;)
But if performance is what you're after, defining the minimal palatable subset of arc as macros on scheme may be the way to go. It won't be as nice, but it might be a lot faster.
Why not try it, and see? Arc isn't that big, and at least three implementations have been written already.
"the main thing that I see ac providing is the ability to do arbitrary transformations on the first element of a list.."
Ah yes. Everytime I try to enumerate them I miss one. But this page has the whole list :)
"ac (and arc in general) give us a succinctly defined language whose source fits in only two files of less than 1k lines each."
Oh, I'm extremely sensitive to that, believe me. That's what's awesome about arc, and it's kept me using it full-time for 9 months now. Making it a layer of macros would actually not detract from that IMO.
As I've done more with arc I've had to deal with the underlying platform anyway. So I feel like there's no point even trying to abstract that away too much.
"Why not try it, and see? Arc isn't that big, and at least three implementations have been written already."
Yes. You know, in any other circumstances I would consider it a fun experiment. But since I started working on readwarp full-time I constantly worry that I'm going to spend time and burn cash on what's fun rather than what takes the product forward. I'll prob still end up doing this, but just agonize a lot more before I do.
Well, if speed is your main requirement, and you have an actual app that you want to run faster, I would examine it and determine a) what subset of arc you use and b) what subset of arc you need. If it's most of arc, then rewriting it in scheme might be pretty hard.
Of course, first determine how much speed you actually need. And as aw said, rewriting in a faster language won't necessarily help you scale much anyway. Finding where you need the extra speed is definitely a prerequisite to figuring out the best way to get it.
Ultimately, we can't really tell you what to do, because we don't know nearly as much as you do about the needs you have for your application, the bottlenecks that are occurring, or how much time it would cost to re-implement your system on top of scheme directly.
Lisp is great for bottom up design, but whether you can get all the way up to your application before reaching arc first is not something that we can't really judge from here. Certainly some subset of arc can be built as mere macros over scheme. But only you can determine whether that subset is comfortable enough to use, and fast enough to solve your problems.
I don't understand your comment that caching isn't an option because it's picking one story at a time to render. You have a bunch of feeds containing a bunch of stories. Some other background thread / process / program / box can fetch stories, render them, and store the rendered story to disk. Now all your user interface has to do is choose which story to present to the user.
In most programs that are too slow it's a small percentage of the code which is causing the slowness. If you work on making all of your code faster, you're wasting 95% of your effort, because it's only 5% of the code which is causing a problem. So it's worthwhile to run timing tests and figure out what exactly is the slow part, and work on algorithms such as caching to make those parts fast enough.
If you can't think of any algorithms or methods to handle the parts that are too slow, then you can rewrite your program (either the whole program or just the slow parts) in a faster language. But this will only take you so far. If you rewrite your program in a language which is 20 times faster, now you can support 60 users instead of just 3, which is great, but how then do you scale beyond 60 users? At some point you need to figure out what your strategy is for handling lots of users, and rewriting in a faster language is only a temporary fix for that issue.
"If you rewrite your program in a language which is 20 times faster, now you can support 60 users instead of just 3, which is great, but how then do you scale beyond 60 users?"
We're talking about 60 concurrent users. If I could do 60 requests per second I'd be a happy man. 60 requests per second is 216k requests/hr. Assuming 30 page views per happy readwarp user, that's 7200 uniques per hour. Let's say daily uniques at that point is 6 times that, 40k uniques per day. With 40k uniques a day I'd be a happy man. More importantly, 40k uniques will take me n years to get to. So you see, I don't need to "scale beyond 60 users".
Right now readwarp isn't even at 3 requests per second because requests can take several seconds with contention. But that's not ideal either. And it's largely because of the language and the platform.
What you describe in the first paragraph is not caching, it's precomputation. It may help, but it's a completely different performance calculation. The reason caching works for HN is that even if it gets 10k users in 5 minutes it has to generate a dozen pages once that they all see. When I have to manage 100x as many pages in the 'cache' I have to be a lot more careful not to waste time on stories that never get shown.
Precomputation doesn't actually save very much work at all. All it does is move latencies around, often by throwing hardware at the problem. And it can easily go off the rails and end up doing a lot of wasted work without helping latency.
Precomputation does help sometimes. In fact I use it in readwarp in a couple of places. But it's not as straightforward a cost-benefit analysis as caching would be. When caching works it's just brain-dead to reason about. With precomputation I have to constantly monitor whether I'm getting the bang for my buck. And I have to be conservative.
The bigger question is why arc behaves like it takes 3-5s to render one page. All I'm doing is selecting a story and rendering it.
Hmm, I probably haven't fully explored all these ideas. Every now and then my server just hangs for a few seconds and becomes completely unresponsive, and I haven't been able to figure out what it's doing at that point, to get a stack trace or something. For any timing instrumentation I insert, times often vary wildly, by an order of magnitude. I'm starting to suspect PLT's timings shoot through the roof because of contention between threads.
I should just try to offload all my state in a way that can handle multiple coexisting arc interpreters. I'll probably do that. Arc seems quite memory-heavy, though. I'm not sure how many interpreters can coexist on a machine.
You're right that I'll still have to scale in another language, but language features can make my life easier or harder when I have to scale to multiple machines. Having a global interpreter lock perhaps makes it hard to avoid thread contention, and to balance responsiveness with throughput. I know the python folks have had similar issues. OTOH, having just one request per interpreter makes arc far more inefficient in RAM than say mongrel.
"The bigger question is why arc behaves like it takes 3-5s to render one page. All I'm doing is selecting a story and rendering it."
Actually there's 3 stages. Selecting a story (A), rendering it (B), and updating user data structures when the vote on it returns (C). Each user request is seeing the total time for C+A+B.
When I eliminate rendering (the server does all the computations, but doesn't send the story to the client) I still see timeouts with 3 concurrent requests.
---
Ah, I got an improvement. Remember those 'couple of places' I was doing precomputation? I was trying to batch up stage-A computations 5 at a time in a precompute thread when a user's running low on stories. I got rid of them, and the average latency for everyone went up from 1s to 3-4s even with just one concurrent request at a time. But I can now handle 6 concurrent requests without timing out.
When I eliminate stage C the number starts inching up to 10. At 10 median request time is 13s. But at least stuff is getting processed.
My lessons:
1. When the interpreter is single-threaded try to minimize use of threads. Contention gets expensive fast. I am now used to seeing processing latency double when #concurrent requests doubles.
2. When things are slow resist the temptation to precompute. It always makes hot spots harder to see in the future. It's a last resort. And it'll always cost you on a single-threaded platform. I'm now going to rerun my profiling.
3. When you're stuck come blather about your problems on the arc forum. Thanks guys.
So I can vary the number of assignments each op does in the URL.
Experiment 1: just a hundred sequential requests.
$ ab -n 100 "http://127.0.0.1:8080/?iters=10"
On my laptop 98% of requests complete in 3ms.
Experiment 2: let's inch up to 2 concurrent requests at a time.
$ ab -n 100 -c 2 "http://127.0.0.1:8080/?iters=10"
Now 50% of requests require at least 4ms. For 98% it's 10-15ms.
Experiment 3:
ab -n 100 -c 10 "http://127.0.0.1:8080/?iters=10"
Now 50% of requests require 20-40ms.
In general the 50% latency number that apacheBench returns seems to increase linearly with the number of concurrent requests. This rule of thumb has held in my app as well over several major changes. Right now readwarp requests take 800ms on localhost and 2-3s from the internet at large. Not much room for those latencies to grow linearly.
I can't compare your first and last example because one says 98% and the other 50%.
If it were an apples to apples comparison would you agree that it wouldn't be surprising that if the server were handling 10x the number of simultaneous requests that each request would take 10x as long?
The only apples to apples comparison is going from one simultaneous request to two simultaneous requests which has the increase from 3ms to 10-15ms, which seems like about double what we'd expect if using threads had no overhead at all.
When I give just the 98% number I meant the 50% number stays the same (+/- 1ms). We're at the limits of ab's time resolution here.
Yes, none of this is surprising given that PLT is single-threaded. I'm trying to remember my class in queuing theory.. If each request takes m ms to process, n concurrent requests will take mn ms to process, (n-1)m ms of queuing delay and m ms of processing time. The average latency seen would be mn/2 which scales by n. As the number of concurrent requests grows, the curve of latency to number of concurrent requests would be linear.
If you now add a second queue, the curve stays flat until 2 requests and then moves linearly up. Each queue is only n/2 long, so average queuing delay is (n-1)m/4 ms. With q queues, on average there's only (n-1)m/2q ms of queuing delay.
Summary: if you can have multiple truly-concurrent threads, you'll be able to overlap that many concurrent requests without any increase in latency, and the slope of the curve past that point will be inversely proportional to the number of threads. Does this make sense?
---
After I wrote my previous comment I started looking in greater depth into why I'm taking 800ms to process a request. I'm really not doing that much. I'll post an update with what I find. Perhaps I haven't spend enough time attacking m before looking to expand q.
When you say that PLT is "single-threaded", is what you're referring to is that PLT only uses one CPU?
Well, yes, if you have 10 CPU's then you can handle 10 simultaneous requests in the same elapsed time as 1 CPU can handle 1 request. If you don't have any overhead in communication between CPU's, locks, etc.
I think it's clearer to talk about having one CPU or multiple CPU's instead of "truly-concurrent threads" or "thread contention".
You can have multiple threads interleaving on a CPU, and in some systems that support more than one CPU, a thread can run for a while on one CPU and then get moved to running on a different, less busy CPU.
Now if you said that that on one CPU 100 requests one at a time took a total of 300ms to process but 100 requests running interleaved on one CPU took a total of 800ms to process, then we would have evidence that there's too much overhead in thread context switching or some other problem.
I think the quality of the thread implementation has a role to play. When a process waits on IO the OS will find something else to do. I don't know if user-level threads can be that smart, and I don't know if PLT threads are that smart.
Without casting aspersions, it's not just about number of processors. Very few programs avoid IO altogether, and if there's IO some programs will be more clever than others in moving work around.
In a properly working thread system, all threads that aren't waiting for something should be given a fair slice of the available CPU time. In particular, no thread that is ready to run should be blocked or delayed because some other thread is waiting on I/O.
If you are wondering if PLT threads work correctly, isn't this rather easy to test? Do an A/B test with some threads that do some computation with and without some other threads waiting on I/O and see if the runtimes are the same.
It seems like things are straightened out here, but I thought I'd point out that the upcoming version of PLT allow multi-core parallelism via 'future':
Also: PLT does not have a GIL, but (futures aside) it has a similar mechanism that works better for implementing continuations (and worse for interoperating with C code)
The syntax highlighter has increased my quality of life substantially (insomuch as my life is coding in Vim, which it is for a good part these days :) .
The red highlighting that suddenly appears for mismatched parens is very useful. Saves me a lot of cursoring around to make sure all are matched up. The only nuisance in my own usage is sometimes it's distracting if I deliberately have a temporary mismatch while editing code.
The parenthesis highlighting is a bit of an all-or-nothing proposition, since Vim can't really tell if you intended to leave unmatched parentheses. You can disable it altogether (which won't highlight any paren errors) with
Thus, instead of doing (defop r), it's kind of like we're doing (defop (eval r)). That is, instead of expanding into
(defop r req (rpage 'r))
we're expanding into
(defop 10a req (rpage '10a)) ; when r == '10a
(defop 10b req (rpage '10b)) ; when r == '10b
; etc
We could've done the eval inside the macro, too, but that's often a sign you're doing something wrong --- macros are usually there to not evaluate their arguments. So we should probably use a function.
However, defop itself is a macro, so the first parameter (the op's name) won't be evaluated regardless. We still need eval.
Since this approach seems to require eval regardless, we should just look for a better solution. aw's works nicely.
Some other nitpicks over your rewrite (hey, you asked!):
1) Unless you need strings for some reason, you can probably default to symbols (they're a bit easier to use).
; Instead of
(= routes* '("10a" "10b" "15a" "20a" "20b" "30a" "30b" "30c" "40a" "59u"))
; we could use
(= routes* '(10a 10b 15a 20a 20b 30a 30b 30c 40a 59u))
2) Proper spacing & indentation saves lives. :)
; Instead of
(defop shuttle req
(each r routes* (link r)(nbsp)))
; why not
(defop shuttle req
(each r routes* (link r) (nbsp)))
; or even
(defop shuttle req
(each r routes*
(link r)
(nbsp)))
3) Though using symbols renders this point moot, fromstring is unnecessary to simply (read) from a string, since Arc's definition of read is
(def read ((o x (stdin)) (o eof nil))
(if (isa x 'string) (readstring1 x eof) (sread x eof)))
So, instead of
(fromstring rs (read))
you can use
(read rs)
If your goal is just to turn a string into a symbol, you should use
(sym rs)
This is an important distinction. e.g.,
arc> (sym "abc")
abc
arc> (read "abc")
abc
arc> (sym "(a b c)")
|(a b c)|
arc> (type that)
sym
arc> (read "(a b c)")
(a b c)
arc> (type that)
cons
Very instructive. Thank you for doing such a thorough analysis.
',r from your snippet
(mac rdefop (r)
`(defop ,r req (rpage ',r)))
was a realization for me. Never thought of quoting a comma'd symbol in a backquoted expression before, but I like knowing it's possible. Do you find yourself doing this much, or is there usually something simpler like aw's solution available to make it unnecessary?
I'll still be combing over pg's pprint.arc, though. By keeping the changes small, modular, and as close to the original as possible, I figure that (a) there's a smaller chance of bugs, as each change is more understandable; (b) the code will be compatible with vanilla Arc; and thus (c) even if just a few fixes are cherry-picked, we might actually get a decent ppr in the next release. If nothing else, it's kind of fun. :)
Indeed, that's the main reason I wrote my version of ppr ;) That's how I originally started my version - I was annoyed by the fact that pg's wouldn't properly print quasiquotes in my source code (and it's hard to debug macros that way) and just couldn't stop because it was so much fun.
And to be honest, I don't really know what the outstanding bugs are, or whether or not I ended up fixing them. It's been a while since I last looked at ppr.
I see how lisps can implement many features... but I can't see how those features could include adding to the syntax...
Actually, that's an interesting feature of several Lisps: reader macros. For example, in Arc you can make anonymous single-argument functions with some special syntax.
arc> [+ _ 1]
#<procedure>
arc> ([+ _ 1] 1)
2
In Common Lisp, you are resigned to the more verbose
[1]> (lambda (x) (+ x 1))
#<FUNCTION :LAMBDA (X) (+ X 1)>
[2]> ((lambda (x) (+ x 1)) 1)
2
What if we want the bracket syntax in Common Lisp? We can extend the parser to dispatch on certain characters, hence "reader macros":
Now when Common Lisp sees an open bracket, it will read from the stream until a closing bracket is found, then shove that all into a (lambda (_) (...)) form.
Arc doesn't have reader macros (though uses Scheme's for bracket functions; see brackets.scm), but they're an interesting way to work in new syntax (though it probably won't be much fancier than Lisp's already-minimal syntax).
Arc does, however, extend syntax in certain ways. I gave an overview of this so-dubbed ssyntax in another post: http://arclanguage.org/item?id=11179.
This is how awesome tables are.
1. Arc doesn't have destructuring assignment built in, though macros make it easy (see http://arclanguage.org/item?id=8122; it's a bit old, so set is now called assign). Otherwise, the standard approach is
arc> (= h (table))
#hash()
arc> (= h!a 1 h!b 2 h!c 3)
3
arc> h
#hash((b . 2) (c . 3) (a . 1))
2. In Lua, I gather that
t = { a = 1, b = 2, c = 3 }
is the same as
t = { ["a"] = 1, ["b"] = 2, ["c"] = 3 }
In Arc, this looks more like strings versus symbols (a datatype I don't think Lua has). Symbols are simply-compared blobs of text, so you often use them as keys in hash tables (e.g., that's what I did in 1). Macros give you a shortcut for this common case.
arc> (= h (obj a 1 b 2 c 3))
#hash((b . 2) (c . 3) (a . 1))
3. obj works by quoting the keys you pass in. Quoting literal data will just return that data, so strings can be used in obj, too.
arc> (= h (obj "a" 1 "b" 2 "c" 3))
#hash(("c" . 3) ("a" . 1) ("b" . 2))
4. We can't use ssyntax here as in 1 because of its inherent limitations. But we can still do
The standard libraries, e.g string, math, etc are tables. e.g math.sin, math.cos.
I wish Arc even had modules. :) But you could do the same thing, in principle.
arc> (= math (obj sin sin cos cos tan tan))
#hash((tan . #<procedure:tan>) (cos . #<procedure:cos>) (sin . #<procedure:sin>))
arc> (math!sin 3.14)
0.0015926529164868282
In fact, Arc uses this approach for sig.
arc> (type sig)
table
arc> (sig 'rand-key) ; looks like a fn call, but it's a hash lookup
(h)
arc> (n-of 5 (rand-key sig))
(urlend urform admin-gate caris trues)
Tables can have any value as keys besides strings, that includes functions and even other tables.
I think that goes for hash tables in general: anything that's hashable. ;)
arc> (def f (x) (+ x 1))
#<procedure: f>
arc> (= h (table))
#hash()
arc> (= (h f) 5)
5
arc> h
#hash((#<procedure: f> . 5))
arc> (h f)
5
arc> (= h2 (obj a 1 b 2 c 3))
#hash((b . 2) (c . 3) (a . 1))
arc> (= (h h2) 'hi)
hi
arc> h
#hash((#<procedure: f> . 5) (#hash((b . 2) (c . 3) (a . 1)) . hi))
arc> (h h2)
hi
lua doesn't come with it [OO] by default, but you can implement it with tables in a dozen lines
That's how Arc's "object system" works.
arc> (deftem object a 1 b 2 c 3)
((a #<procedure: gs1820>) (b #<procedure: gs1820>) (c #<procedure: gs1820>))
arc> (inst 'object 'a 5 'c 10)
#hash((b . 2) (c . 10) (a . 5))
A table can have a `metatable` (mt).
I think I understand what you're saying (it sounds a lot like Python's operator overloading), but I'd probably need to use Lua to appreciate it.
Here's a linked list created with a table.
Aha. I was expecting this earlier: in Lua, you can use tables to make arrays. It's all pretty unified syntactically, whereas Lisps tend to add functionality with Yet Another Macro (instead of overloading syntax).
arc> (def array values
(w/table a (on value values (= (a index) value))))
#<procedure: array>
arc> (array 'a 'b 'c)
#hash((0 . a) (1 . b) (2 . c))
arc> (array 'a (array 'b (array 'c (array))))
#hash((0 . a) (1 . #hash((0 . b) (1 . #hash((0 . c) (1 . #hash()))))))
If Arc had reader macros, we could make {...} expand into (array ...). But saying "hey, you could bolt that on with a function" is a bit disingenuous: we're still comparing apples and oranges.
Okay you got me. Lisp can do all that, too ;). About reader macroes: If arc had them and you really liked the {} syntax, would to write a macro for it? And what if you're writing an arc library, would it collide if other people did something else with their braces?
(it sounds a lot like Python's operator overloading)
re mt: (+ t t1) would fail because they aren't numbers. But if they overloaded the + operator then it would do something instead. Yes it's like python's op-overload but a bit better because of __index and __newindex.
But it's more important whether a language can do something nicely, which I don't necessarily claim Lisp can. After all, Turing machines can do all that, too. :)
would it collide if other people did something else with their braces?
You could hypothetically package the reader changes into their own modules so that you'd need to explicitly say which braces you're using, modules keep their changes local, etc. Anyone know of work done on this front?
Yes it's like python's op-overload but a bit better because of __index and __newindex.
So are __index and __newindex roughly like in the following (silly) Python code?
$ python
Python 2.5.4 (r254:67916, Jan 24 2010, 12:18:00)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class foo:
... def __init__(self, x):
... self.x = x
... def __getitem__(self, index):
... return self.x + index
... def __setitem__(self, index, value):
... self.y = index + value
...
>>> bar = foo(5)
>>> bar[10]
15
>>> bar[20] = 5
>>> bar.y
25
If so, I imagine that overloading table operators in Lua becomes all the more powerful since tables are so ubiquitous. Thanks for the explanations.
You could hypothetically package the reader changes into their own modules so that you'd need to explicitly say which braces you're using, modules keep their changes local, etc. Anyone know of work done on this front?
Hmm, giving the job of custom readers to modules strikes me as making implementing both modules and custom readers complicated and difficult; and makes unnecessary impositions on the programmer: I might well want to implement one part of my module using one syntax and another part of my module using a different syntax.
On the other hand I think being able to choose your custom reader by source code file would simple to use, simple to understand, easy to implement, would make it easy to customize your editor or IDE to understand your different syntaxes by file, and means that modules can also be easily implemented as they could do their work entirely after the source code has been read in.
I wonder if modules could be implemented with namespaces? You want to call my function foo which calls my function bar, but you don't necessarily want to have to name it "foo" in your own code and you want to be able to have your own "bar" without breaking my function foo. I'll have to think about that one.
I wonder if modules could be implemented with namespaces?
I guess I use the words "module" and "namespace" interchangeably. I'm not sure what else "module" ("package", "namespace", sometimes "library" (loosely), etc.) would mean.
(use "braces-for-set-builder-notation.arc")
(load "a.arc")
(def set-I-really-want ()
{ (* v v) for v in (vals (table-I-really-want)) })
You wouldn't want a.arc's use of braces for tables to leak into b.arc, which uses braces for set-builder notation (which you further don't want to clobber the braces in the definition of table-I-really-want).
With a fancier module system, you might even have
b.arc
(qualify "braces-for-set-builder-notation.arc" s)
(load "a.arc") ; doesn't qualify its braces for tables
(def set-I-really-want ()
s{ (* v v) for v in (vals (table-I-really-want)) })
(def another-table-I-want ()
{foo 5 bar 10 baz 15})
Though there shouldn't be a reason you couldn't do something like
c.arc
(use "braces-for-table.arc")
(def some-table ()
{x 5 y 10})
(use "braces-for-set-builder-notation.arc")
; braces hereafter are for set-builder notation
(def some-set ()
{ x for x in (vals sig) })
As Arc lacks a proper module system, you can already see the clobber-some effects of definitions being global/unqualified. APIs leak all over the place:
arc> (load "sscontract.arc") ; only really needs to provide sscontract
nil
arc> (priority #\!) ; but the user gets all of the implementation details!
1
Similarly, I don't want to load some library and have implementation-detail reader macros polluting the reader. (If the library's specifically for reader macros, then of course that's okay.)
Even some simple public/private control would be nice. I've played around with this in Arc before. E.g.,
(let localize nil
(assign localize
(fn (expr fs)
(when (caris expr 'def)
(if (~mem (cadr expr) fs)
`(assign ,(cadr expr) (fn ,@(cddr expr)))
expr))))
(mac provide (fs . body)
(let private (trues [and (caris _ 'def)
(~mem (cadr _) fs)
(cadr _)]
body)
`(let ,private nil
,@(map [localize _ fs] body))))
)
arc> (macex1 '(provide (f g)
(def f (x) (h (+ x 1)))
(def g (x) (+ (h x) 1))
(def h (x) (* x x))))
(let (h) nil
(def f (x) (h (+ x 1)))
(def g (x) (+ (h x) 1))
(assign h (fn (x) (* x x))))
arc> (def foo () (prn "outer foo") nil)
#<procedure: foo>
arc> (provide (bar)
(def foo () (prn "inner foo") nil)
(def bar () (prn "bar") (foo)))
#<procedure: bar> ; note: no redef warning, since def changed to assign
arc> (bar)
bar
inner foo
nil
arc> (foo)
outer foo
nil
But this particular approach doesn't interact with macros nicely, doesn't let you qualify namespaces, isn't general, etc. So it's really suboptimal.
I find this line really interesting. If I'm reading it right, you can alter the metatable (operator overloading) on the fly? So you could pass in a table as a parameter, set a metatable, play with some operators, then set a different metatable to overload operators differently?
In Python, the operators are kind of tied to specific classes. I guess you can go about casting and converting and inheriting and monkeypatching and such, but I don't think it's as straightforward as just setting a new metatable. Admittedly, I can't think of when I've ever needed to alter operator-overloading dynamically like that, so I've never really tried. Do you know if it's useful in Lua code? Or am I completely off-base here?
Hmm, the only times I've used it as where I'd be casting objects in other languages... I've used it to 'cast' between 'tables' doing similar things. like `my1.rect` to `my2.rect`, if they both use .x, .y, .width and .height to store their coordinates. (Not sure that's a good practice because they can change.) Yes, you can alter metatables any time, but I don't do it that often.
