I didn't forget: Nulan completely solved the macro hygiene problem after all. But that's a more extensive change so I figured I'd save it for after the basic hyper-static scope system is in place.

In fact, assuming Arcueid does implement my proposal, I would actually go in and make a new version of arc.arc that uses "var" and has hygienic macros by default. Then you could simply load up the new arc.arc to get all the shininess. But loading the old arc.arc would have full compat with existing Arc 3.1 programs.

---

"I think you've been resolving this by writing macros so that the procedures are inserted directly into the macro result, rather than referred to indirectly by symbols. Right?"

Nope. Macro hygiene in Nulan just uses the already existing box implementation. It's really easy, really simple, and really fast. Seriously, boxes are awesome. No need to complicate things.

The way to solve it in Arc: just provide a function called "get-variable-box" which is only available at compile-time and it returns the box for the variable.

Then you change quasiquote so it uses "get-variable-box" rather than inserting the symbol directly. Bam, hygienic macros with no additional runtime cost, and extremely small compile-time cost. And they look and feel just like Arc macros, so you don't lose any power or convenience. No clunky Scheme macros, huzzah!

Once I understood that the fundamental problem was with dynamic scope, and the best way to solve it is with boxes (or similar), everything became super easy and awesome.

---

"The behavior I'd use is that any compile-time variable access (even under a lambda) creates a new, uninitialized variable binding if a binding doesn't already exist."

The middle ground doesn't seem worthwhile to me. When programmers work with with Arc-style unhygienic macros, at each use site, the variables in scope must (mostly) match the variables the macro author expected. So I think people who like using macros will be happiest if they systematically keep the variable names consistent across all the code in their program (even others' code), at which point namespace mechanisms just get in the way.

---

"Nope. Macro hygiene in Nulan just uses the already existing box implementation . It's really easy, really simple, and really fast. Seriously, boxes are awesome. No need to complicate things."

I think you caught me on a technicality. :) I see "procedures are inserted directly into the macro result" as a general approach. Mutable boxes make it possible for this approach to achieve late binding. Elsewhere in this discussion you tilt the technicality closer to my phrasing, since you recommend to let users build boxes out of getter and setter procedures.

Anyway, I'm a fan of that approach when it works, but it doesn't work so well when compilation is involved: The macroexpanded code contains unserializable values--namely, the procedures or boxes we're talking about. This is a lesson I learned with Penknife, where I at first had macros insert boxes, and then had to reengineer this so macros inserted step-by-step treasure maps for how to find a variable from the run time environment.

---

"Yeah I'd do that too, if I wanted to graft dynamic variables onto a hyper-static system. But since Arc uses dynamic variables, I proposed to graft hyper-static onto it instead."

Retaining Arc compatibility in general doesn't seem worthwhile to me, but a lot of people want it, so I gave a system that retains Arc compatibility while tacking on some new shininess. Nulan doesn't have to worry about Arc compatibility, so it has pure hyper-static scope and hygienic macros by default.

---

"How do you make the even/odd code work? Under the approach you described, the first line refers to an undefined variable (odd), and I interpret that as an error. I was recommending a fix."

Easy: I have a macro called "defs" that handles mutual recursion:

  (defs
    even (x)
      (if (is x 0)
        t
        (odd (- x 1)))
    odd (x)
      (if (is x 0)
        nil
        (even (- x 1))))

  (var even)
  (var odd)
  (= even (fn (x)
    (if (is x 0)
      t
      (odd (- x 1)))))
  (= odd (fn (x)
    (if (is x 0)
      nil
      (even (- x 1)))))

---

"I think you caught me on a technicality. :)"

Maybe it was just a simple miscommunication. What you were talking about sounded exactly like the technique of splicing function values using macros:

http://www.arclanguage.org/item?id=14507

What I'm talking about happens entirely at compile-time using boxes. The effect is very similar, but the implementation is very different.

---

"Anyway, I'm a fan of that approach when it works, but it doesn't work so well when compilation is involved: The macroexpanded code contains unserializable values--namely, the procedures or boxes we're talking about"

Sure, if I cared about serialization, I'd have to make it more complicated. Thankfully, the only serialization I care about right now is compiling to JavaScript code, which is easy enough to do with variable renaming.

While I appreciate 'defs, it's a non-answer. The even/odd example I posted and the evenp/oddp example dido posted are idiomatic Arc code. While you and I don't care much about Arc compatibility, it's something dido wants for Arcueid, so these examples should work without modification.

---

I'm about to disagree with myself, but first I want to reiterate and clarify what I was saying at "caught me on a technicality":

For this discussion I don't see a much of a reason to distinguish between macros which insert mutable boxes and macros which insert functions. Either system can pretty much support the other as a special case: We can translate spliced boxes into spliced getter/setter functions, and we can translate spliced functions into spliced functions-in-the-box. Because of that equivalence, these systems share the disadvantage of being challenging to serialize.

If dido considers compilation to be important (do you, dido?), then this hygiene approach might be unsuitable, and thus the use of first-class namespaces might be unsuitable. (As I explained at "match the variables the macro author expected," first-class namespaces make hygiene more important.)

---

"What I'm talking about happens entirely at compile-time using boxes."

Ah. I think you have a point!

For compiling Nulan to JavaScript, I guess the boxes you're using aren't arbitrary getter/setter functions, and they aren't merely some mutable container either; they're globally associated with a JavaScript variable name. When you compile the macroexpansion result and it contains a (get-variable-box ...) form, you decide on its JavaScript variable name at that time. If the macroexpansion result contains a box, you use the attached variable name to compile it to JavaScript. Am I getting this right? This sounds very workable. :) And whaddayaknow, Nulan works. ^_-

I seem to remember understanding this before, when you and I talked about Nulan compilation in depth, but I guess I had to retrace the steps just now.

Anyhow, get-variable-box is fantastic IMO, but first-class namespaces still might not be ideal for Arcueid due to Arc's unhygienic macros.

dido, are you comfortable with breaking existing Arc macro idioms in favor of hygiene?

---

I have a convoluted but surprisingly comprehensive idea of how to integrate get-variable-box into a system that's compatible with unhygienic Arc macros, but I've put it in a separate simultaneous post: http://arclanguage.org/item?id=17464

For this example, let's suppose there was a file "foo.arc" that contained idiomatic Arc code that implements evenp/oddp. This code works in Arc 3.1. It will work in my system as well, because undefined symbols automatically create new boxes. Basically, it'll work, but name collisions are possible, just like in Arc 3.1.

If you then write a new file "bar.arc" that uses hyper-static idioms (var, defs, etc.), it can import "foo.arc" and everything will work fine. "foo.arc" will clobber any existing evenp/oddp definitions, but "bar.arc" will not clobber "foo.arc". And of course "bar.arc" can use "w/include" and "w/exclude" to prevent "foo.arc" from clobbering things.

If you wanted to make it so that "foo.arc" behaves correctly without needing to use "w/include" and "w/exclude", you would indeed need to rewrite it to use "defs". But it's still usable even without a rewrite. So it's a perfectly graceful degradation.

My system is designed so that it can correctly use all existing Arc 3.1 code, while new code is written with the hyper-static idioms. Then, slowly, old code can be migrated to use hyper-static scope, until eventually you could make Arc purely hyper-static.

There's three issues I see with my proposal:

1) If you're writing Arc code in a hyper-static fashion, you really want "arc.arc" to be changed to be hyper-static. But old Arc code will need the non-hyper-static "arc.arc". I think the simplest solution to this is to have two versions of "arc.arc", one that uses hyper-static scope, and one that doesn't. Then you would need to make sure to load the non-hyper-static version before loading Arc 3.1 code. This could be automated a tiny bit by using a macro, something like "w/arc3".

2) "load" occurs at run-time, which is why my definition of "w/include" needed to use "eval". Nulan doesn't have this problem because file importing occurs at compile-time. Perhaps the best way to solve this is to keep "load" as-is, and add in a new "import" macro that does all its work at compile-time.

3) If you think (eventually) making Arc purely hyper-static is a bad thing, you won't like my proposal.

---

"Am I getting this right? This sounds very workable. :) And whaddayaknow, Nulan works. ^_-"

Another way to think about it is... the compiler would replace this:

  (= foo (fn () ... bar ...))
  (= bar (fn () ... foo ...))

  (var bar)
  (var foo)
  (= foo (fn () ... bar ...))
  (= bar (fn () ... foo ...))

We've just had a long exchange about you creating boxes at assignment time and me using compiling-a-reference time instead. Here's a recap:

---

You: Here's how you do it. The definition of "=" is the same: if the variable exists, mutate it, otherwise create a new variable. But now you add in a new primitive called "var"[...]

Me: The behavior I'd use is that any compile-time variable access (even under a lambda) creates a new, uninitialized variable binding if a binding doesn't already exist.

You: Yeah I'd do that too, if I wanted to graft dynamic variables onto a hyper-static system. But since Arc uses dynamic variables, I proposed to graft hyper-static onto it instead.

Me: How do you make the even/odd code work? Under the approach you described, the first line refers to an undefined variable (odd), and I interpret that as an error. I was recommending a fix.

You: Easy: I have a macro called "defs" that handles mutual recursion

Me: While I appreciate 'defs, it's a non-answer. The even/odd example [...] should work without modification.

You: It will work in my system as well, because undefined symbols automatically create new boxes.

---

A large part of the problem is that I've been thinking about my proposal as two separate parts: one part deals with backwards compat with Arc, and the other part describes a hyper-static system for Arc.