Arc Forum | "ac-tunnel" proposal

Arc Forum

"ac-tunnel" proposal

3 points by CatDancer 5675 days ago | 27 comments

I propose adding a primitive called "ac-tunnel" to Arc:

   (define (ac s env)
     (cond ((string? s) (string-copy s))  ; to avoid immutable strings
           ...
  +        ((eq? (xcar s) 'ac-tunnel) (cadr s))

ac-tunnel is a syntax that passes its arguments unchanged through the Arc compiler:

  (ac-tunnel (+ 1 2))

"(+ 1 2)" appears unchanged in Arc's compiled Scheme output and so is evaluated by Scheme as a Scheme expression. The syntax is not limited to literal Scheme code: an Arc macro can expand into ac-tunnel, and so can generate Scheme code.

Name

I suggest calling this feature "ac-tunnel" because it is part of the Arc compiler. Although people liked the "mz" name that I had been previously using, I think that "ac-tunnel" fits in better with the naming of other compiler functions such as ac-denil, ac-niltree, ac-global-name, etc. And, it's easy to define "mz" in terms of "ac-tunnel", for people who would like easy access to Scheme features for quick hacks.

Motivation

"ac-tunnel" can be used as a mechanism to call Scheme libraries from the MzScheme implementation of Arc, or to modify Arc's runtime.

The advantage I see in adding "ac-tunnel" as a primitive to Arc is that it enables capabilities without them having to be made part of the official release of Arc. Most everyone who programs in Arc wants to see their favorite feature X as part of Arc. (Often expressed online as "how can anyone claim that Arc is any good when it doesn't even have X??!") However pg may not want to add X to Arc until he's had a chance to think about a good way to do it.

Any code (aside from Arc itself) that uses "ac-tunnel" is by definition going outside the boundaries of official Arc, but "ac-tunnel" creates a safety valve for people who want an X that isn't in Arc by making it easy to get X.

Comparison to Anarki's "$"

"ac-tunnel" is a more general syntax than Anarki's "$" function which evals a Scheme expression: "$" can be defined using "ac-tunnel", but the reverse is not true.

Comparison to xdef

For calling libraries written in Scheme, this would have a different purpose than the implementation of Arc primitives done with xdef in ac.scm. We have to have primitives: we can't implement Arc without haven't something to implement Arc on top of. Calling libraries written in Scheme lets us use code that could be ported to Arc but hasn't been yet; or might be done for efficiency; but we can call a Scheme library without having to make that library part of the definition of Arc.

Why not name after the backend

To answer the question of why shouldn't the "pass-through-compiler" feature be named after the backend ("mz") since a particular "ac-tunnel" expression will be useless on some other backend, consider a scenario where an Arc program running on top of MzScheme and one running on top of some other backend such as Common Lisp might both want to call a Scheme library.

To make calls to a Scheme library, we'd probably have some interface functions such as, for example, a toscheme to convert Arc values to Scheme values and fromscheme for the other direction; a function to call a Scheme function with the Arc arguments automatically converted into Scheme values and back again for the return value, and so on.

The implementation of the interface functions would naturally be very different depending on the backend: in MzScheme, the interface functions would be calling parts of the Arc compiler such as ac-niltree to help with the conversion of Arc and Scheme values; while in Common Lisp version might be communicating with a separate Scheme process, or be calling an implementation of Scheme written in Common Lisp.

Thus the interface functions need to know which backend they're running on, and so will know whether it can use "ac-tunnel" in its implementation or not. The Arc code which only wants to call a Scheme library will use the interface functions, and so doesn't need to know about or use "ac-tunnel" directly.

2 points by rntz 5675 days ago | link

I do not see why you feel the need to change a perfectly good name.

It may be similar to other names that are part of the arc compiler, but those names are scheme names. This name is visible in arc, and arc naming conventions are "short and sweet". 'mz is a perfectly good mnemonic for what it does, and is very unlikely to need to be taken for anything else.

I do not understand how your "Why not name after the backend", in fact, explains why we should not name it after the backend. You admit yourself that "the implementation of the interface would naturally be very different depending on the backend" - so it won't be a problem that the "lower to CL" version of mz is named something different. You don't, AFAICT, present any argument that 'ac-tunnel should be named the same with different backends, merely an argument that, for people building library interfaces, it won't be much trouble. But not everyone is engineering libraries: arc is to a large extent about experimentative coding, and there are no significant disadvantages to just letting it be named 'mz. Sure it's easy to just (mac mz body `(ac-tunnel ,@body)), but why should it be necessary?

In the end, this is a fairly small issue. I hardly want to get into a flame war over it, and it's your hack... but, as I say, I don't get why you want to fix something that isn't broke.

-----

2 points by CatDancer 5675 days ago | link

I actually don't care what the name is! :-)

I do think that this would be a useful facility to add to Arc. I think it has high leverage, in that it is a small addition to Arc that has a lot of power: with this change several features can be written with code that can be loaded instead of needing patches.

pg tends to go for short names for function/macro names that need to be typed a lot. I was concerned that something as short as "mz" was too valuable an identifier to be used for something that most people wouldn't be typing.

But if people want to name it "mz" or something else that refers to MzScheme, that's fine with me!

-----

2 points by shader 5675 days ago | link

Maybe it's just that I've been writing a lot of libraries, but I use 'mz a lot ;)

Besides, I can't imagine needing that combination of letters for anything else. Especially after having it refer to mzscheme for so long now, it would be hard to switch it.

-----

2 points by CatDancer 5675 days ago | link

I also really like "mz".

It doesn't matter to me what it gets named as. pg could add it to the Arc core as %%%internal-compile-to-mzscheme, and I could still use "mz" in the code I write using load-w/rename or some similar mechanism.

But hey, if everyone likes "mz", let's use that!

-----

1 point by Adlai 5675 days ago | link

I think that FOR NOW, 'mz is fine. Even though each person has their own patched-up Arc, it seems as though the vast majority are based on pg's Arc on mzscheme. If somebody is using an Arc built on some other VM (eg Rainbow on the JVM), then they will be putting quite different code within the body of an 'mz, so I don't think it's an issue that they'd also have to change the name of the symbol.

-----

1 point by conanite 5675 days ago | link

I completely agree ac-tunnel (or mz) is a useful facility to add to arc, especially to use for once-off hacks, or to figure something out before committing to an xdef. I wrote rainbow just so I could use java libs while playing in arc (among other reasons).

As the author, you get to name your babies. I didn't agonise over java-new, java-implement, or (ugh) java-static-invoke - they're mostly buried under arc macros in any case.

In the worst case, if there's a conflict with mz (or ac-tunnel), I would recommend an excellent symbol renaming hack available on github at http://catdancer.github.com/load-rename.html

:))

-----

1 point by rntz 5675 days ago | link

Oh. That's good then. Whoops. :) Now I feel like a bit of a jerk for ragging on about the name so much.

I am completely in agreement that this is a good feature. When anarki was still up-to-date, I used '$ quite a bit, and this is essentially just a better version of that - one that lets you access local variables and has less overhead to boot!

-----

1 point by shader 5675 days ago | link

Speaking of Anarki, what needs to be done to get it back up to date?

I haven't heard much about that. Maybe we should make a new thread about updating it?

You're comment about local variables made me think of something:

What if we used ac-tunnel, like CatDancer suggested, and then make mz defined as:

  (mac mz body
     (list 'ac-tunnel (cons 'quasiquote body)))

That way mz is defined in terms of the tunnel, and adds the feature of being able to unquote variables to pass the value through to scheme. $ had that feature, but called seval instead, so it lost the local environment. This way, we get the local environment, and the quasiquoting.

Thoughts?

-----

1 point by rntz 5675 days ago | link

Making the entire thing quasiquoted doesn't work. That would mean that the following:

    (let x 2 (mz (+ 2 ,x)))

Would compile down to this:

    ((lambda (x) `(+ 2 ,x)) 2)

Which obviously is not the intended functionality - it evaluates to the list (+ 2 2), not to the value 4. A quasiquote is, after all, a quote. So you'd need to add an 'eval outside it, and then you just have '$. Anyway, you already get the local environment with 'mz, because arc compiles local variables down to the same-named variables in mzscheme. That is to say, with 'mz as it currently stands, the following:

    (let x 2 (mz (+ 2 x)))

Compiles into:

    ((lambda (x) (+ 2 x)) 2)

So it just works! This does depend on the internals of the compiler, but then, that's the whole point of 'mz.

As for getting anarki back up to date, I discussed that a little in the arc3 release thread. Ultimately I think it's a wasted effort to try and port the entire thing. Instead I think the best idea is probably to port the best features from anarki (the help system, the ability to define how to call tagged objects, various utility functions and macros) separately to arc3, a la CatDancer's "Sharing arc hacks", and publish them as a sort of stripped-down new anarki. I've done some preliminary work here; in particular, I've ported the help subsystem to arc3. I have a repository at http://github.com/rntz/arc, though I haven't uploaded the help changes to it just yet and it's very badly organized. I think I'll probably make a more concerted effort once arc3.tar has been finalized.

-----

1 point by shader 5675 days ago | link

Right, I must have been thinking about globals, which need to be prefixed with "__". A bit uglier than a comma, but more effective ;)

Would it be possble to override 'unquote to temporarily mean

  [sym (string "__" _)]

? That way the syms would be transformed to the same way they are in mzscheme.

Ok, I guess I'll wait for upgrading Anarki until the final version of arc3.

-----

1 point by CatDancer 5674 days ago | link

Don't forget that when you write a macro

  (mac foo args ...)

You are given the full literal contents of what appears in (foo ...) in args. You can do anything you want to with it, including searching for (unquote x) forms and replacing them with something else. (The reader expands ,x into (unquote x), so that's what you'll see when you look at args).

-----

1 point by shader 5674 days ago | link

I've tried doing it several different ways, but so far nothing has worked.

I guess I'll just have to use __, or go with $, unless you can suggest a method to get that to work.

-----

1 point by CatDancer 5674 days ago | link

The first step is to start with an example:

  (= a 5)
  (= b 10)
  (mz (+ ,a (* ,b 2))

Now, what would you like this macro expression to expand into? Something like this?

  (with (x1 a x2 b)
    (ac-scheme (+ x1 (* x2 2))))

(yeah, this morning I'm trying out the name "ac-scheme" for "ac-tunnel" :-)

Now, let's say that expansion is what you want. So your next step is to write a program that takes

  (+ (unquote a) (* (unquote b) 2))

and turns it into

  (with (x1 a x2 b) (ac-scheme (+ x1 (* x2 2))))

this part has nothing to do with macros; you're just creating a list from another list.

  arc> (myprogram '(+ ,a (* ,b 2)))
  (with (x1 a x2 b) (ac-scheme (+ x1 (* x2 2))))

note that's a regular quote ' character there, not a backquote `. We're just feeding a regular old list to myprogram, and it's giving us a regular old list back.

OK! Got that program written? Now the macro is easy:

  (mac mz (x)
    (myprogram x))

-----

1 point by shader 5674 days ago | link

I'm aware of the process for making macros, however my attempts at writing the transformation itself have been rather unsuccessful.

What I originally tried was transforming

  (mz (+ 3 ,a))

into

  (mz (+ 3 __a))

But I couldn't get it to work. The problem is that I seem to need two levels of transformation and evaluation, but macros only give you one.

What is the reason that 'eval is so looked down upon? Isn't that pretty much what macros do themselves? How hard would it be to get an eval that could use the current local scope?

-----

3 points by rntz 5674 days ago | link

    (def transform (e)
      (if atom.e e
          (is car.e 'unquote) (ac-scheme.ac-global-name cadr.e)
          (map transform e)))

    (mac mz body
      `(ac-scheme (begin ,@(map transform body))))

For example:

    arc> (= e 'foo)
    foo
    arc> (mz (symbol->string ,e))
    "foo"

-----

1 point by shader 5674 days ago | link

Nice. Say, that brings up a thought I had earlier.

What if we made that pattern into a function called "maptree". I've needed something like that several times:

  (def maptree (f tree)
    (map (afn (node)
             (if atom.node
                 (f node)
                 (map self node)))
         tree))

It should apply f to each node of the tree, preserving the tree structure. Seems pretty useful to me, if you're going to be doing much transformation of arbitrary list structures.

-----

1 point by shader 5674 days ago | link

In retrospect, it's not much good for transforming the structure, but great for many other things.

For everything else, the other tree functions like trav, treewise, ontree, etc. might be more useful.

How would you make a more general purpose version of maptree that could encompass your transform function? It needs to be able to selectively modify some branches of the tree, and avoid calling map on them. Mine can't do that because it restricts you to atoms. Maybe if it took in two functions, one that operates on atoms, and one on lists, which can then optionally continue the mapping on a subnode. But then what has been gained? You're pretty much writing the whole pattern over again anyway.

Hmm. There seems to be a pattern here, but I can't see how to abstract it out, beyond the somewhat useful but restricted maptree posted above. Ideas?

-----

1 point by rntz 5674 days ago | link

'eval is not particularly looked down on, it's just rather inflexible. Given the way arc works, an 'eval that can use the current local scope is impossible - because arc compiles down into scheme, and an 'eval in arc compiles down to an 'eval in scheme, and an 'eval in scheme cannot use the current local scope. In order to get such an eval, you'd need to rewrite arc as an interpreter.

Edit: Unless mzscheme itself has some special 'eval with that functionality. I didn't think of that. I don't think it does, though.

-----

1 point by shader 5674 days ago | link

I guess I need to study how arc works a little more before I make any bold statements.

Couldn't you just apply the arc compiler/interpreter functions directly to a form? Didn't the parser.arc library do something like that?

-----

1 point by conanite 5673 days ago | link

parser.arc (at least, the parser lib I wrote; there are at least two others in anarki) uses coerce to get corresponding atoms from the token in string form. 'eval would probably have worked too. Otherwise, parser.arc just returns the forms it reads in, only transforming bracket syntax and string interpolations.

-----

1 point by CatDancer 5674 days ago | link

You actually only want one underline:

  arc> (= a 5)
  5
  arc> (ac-scheme _a)
  5

The ac-global-name function will prefix a symbol with an underline for you to create the global symbol name.

As for getting at the lexical scope, with eval or something else, that's up to the Scheme implementation. You'd need to go looking in the MzScheme documentation, or ask on the MzScheme user mailing list, to find out if there's some way to do that. Though why would you need that for this?

-----

1 point by shader 5674 days ago | link

This is what I got, using your mz patch:

  arc> (= a 5)
  5
  arc> (mz (+ 3 _a))
  Error: "reference to undefined identifier: _a"
  arc> (mz (+ 3 __a))
  8

So as far as I can tell, it's two, at least in Anarki on mzscheme 360.

-----

1 point by CatDancer 5674 days ago | link

Oh, I'm running arc3.

So, anyway, when you say you couldn't get "it" to work, what is the "it" that you're trying to do? I thought you were saying you didn't know how to transform "(mz (+ 3 ,a))" into "(mz (+ 3 __a))", but you say you know how to do that, so what is the it that's not working?

-----

1 point by CatDancer 5675 days ago | link

I've never used that feature, do you have an example handy of when auto-quasiquoting is useful?

-----

1 point by shader 5675 days ago | link

I'm working on that. First I have to get my snippet to work.

My goal is to get

  (= a 5)
  (mz (+ 3 ,a))

to transform into

  (ac-tunnel (+ 3 5))

so that you can use arc variables easily inside of a mzscheme call. That's what the original $ does.

-----

1 point by CatDancer 5675 days ago | link

Actually what's funny is that I had no idea that people were using it! I mean, it's not like a web application where I can see how many how any hits it's getting, right? I just throw this code out there and it drifts away into the Internet, never to be see again. Then I suggest naming it something else and people say "no! no! Don't take away my mz!" :-) Oh. I guess some people found it useful! ^__^

-----

1 point by CatDancer 5674 days ago | link

How about "ac-scheme"?

To implement my hacks I've already needed to pull a few pieces of the Arc compiler up into Arc, such as ac-denil and ac-global-name. So the direction this is heading in is to make parts of the Arc compiler available in Arc: ac-denil, ac-global-name, ac-scheme.

This allows us to do various powerful things, of which one is getting at Scheme functionality easily.

You could then write a plain "mz" macro that expanded into "ac-scheme" directly, or, if you wanted, a more sophisticated one that did things like giving easy access to global variables in the expansion.

The "scheme" part of the name is a visible reminder that the code inside is literal Scheme code instead of Arc code.

  (ac-scheme (+ 1 2))

-----