Arc Forum | Access Lexical Variables

Arc Forum

Access Lexical Variables

6 points by fallintothis 5132 days ago | 2 comments

I don't think I've seen anyone "solve" this yet, and it seems like one of those things that gets brought up a lot (e.g., when I had troubles doing it before: http://arclanguage.org/item?id=10504), but I somehow saw how this might be done after seeing waterhouse's aps macro again (http://arclanguage.org/item?id=12881).

It's pretty easy to get access to the lexical variables. You just need to add a new special form (similar to if, fn, etc.) that compiles to an expression that calculates the environment -- almost like a macro. There are some snafus with ac being so unwieldy, and I'm not sure I got them all, but here's the patch I came up with:

  $ diff -u old-ac.scm new-ac.scm
  --- old-ac.scm  2010-12-03 15:55:02.000000000 -0800
  +++ new-ac.scm  2010-12-03 15:57:56.000000000 -0800
  @@ -26,6 +26,7 @@
           ((eq? (xcar s) 'if) (ac-if (cdr s) env))
           ((eq? (xcar s) 'fn) (ac-fn (cadr s) (cddr s) env))
           ((eq? (xcar s) 'assign) (ac-set (cdr s) env))
  +        ((eq? (xcar s) 'locals) (ac-locals env (cdr s)))
           ; the next three clauses could be removed without changing semantics
           ; ... except that they work for macros (so prob should do this for
           ; every elt of s, not just the car)
  @@ -1485,5 +1486,20 @@
                                        (cons (car cs) (unesc (cdr cs))))))))
                     (unesc (string->list s)))))
     
  +; remove stupid cruft from env
  +
  +(define (clean-env env)
  +  (keep (lambda (v) (and (not (pair? v))      ; due to ac-dbname
  +                         (not (eq? v 'nil)))) ; due to, e.g., (def foo () ...)
  +        env))
  +
  +(define (ac-locals env args)
  +  (unless (null? args) (err "Too many arguments to (locals)"))
  +  (list 'fill-table
  +        '(make-hash-table 'equal)
  +        (cons 'list
  +              (map (lambda (v) (list 'list (list 'quote v) v))
  +                   (clean-env env)))))
  +
   )

In action:

  arc> (def scope (var env)
         (if (env var)
             (prn var " is in scope")
             (prn var " is out of scope")))
  #<procedure: scope>
  arc> (let x (locals)
         (pr "Outside: ") (scope 'x x)
         (pr "Inside:  ") (scope 'x (locals))
         nil)
  Outside: x is out of scope
  Inside:  x is in scope
  nil

Notice that it's kind of an automagic macroexpansion:

  arc> (def foo (x) (locals)) ; this is like (def foo (x) (obj x x))
  #<procedure: foo>
  arc> (foo 5)
  #hash((x . 5))
  arc> (let x 10 (foo 5))
  #hash((x . 5)) ; the x comes from inside the scope of foo

  arc> (def bar () (locals)) ; similarly, this is like (def bar () (obj))
  #<procedure: bar>
  arc> (bar)
  #hash()
  arc> (let x 10 (bar))
  #hash() ; no locals at the time bar was defined

  arc> (let quux 10             ; this will be a locally-scoped variable ...
         (def baz () (locals))) ; so this is like (def baz () (obj quux quux))
  #<procedure: baz>
  arc> (baz)
  #hash((quux . 10)) ; quux was in scope at the time baz was defined
  arc> (let x 10 (baz))
  #hash((quux . 10)) ; x was NOT in scope at the time baz was defined

Note that this doesn't give us a full-on lexical eval. But it's still useful for code instrumentation (i.e., some sort of code-walking macro or other could insert bits that look over (locals)). If we're careful, we can use it to good effect. E.g.,

  (mac lexbound (x)
    `(if ((locals) ,x)
         t
         (bound ,x)))

Notice that if we used def, (locals) would be "contaminated" with x in the body. We don't want that, so we sneak the call to (locals) in the macroexpansion, giving us the following behavior.

  arc> (load "macdebug.arc") ; see http://arclanguage.org/item?id=11806
  Expression:

    (let x 10 (lexbound 'x))

  Macro Expansion:

      (let x 10 (lexbound 'x))

  ==> (with (x 10) (lexbound 'x))

  Expression:

    (with (x 10) (lexbound 'x))

  Macro Expansion:

      (with (x 10) (lexbound 'x))

  ==> ((fn (x) (lexbound 'x)) 10)

  Expression:

    ((fn (x) (lexbound 'x)) 10)

  Macro Expansion:

      (lexbound 'x)

  ==> (if ((locals) 'x) t (bound 'x))

  Expression:

    ((fn (x)
       (if ((locals) 'x) t (bound 'x)))
     10)

  nil
  arc> (let x 10 (lexbound 'x))
  t

but

  arc> (lexbound 'x)
  nil
  arc> (= x 100)
  100
  arc> (lexbound 'x)
  t

We could also add the complementary function to ac.scm, which is easier since globals can be accessed without the env parameter. (This is really just a Scheme version of what waterhouse was doing.)

  (xdef globals
        (lambda ()
          (fill-table (make-hash-table 'equal)
            (map (lambda (v) (list (string->symbol
                                     (substring (symbol->string v) 1))
                                   (namespace-variable-value v)))
                 (keep (lambda (s)
                         (and (eqv? (string-ref (symbol->string s) 0) #\_)
                              (not (eqv? s '_))))
                   (namespace-mapped-symbols))))))

Then we can get access to the full environment with a macro.

  (mac env ()
    (w/uniq (lex bindings)
      `(let ,lex (locals)         ; want to bind (locals) first, so next let-
                                  ; expr for (globals) won't "contaminate" them
         (let ,bindings (globals)
           (each (var val) ,lex (= (,bindings var) val))
           ,bindings))))

  arc> (= glob* 100)
  100
  arc> ((env) 'glob*)
  100
  arc> (let glob* 500 ((globals) 'glob*))
  100
  arc> (let glob* 500 ((locals) 'glob*))
  500
  arc> (let glob* 500 ((env) 'glob*))
  500

  arc> ((env) 'local)
  nil
  arc> (let local 5 ((locals) 'local))
  5
  arc> (let local 5 ((globals) 'local))
  nil
  arc> (let local 5 ((env) 'local))
  5

Strictly, you could make (locals) look like a variable instead (just change the clause in ac), but I decided against that because lexical bindings aren't static, like a variable might imply.

2 points by shader 5128 days ago | link

This looks promising.

Do you think it could be used to implement alphabetic br fns like I suggested at http://www.arclanguage.com/item?id=8997 or does the original problem still apply? (nested bindings are not detected properly at mac expansion time)

-----

2 points by fallintothis 5127 days ago | link

[Insert caveats about how crazy the idea is, yadda yadda, stuff rntz already pointed out, blah blah, who cares?]

There are too many edge-cases for the idea to work well once implemented (as I said in the giant rant above :P), but you can get the basic behavior down, which fixes the issues you were finding early on in the thread. I gather that's what you were asking. So, for the sake of fun:

  (mac lexbound (x)
    `(if ((locals) ,x)
         t
         (bound ,x)))

  ; ac won't care about shadowing a special-form, but we don't want to treat it
  ; like an arg to the br-fn below

  (def keyword (name)
    (or (nonop name)
        (in name 'nil 't 'if 'fn 'assign 'locals)))

  (mac make-br-fn (body)
    (w/uniq args
      `(fn ,args
         (with ,(mappend
                  (fn (v) `(,v (if (lexbound ',v) ,v
                                   (no ,args)     (err "Too few args to br-fn")
                                                  (pop ,args))))
                  (sort < (keep (fn (_) (and (isa _ 'sym) (~keyword _)))
                                (dedup (flat body)))))
           (when ,args
             (err "Too many args to br-fn"))
           ,body))))

  arc> ((make-br-fn (+ x y z)) 1 2 3)
  6
  arc> ((make-br-fn (+ x y z)) 1 2 3 4)
  Error: "Too many args to br-fn"
  arc> ((make-br-fn (+ x y z)) 1 2)
  Error: "Too few args to br-fn"
  arc> (let x 3 ((make-br-fn (+ x y z)) 1 2))
  6
  arc> (let + (fn (x y z) (prf "x = #x, y = #y, z = #z") (prn))
         ((make-br-fn (+ x y z)) 1 2 3))
  x = 1, y = 2, z = 3
  nil
  arc> ((make-br-fn (do (prs x z y) (prn))) 1 2 3) ; bound in alphabetic order
  1 3 2
  nil

You have to do the work within the expansion itself (at run-time), since that's where you need to check whether variables are lexically bound. I.e., if you do it in the macro, you'll just get the locals from macroexpansion time. This essentially works the same way the following do.

  arc> (let x (if (lexbound 'x) x 5)
         (+ x 5))
  10

since the call to locals winds up happening outside the outside of the fn that let generates:

  arc> (macsteps '(let x (if (lexbound 'x) x 5) (+ x 5)))
  Expression:

    (let x (if (lexbound 'x) x 5)
      (+ x 5))

  Macro Expansion:

      (let x (if (lexbound 'x) x 5)
        (+ x 5))

  ==> (with (x (if (lexbound 'x) x 5))
        (+ x 5))

  Expression:

    (with (x (if (lexbound 'x) x 5))
      (+ x 5))

  Macro Expansion:

      (with (x (if (lexbound 'x) x 5))
        (+ x 5))

  ==> ((fn (x) (+ x 5)) (if (lexbound 'x) x 5))

  Expression:

    ((fn (x) (+ x 5)) (if (lexbound 'x) x 5))

  Macro Expansion:

      (lexbound 'x)

  ==> (if ((locals) 'x) t (bound 'x))

  Expression:

    ((fn (x) (+ x 5)) (if (if ((locals) 'x) t (bound 'x)) x 5))

  nil

Whereas

  arc> (let x 10
         (let x (if (lexbound 'x) x 5)
           (+ x 5)))
  15

since the call to locals winds up happening inside the fn that the outermost let generates:

  arc> (macsteps '(let x 10 (let x (if (lexbound 'x) x 5) (+ x 5))))
  Expression:

    (let x 10
      (let x (if (lexbound 'x) x 5)
        (+ x 5)))

  Macro Expansion:

      (let x 10
        (let x (if (lexbound 'x) x 5)
          (+ x 5)))

  ==> (with (x 10)
        (let x (if (lexbound 'x) x 5)
          (+ x 5)))

  Expression:

    (with (x 10)
      (let x (if (lexbound 'x) x 5)
        (+ x 5)))

  Macro Expansion:

      (with (x 10)
        (let x (if (lexbound 'x) x 5)
          (+ x 5)))

  ==> ((fn (x)
         (let x (if (lexbound 'x) x 5)
           (+ x 5)))
       10)

  Expression:

    ((fn (x)
       (let x (if (lexbound 'x) x 5)
         (+ x 5)))
     10)

  Macro Expansion:

      (let x (if (lexbound 'x) x 5)
        (+ x 5))

  ==> (with (x (if (lexbound 'x) x 5))
        (+ x 5))

  Expression:

    ((fn (x)
       (with (x (if (lexbound 'x) x 5))
         (+ x 5)))
     10)

  Macro Expansion:

      (with (x (if (lexbound 'x) x 5))
        (+ x 5))

  ==> ((fn (x) (+ x 5)) (if (lexbound 'x) x 5))

  Expression:

    ((fn (x)
       ((fn (x) (+ x 5)) (if (lexbound 'x) x 5)))
     10)

  Macro Expansion:

      (lexbound 'x)

  ==> (if ((locals) 'x) t (bound 'x))

  Expression:

    ((fn (x)
       ((fn (x) (+ x 5)) (if (if ((locals) 'x) t (bound 'x)) x 5)))
     10)

  nil

That said, there are plenty of bugs.

- The most obvious is that you'd need a full-blown walker of the body argument to see where you really need to make new variables. E.g.,

  (make-br-fn (let x 5
                (+ x 10)))

macroexpands to

  (fn gs2296
    (with (+   (if (lexbound '+) +
                   (no gs2296)   (err "Too few args to br-fn")
                                 (pop gs2296))
           let (if (lexbound 'let) let
                 (no gs2296)       (err "Too few args to br-fn")
                                   (pop gs2296))
           x   (if (lexbound 'x) x
                   (no gs2296)   (err "Too few args to br-fn")
                                 (pop gs2296)))
      (when gs2296
        (err "Too many args to br-fn"))
      (let x 5
        (+ x 10))))

which winds up expecting an x argument, because it wasn't bound outside of the br-fn, even though it's just a local binding introduced in the body by the let.

  arc> ((make-br-fn (let x 5 (+ x 10))) 'ignored-x)
  15

- make-br-fn should probably start with (zap macex-all body), since macros may introduce arbitrary unbound variables. E.g.,

  arc> (macex1 '(prf "x = #x"))        ; introduces an unbound reference to x
  (let gs396 (list) (pr "x = " x))
  arc> ((make-br-fn (prf "x = #x")) 1) ; prf not expanded
  Error: "Too many args to br-fn"

Couple this with the above bug, and you have even more problems, since in the above expansion of prf, you can't tell that gs396 is a bound local in body without doing a code-walk.

- You'll still have problems from vanilla Arc's lexically-shadowed-macros bug in some cases, but they're the ones you'd expect from knowing about the bug to begin with. That is, the following works because do is not in a function position.

  arc> (let do 10 ((make-br-fn do)))
  10

But, instead of generating an error (applying 10 to 5), the following macroexpands do.

  arc> (let do 10 ((make-br-fn (do 5))))
  5

This thread's made local variables in Arc interesting in ways I hadn't thought of before. Macros introduce odd contours to lexical environments that don't exist in, say, Python (which still has a locals() function). Basically, any use of (locals) should come with a big, fat warning: "magic happens here".

-----