"When one writes a new tool it's usually (hopefully) to satisfy a pretty frustrating need."

Building a programming language is a pretty natural itch. For a certain kind of programmer, at least. I'll give you some reasoning and rationalizations for wart, without any certainty that I can separate the two.

I've been vaguely aware of lisp almost since I was exposed to computers, back in 1996. But in India algol-style languages reigned. Basic, fortran, cobol, pascal, C/C++. A little behind the times. I took a lisp course when I moved to the US back in '99 and I could program little things[1] in common lisp after that. But I'd still stare at things like the recursive definition for reverse and wonder precisely what it was that made it so different from what I'd write in C. I periodically go back to that question, but more on that further down.

After programming mostly in C for 10 years, burning out, recovering, the usual coming-of-age stuff, I found myself in the bay area doing rails and the startup scene for a couple of years ('07-'10). I'd been interested in social software and recommendation systems since Reddit got started. I even signed an NDA with the Reddit guys to get at their code and try to build a recommendation system for them. But the lawyers took too damn long and I'd lost interest by the time they sent it to me (sometime '07). But around '08 I started thinking about recommendation systems again. Arc was out by this point[2], and it acted as a great gateway drug. Here, finally, was a lisp that helped filter out all the cruft and let me focus on how lisp was different from C, and that was ergonomic enough that I had to work to stop programming in it. At some point I looked at the logs generated by news.arc, and noticed that they were just s-exprs that could be read back in if I so desired. Hmm, crazy idea: a recommendation system that can see the recent actions in a session to inform recommendation decisions. The easiest way to do that would be to stick with lisp.

So I left rails behind and started a fresh (and my final, so far) startup in arc, to build a feedreader that didn't require understanding RSS. I never did do much with the idea of adapting to the current session, because I could never get to a point where I knew enough about a user to make any intelligent decisions. Providing a bootstrap experience that kept them coming back long enough turned out to be an eternally unsolved problem. I found out the importance of UX and how much I sucked at it. A year later the project was dead and I was working at Google. But the interest in lisp stuck, and I continued doing stuff with it, noodling on a series of little questions about its design, why it did things this way and not that. Sometimes I learned the hard way something people knew back in the '60s. Occasionally I discovered something new[3].

But I was still dissatisfied. One of the things that had bothered me all through this period was webservers and memory leaks. No matter what platform I used -- rails, python, racket, common lisp -- it seemed any non-trivial website would periodically run out of memory and die. Everywhere I looked, people dealt with this in a brutal way: by periodically restarting their servers and clearing the slate. Ugly, and it seemed to point at an endemic problem: languages spent tons of effort tweaking their garbage collectors, but programmers would mess up in some subtle way and accidentally prevent stale objects from being collected. Often this was because of interactions between multiple projects. Often there were no tools for the low-level debugging required. Or if there was, you needed to learn too much about the high-level internals of the stack you were relying on.

There were other dissatisfactions. Common lisp was a lisp-2, and its APIs were ancient and baroque[4]. I could sense that I could get used to them, but the mind rebelled at polluting my brain with all those warts. Racket was more modern and a lisp-1, but it gave up a lot of the dynamism of lisp with its phase-separation rules, hygienic macros and module system. Both had oodles of documentation, but common lisp's docs were poor and examples often wouldn't run. It had a 'standard' constraining creativity and unionizing bad APIs with lots of detailed rules, but the standard hadn't been updated in years so you were often doing stuff out of its ambit anyway (I still don't understand its condition system). It was the worst of both worlds. Racket has a much nicer documentation system, and yet it was too overwhelming for reasons I still struggle to pinpoint[5]. At some point, it seemed, the documentation was complex enough to ask, why can't I just deal with the codebase directly rather than some interface to it? How can I fork racket and throw out hygienic macros until I gain the wisdom of their need?

So eventually, after leaving arc, after trying to build arc out of common lisp macros[6], after some time agonizing over just switching to Factor[7], I bit the bullet, threw out all the dependencies, and returned to my roots in C. My goals were to create a dynamic lisp-1 with non-hygienic macros where I could define code in any order[8], and to do so with as little code as possible, in such a way that others could come after me and query the design space I had explored, asking what-if questions: what if I want to turn off this feature? Why do we implement this like so? The goal was never to build a 'real' language the way the world thinks about it. I wanted to see how far I could push a design without any black-box interfaces, where programmers had to understand the entire stack, and therefore where the entire stack had to be simple enough to fit in a single head[9], one that encouraged the combination of high- and low-level fluency necessary to track down problems like memory leaks, one where the documentation was intertwined inextricably with the implementation, so it couldn't go out of date and it could be queried interactively[10]. One that didn't spend thousands of lines of code trying to 'optimize' arbitrary code, but provided programmers the tools to measure their applications and attack bottlenecks for themselves without compromising readability, the way 'macros' provide a la carte mechanisms for reducing our own boilerplate. That's The Lisp Way.

What I have so far is a far cry from this goal. Sometimes I worry that I'm just playing in the shallows where it's fun[11] and avoiding the scary abyss. Programming languages are fun, but if I can stay disciplined wart will be more than a language. And nobody will have to ever rediscover why something is built just like so.

[1] From 2004 or so: http://akkartik.name/lisp.html

[2] After years, it felt like, at the time.

[3] http://arclanguage.org/item?id=16378; arguably this little idea is the coolest thing I've done so far.

[4] http://arclanguage.org/item?id=13664, footnote 3.

[5] Many of my initial annoyances were bugs in arc at the time (http://arclanguage.org/item?id=13616), or a result of the legacy mzscheme language (since fixed by Arc/Nu). Would I have embarked on this if the timing had been just a little different?

[6] https://github.com/akkartik/wart/tree/sbcl

[7] http://factorcode.org; awesome, awesome language. But I just liked the readability of keyword args too much to give them up.

[8] http://arclanguage.org/item?id=15587, footnote 1.

[9] VPRI is working toward a similar goal: a computer stack in orders of magnitude less code (http://www.vpri.org/html/work/ifnct.htm).

[10] I don't want to require new programmers to understand every last line of code before they can use a program. I want the program to reward curiosity, so that newcomers are able to drill down into the right bits when they run into problems, to answer questions for themselves without needing to find an expert to talk to (http://alistair.cockburn.us/ASD+book+extract%3A+%22Naur,+Ehn...), and to iteratively try tweaking the code with instant feedback about regressions they introduce.

Basically, before Arc/Nu, there was ar, which was created by awwx. It was a rewrite of Arc in Racket, where the compiler was intentionally exposed to Arc. This meant you could not only access the compiler from Arc, but could actually change it!

So rather than writing the compiler in Racket, you'd write the minimum necessary to get things going, then write the rest of the compiler in Arc. This naturally made the compiler very easy to hack for Arc users.

I really liked that, but I also disliked certain things in ar. One of those is that all the compiler stuff is prefixed with "racket-", which is really verbose.

Another dissatisfaction is that although awwx did add in a module system for Arc, it was very... uh... basically you could create multiple Racket namespaces, which are completely separate from each other, and load Arc into each one.

The problem is that because they're separate, it's really difficult for them to communicate with each other. Which kinda defeats the point.

Another problem is that you had to reload Arc every time you created a new namespace, which meant ~1-2 second pauses every single time. Which is pretty ridiculous.

I felt I could do better, so I made a fork of ar and hacked away on it. But having to use the "racket-" prefix really bothered me, so I experimented with ways to get rid of it.

Eventually I decided to start over with pg's vanilla Arc 3.1, rather than ar. I fixed it so it would work in Racket rather than mzscheme, and then I did significant refactoring, making things faster, using new techniques I had discovered, etc.

I also added in a module system which was exactly the same as ar's module system, except that you could make a module that inherits from another module. This solved the "have to reload Arc all the time" problem, and also made it a bit easier to communicate between modules.

The end result was Arc/Nu. It had all the benefits of ar, all the benefits of Arc 3.1, and none of the downsides of either one.

In the end, though, Arc/Nu's module system was terrible. Yes, you could now make modules that inherit from other modules, but I kept running into all kinds of problems, both theoretical and practical. But it was the best module system I knew of.

I wrote some programs in Arc/Nu, and after many months passed, I discovered hyper-static scope.

Before I continue with this story, I'll need to explain Nulan. Nulan is a Lisp I created from scratch. It has hyper-static scope, hygienic macros, and it compiles to JavaScript and can thus be used in web browsers.

I decided to make Nulan because I wanted to experiment with cool new things like hyper-static scope, but I couldn't do that in Arc/Nu because that would break backwards compatibility with Arc 3.1. I also wanted a language for web browsers that would be better than JavaScript.

While implementing hyper-static scope in Nulan, I discovered how amazing boxes are. Seriously, they're great. Using them, you can create a module system which is easily thousands of times better than any other module system I had ever seen.

Dido posted a topic about a module system he was designing for Arc, and I replied with this: http://arclanguage.org/item?id=17408

After that, I realized that I had to change Arc/Nu to use boxes. And so I did. The module system in Arc/Nu is now amazing, and no longer based upon Racket namespaces.

I've also wanted to have a language like Tcl or Lua that one could easily embed as a scripting language in a larger project. This is one of the things that Arcueid is designed to do: just link a single library and have an interpreter available with a relatively simple API. Arcueid is also be designed to be easily extensible via a simple foreign function interface because I don't want to use it for just programming web applications, although that is something that I would doubtless use it for a lot as well.

Writing in C (or something that compiles to C) is also probably the only way to get an Arc faster than the implementations we have today. Of course my present implementation is still somewhat slower than Arc on Racket, but then again the goal for now is to build a foundation upon which further optimisation can later be done. As always, it's better to get it to work right first, and then make it run quickly later... Eventually Arcueid will have a JIT compiler of sorts but that's a little further down the roadmap.

As for Arc, it seems like it's much better designed than any other Lisp dialect out there. The parentheses are still there, but they are used more sparingly than Scheme, which seems out to irritate you by putting extra parentheses for something so basic as a let binding when you could easily do with less and have more comprehensible code. This fact was illustrated to me about a year ago when rocketnia gave me versions of the Hanson-Lamping algorithm for dynamic-bind in Scheme and in Arc here:

http://arclanguage.org/item?id=15536

The contrast was rather startling to me. I understood the algorithm immediately on seeing the Arc code.

Another thing about other Lisps that is really irritating to me are the overly long function names like multiple-value-bind and invoke-restart-interactively and stuff like that. They make a program look rather cluttered. Arc has for the most part avoided excessively long identifiers in the core and that has been good.

I don't think a third-party tool should count as language support, unless (and inasmuch as) the people maintaining the language regularly go out of their way to avoid breaking that tool.[1][2] This is where I draw the line, because of how it affects backwards compatibility.

Everyday language users tend to think of language updates as backwards-compatible as long as their existing code has the same behavior. But tools like eval(), JavaScript minifiers, and Ruby2Ruby place the bar higher, since their input is arbitrary language code, and it might be written in a newer language version than the tool itself. Incidentally, even a program that calls these tools will continue to work after a language update, as long as the input of these calls is all generated programmatically, rather than directly taken from the program's input.

[1] Well, I guess the term "third-party" might not apply in that case!

  | Nulan seems like an that also compiles to Javascript
  | Rainbow is a Java implementation of Arc
  | RainbowJS seems like a port of Rainbow to Javascript