I used to think that, but these days I'm coming around to applying the "Arc philosophy" to libraries: every implementation should contain the minimum amount of code needed to implement just what it does.

So we have one commit in git which is parser.arc; and another commit in git which is parser + table reader.

Now, this is a weird thing to do. What everyone does is write software so that it can be configured and extended without having to change the source. Like everyone else, I've done this myself: just look at my "extend" function for example.

Yet Arc is different. It doesn't try to let you configure it or extend it. It just does what it does in the simplest way. And that simplicity makes it easy to hack Arc to do what you want. While the usual way, of having configurations and hooks and extension methods often fails: whatever I want to do turns out not to be supported by what the original author thought of. So I end up having to hack the source anyway, which is then made difficult ironically by the added complexity of all the configuration and extension code!

So to make the parser easily extensible, rather than adding hooks, instead make sure that you've removed any code duplication.

Are all those functions defined in arc-tokeniser just to keep them out of the top level namespace?

Yes. I've noticed that most arc code is not like this, so maybe arc-tokeniser is really bad style. What's the correct way to deal with namespace clashes? I'd like to avoid having a whole bunch of functions called arc-tokeniser-<something>.

Although upon reflection, it's true that popping these kinds of functions up into the toplevel makes them more readily hackable (by simple substitution, or 'extend-ing). I wonder if there's a way to do that without spawning a crowd of verbosely-named little functions?

We publish a commit which is a merge of the two patches. You can see an example in my arc2.testify-table0+testify-iso0 commit, which is a merge of my arc2.testify-table0 and my arc2.testify-iso0 patches. Here's the original testify from arc2:

  (def testify (x)
    (if (isa x 'fn) x [is _ x]))

   (def testify (x)
     (if (in (type x) 'fn 'table) x [is _ x]))

  (def testify (x)
    (if (isa x 'fn) x [iso _ x]))

    (def testify (x)
  -   (if (isa x 'fn) x [iso _ x]))
   -  (if (in (type x) 'fn 'table) x [is _ x]))
  ++  (if (in (type x) 'fn 'table) x [iso _ x]))

and your "extend" function makes that even easier and safer

Right, I think that functions like "extend" would arise from seeing patterns in code and abstracting them out, making the code more succinct. It was more my thought process I was commenting on, I had thought "patches are hard to deal with so I'll make functions like extend". Now I'm thinking, "what if patches were easy?".

What's the correct way to deal with namespace clashes?

I've been wondering about that. One possibility I've wondered about is to have an abbreviation macro:

  (prefix-abbrev arc-tokeniser- (make-token make-character tokenize ...)

    (def make-token (kind tok start length)
      (list kind tok start (+ start length)))

    ...
  )

This is even more speculative, but I've also wondered if maybe it needn't be your job as a library author to worry about namespace clashes. What if the user of the library said "oh, look, make-token is clashing", and could easily load it with particular symbols renamed...

  (set tokeniser-helpers nil)

  (def arc-tokeniser (char-stream)
    (w/locals tokeniser-helpers
      body-of-arc-tokeniser-here))

Use an alternative def to put function definitions in such a table instead of the global namespace:

  (def-in tokeniser-helpers make-token (kind tok start length) ...)

Is this something like how your prefix-abbrev macro would work?

Yes, but how do you know that your Arc parser functions are only going to be relevant in the code you've written? Perhaps someday I'll be writing my own parser, or something completely different, and I'll find it useful to use one of your functions in a way that you didn't think of!

I suggest trying out writing your code is in the simplest possible way. For example, in your original:

  (def arc-tokeniser (char-stream)
    (withs (make-token     (fn (kind tok start length)
                               (list kind tok start (+ start length)))

  (def make-token (kind tok start length)
    (list kind tok start (+ start length))

And sure, down the road there may be some other library that also defines "make-token". At that point, it will be easy to make a change so that they work together. Perhaps by renaming one or the other, or by doing something more complicated. The advantage of waiting is that then we'll know which functions actually conflict, instead of going to a lot of work now to avoid any possibility of future conflict, the majority of which may never happen.