The following properties are desirable:

- Convenience (auto-imports, imports not needing to specify the variables that are used, etc.)

- Correctness (ensuring that your code uses the variables you expect it to use, and that it doesn't easily break by changing some random unrelated file)

- Performance (only loading the functions/variables that you actually use)

Unfortunately, it is probably impossible to get all three properties at the same time.

By making code more convenient (e.g. implicit imports), you make it harder to reason about code ("I'm using the variable foo, but where does it come from?"), and you make it more likely for there to be variable conflicts (two different files defining a variable with the same name).

You also make it more likely for your program to break when upgrading a library to a new version, or even just when making a change to a seemingly unrelated file.

And if there is a variable conflict (which will happen), then you need some way of disambiguating, which generally requires you to specify which file should take precedence. So in the situation where there are variable conflicts, you lose all of the convenience benefits for that variable.

(And if you choose to not disambiguate, and instead ignore the variable conflicts, then you lose correctness, so that only works for very small projects)

Convenience also conflicts with performance. The language doesn't know which variables are defined in which file, so if you use the "foo" variable, it has to search through all of the files in order to determine which file contains the "foo" variable.

That also means your language needs to define which directories it will automatically import files from (you don't want it searching through your entire harddrive!)

Various languages (including Python[1] and Node.js[2]) have defined their own rules for determining which folders will be searched through for files, so this is a solvable problem.

But unlike Python and Node.js, implicit imports requires it to also search through all of the files inside of those folders... this is obviously an expensive operation.

You could make it work. If your language allows you to pre-compile a binary, then it only needs to lookup the files once (at compile-time). And with tree shaking, the final binary would only contain the variables that you actually use.

You would lose some correctness, and you would lose compile-time performance, but some people would consider the gains in convenience to be worth it.

Arc, however, does not have a separate compile-time, so it would need to do the folder/file lookup every single time you run any Arc program. I think the performance loss would be too great for any medium or large programs.

----

By the way, there's another interesting issue... mutually recursive functions:

  (def my-even (a)
    (if (is a 0)
      t
      (my-odd (- a 1))))

  (def my-odd (a)
    (if (is a 0)
      nil
      (my-even (- a 1))))

If there is a "my-odd" variable defined in another file, then the "my-even" function will call that variable, rather than the "my-odd" variable defined in the current file.

There are various ways to fix this problem, probably the easiest of which is to define the "my-odd" variable before it is used:

  (= my-odd nil)

  (def my-even (a)
    (if (is a 0)
      t
      (my-odd (- a 1))))

  (def my-odd (a)
    (if (is a 0)
      nil
      (my-even (- a 1))))

So I think an auto-import system could work, but your language has to be designed around it. You can't just add an auto-import system to an existing language, because of these (and other) issues.

----

* [1]: https://docs.python.org/3/glossary.html#term-import-path

http://unisonweb.org/2015-05-07/about.html#post-start

In Unison, you do not need to import code. That's because every variable is actually the hash of its definition, and when you use that variable in your code, it automatically "imports" the hash. That also means that your code contains only the variables that you actually use.

You do still lose some correctness, though. As an example, if you have a function and you change the definition of that function, it will create a new hash (for the new definition), but the old hash is still being used in your code.

Which means that you have both the old version of the function and the new version of the function existing at the same time. So you need to manually update all of your code to use the new hash:

http://unisonweb.org/2015-06-12/editing.html

This problem hasn't been solved yet, but there's been discussions about providing some tools that can automate certain tasks (such as replacing an old hash with a new hash, in your entire code base).

Assuming the above problem is solved, it may actually be possible to have convenience, correctness, and performance with an auto-import system.

Emacs's autoloading is just about making startup faster. Autoloads are pieces of code that say "There's a function named `#'foo` in file `bar.el`". Yes, you don't load bar.el, but you do have to load `bar-autoloads.el`, which is the file that contains the autoload pointers. I'm not sure it's a win, bureaucratically. There's some documentat about Emacs autoloads here: https://www.gnu.org/software/emacs/manual/html_node/eintr/Au... .

There's a difference between included libraries and non-included libraries. You make this point pretty clearly in your post, but I saw "Emacs" and jumped to that sentence. Right now Arc loads all included libraries. Is there a particular reason you want to not do that? It seems like the main advantage is startup speed, in which case I wonder if we could just profile^1 and optimize Arc, and get better results from the same amount of work that it would take to write some sort of autoload system.

[1] pg's talked several times about profilers being vastly important^2, 3. It's a shame there isn't one in Arc.

[2] From http://paulgraham.com/arcll1.html,

> From very early, maybe from the begining, the Scheme spec said that conforming implementations must do tail call elimination. The first time I read this, I thought "wait, can you require this in a spec?" Arc will see this increase, and raise it by some standards for profiling.

> The way to get fast code in Arc will be to profile it and then add declarations that improve efficiency where needed.

[3] From http://www.paulgraham.com/langdes.html,

On the one hand, I don't want my .arc files filled with boilerplate code to load the libraries I want; on the other, I don't want a huge number of functions to be loaded that I won't actually use.

I feel that autoloading is the perfect compromise: granted, the 'autoload pointers' will have to be loaded (by default and automatically -- otherwise that's just more boilerplate) -- but this yields far less overhead than loading the actual functions.

For big programs, pg's dead right: far more of the speed will come from optimising the program -- however, for small, sysadmin-type programs, where your program is so small that your algorithm is already optimal, but the actual problem is quite large (e.g. a gargantuan directory structure), the speed comes from a tight language. (And you'll write far more of these microprograms in your lifetime than big, optimisable programs.)

Please don't get me wrong: I agree with you wholeheartedly that there should be a profiler in Arc, and that this should be used to optimise Arc --

What's the concern here? I don't really care how many functions are loaded, even if I don't use them, because it doesn't affect me.

Now, startup time? That effects me. But I don't know how much of the startup time is because of loading more functions.

Alternately, name collisions? I don't see that as a giant issue, and isn't solved by autoloading anyway.

> And I don't think it needs to be an either-or.

I agree. I'm just not all that sure what autoloading gets us. I'm not against it, per se, I just think it's something moderately complicated that might not help us get what we want.

For some reason (and believe me, it seems stunningly obvious to me now), I was under the impression that loading a load of functions into memory would somehow not only affect startup, but also the speed of programs after startup. (ik, ik) I think I've sidetracked myself from my original goal of discussing ways of getting rid of typical '#include'-esque boilerplate.

Though my argument about already-optimal small programs (which have a large workload) might apply to compiler optimisations, it has little to do with autoloading.

I think autoloading would become useful if startup time were pushed up to anything more than a second -- after that, I think most people (that is, I) get quite angry!

Again: thanks for clearing this up for me.

It might! I wouldn't heavily bet on it, but I don't actually know. We certainly could^1 test it! We probably should! But if it doesn't change it much, I think the autoloading isn't worth the complexity cost.

> I think I've sidetracked myself from my original goal of discussing ways of getting rid of typical '#include'-esque boilerplate.

Yeah, that happens. They're certainly both worthwhile conversations.

> I think autoloading would become useful if startup time were pushed up to anything more than a second -- after that, I think most people (that is, I) get quite angry!

Yeah. Certainly right now I wouldn't write something small that I want to use from the command line in Arc (e.g., a Unix utility), because of the runtime. It would be nice to be slower.

> Moving on from autoloading: how are name collisions dealt with in Arc? The only thing I can think of is the old elisp "prefix all yer functions with the package name" ideology. (So the functions of the string library would be called `string-foo`, `string-bar-baz-qux`, etc.) This'd be important, given that (one way or another) ALL THE FUNKSHUNZ are loaded.

Defining a function with the same name as an old one overrides the old one's behavior. So yeah, you either hope nothing collides or you prefix it somehow^2. This is less of an issue in Arc than it is in Emacs both because fewer functions come in the box, and because fewer libraries are pulled in. But certainly collisions can/will happen.

[1] The obvious way I think of is by removing a bunch of unused things from various files that make up Arc, and then see if either startup time, or runtime of a sample program changes.

Just to add to this, and at the risk of boring the veterans here to death by repeating myself :)

Reading between the lines of the implementation, I think the Arc way of dealing with the possibility of collisions is to not worry about them until they happen. When they happen Arc will show you a warning:

  *** redefining foo

This makes a lot more sense than adding prefixes because the names chosen are in the context of the application being built rather than some one-size-fits-all name chosen by a library. That makes them likely to be better names in the context of the application.

Avoiding prefixes also keeps most names from becoming unnecessarily long when they don't need to be.

I'm talking about "a namespace mechanism" here, without actually knowing how one in Arc would work.

Just thinking about it for a moment, I think you'd set the prefix

  (= namespace-prefix 'awesome-package)

  (def /awesome-function (...)
    (...))

  (w/namespace awesome-package
    ...
    (def /awesome-function (...)
      (...))
    ...)

In the end, you have to have some way of avoiding conflicts -- and all of these boil down to tacking something on the front (math.sqrt in python, for instance); I (personally) would rather have a better way of doing this than manually typing out `awesome-package` at the start of every function/macro/variable/whatever.

I think, given that Arc is supposed to be "a language for good programmers" [1], that it's silly to impose restrictions like withholding a namespacing mechanism to encourage a certain type of programming.

Though I like keeping codebases minimal, I also like freedom.

I try not to use words like 'freedom' in these discussions because navigating by them is a subtle business. While having a feature can sometimes provide freedom, in general I tend to assume that features cost freedom. Like the saying that your possessions own you. A namespace mechanism is more code to write, more places for bugs to be hiding in, more errors for the user to run into, more places for one programmer to hide things where they can trip another programmer up. All these issues are places for degrees of freedom to be used up rather than created.

So rather than importing a single big library in your application, you would instead import a lot of micro libraries (each of which is versioned separately, and might have their own dependencies).

What I would like to happen is that my application only contains dependencies it really needs, and that each dependency includes no superfluous/dead interfaces or code. Under these circumstances I would like to live in a world where I can go in and modify the libraries to have different names, with the difference in names making sense in the context of the application. Then I would bundle the application with all its libraries included.

Of course this doesn't scale to large libraries, because managing a fork today involves an amount of work that ranges from non-trivial to intractable. But this would be my ideal.

Past writings on this subject: http://akkartik.name/post/libraries; http://akkartik.name/post/libraries2. My current project which tries to make fork-management tractable: https://github.com/akkartik/mu

  Unrecorded reasoning, existing mainly in our minds.
  -->
  Codebases dedicated to proofs or tests.
  -->
  Proofs or tests located in the codebase they apply to.
  -->
  A type/contract declaring a module interface.
  -->
  A type/contract annotation for a function definition.
  -->
  A type/contract annotation for an individual expression.
  -->
  A type/contract annotation for an individual built-in operator, but at
  this point it becomes implicit in the operator itself, and we just
  have structured programming, enjoying properties by construction.

  Unrecorded how-to knowledge, existing mainly in our minds.
  -->
  Codebases or how-to guides dedicated to curated builds (e.g. distros).
  -->
  Build scripts and docs located in the codebase they apply to.
  -->
  Macroexpansion-time glue code, importing compiler extensions by name.
  -->
  Load-time glue code, importing runtime extensions by name.
  -->
  Service-startup-time glue code, obtaining dependency-injected fields
  by name.
  -->
  An expression, taking free variables from its lexical scope by name.
  (This is a build at "evaluation of this particular expression" time.)

And this means that even local syntactic concerns extrapolate to social decisions about how we expect to deal with our unrecorded knowledge. Every design decision has a lot to go by. :)

---

Another exciting part is that I think nested quasiquotation shows us a more general theory of lexical locality. If we're dealing with syntax as text, then locations in that text have an order, and we can isolate code snippets at intervals along that order (and mark them with parentheses). Intervals are partially ordered by containment, so we can isolate code snippets at meta-intervals between an outer interval and multiple nonoverlapping inner intervals (and mark them with parentheses with nonoverlapping parentheses-shaped holes: quasiquotations).

That "nonoverlapping" part seems awkward, but I think there's a simple concept somewhere in here.

With this concept of intervals, I'm considering higher degrees of lexical structure past quasiquotation, and I'm considering what kind of parentheses or quasiquotations would exist for non-textual syntaxes.

A module system deals with a non-textual syntax: The syntax of a bundle of modules. If the modules have no order to them, then we don't even have parentheses to work with, let alone quasiquotation. But they can have an order to them. We can impose one from outside:

  Module A precedes module B.

  Module A says, "..."
  Module B says, "..."
  Module C says, "Module A precedes module B."

  Module A says, "..."
  Module B says, "..."
  Module C says, "Module A precedes module B."
  Module D says, "Module B precedes module A."
  Module E says, "If module C and module D disagree, listen to module C."
  Module F says, "If module C and module D disagree, listen to module D."
  Module G says, "If module E and module F disagree, listen to module E."

With this ability to disambiguate when things go wrong, we can model lexical scope:

  Module A says, "Export foo = (import bar from system {B, C})."
  Module B says, "Export foo = 2."
  Module C says, "Export bar = foo + foo."
  
  Result: foo = 4.

If we really want access to bar at the top level, we can refer to it again, and we can even be sloppy about it and make up for our sloppiness with disambiguations:

  Module A says, "Export foo = (import bar from system {B, C})."
  Module B says, "Export foo = 2."
  Module C says, "Export bar = foo + foo."
  
  Module D says, "Export all imports from system {B, C}."
  Module E says, "If A and D export the same variable, listen to A."
  
  Result: foo = 4; bar = 4.

  Module A says, "Export foo = (import bar from system {B, C})."
  Module B says, "Export foo = 2."
  Module C says, "Export bar = foo + foo."
  
  Module D says, "Export all imports from system {A, B, C}."
  Module E says, "Export all imports from system {C, F}."
  Module F says, "Export foo = (import foo from system {A, B, C})."
  Module G says, "If D and E export the same variable, listen to D."
  
  Result: foo = 4; bar = 8.

  Module A says, "Export foo = (import bar from range R1)."
  Module B says, "Export interval R1, and begin it here."
  Module C says, "Export foo = 2."
  Module D says, "Export bar = foo + foo."
  Module E says, "End interval."
  Module F says, "These modules are in order: B, C, D, E."

Do you want to avoid the situation, wich happens in c, where when you need the sqrt you have to include the whole math file? I totally agree.

>Under these circumstances I would like to live in a world where I can go in and modify the libraries to have different names, with the difference in names making sense in the context of the application

This looks right to me. Perhaps it could be something like python's

    from library import function as good_name_for_your_project

Yes, definitely. In the Javascript world it's called tree-shaking: https://medium.com/@Rich_Harris/tree-shaking-versus-dead-cod...

> from library import function as good_name_for_your_project

What's happening here is that you're a) adding a feature in Python to support 'from..as', b) including an external library and c) continuing to keep around an old name that you don't really care about. You're essentially preserving the old name just because other people who your application doesn't care about use it.

Imagine a world where maintaining forks was tractable. Would this still be a good idea? Why not just do a search and replace and maintain a private fork, eliminating all this complexity in your private stack? Just delete 'from..as' from your private Python! :o)

> I'm not sure making a copy of a library is a good idea because it would lead the user to have many copies of the same libraries.

Yes, this is a fundamental difference in outlook/ideology. I think that copying isn't always bad. We culturally tend to emphasize the issues with copying a lot more than the costs of avoiding duplication.

A degenerate example is to observe that there are tons of 'e's in the novel I'm reading and try to deduplicate them. That is of course obviously farcical, but it at least serves to illustrate that there's a trade-off, and that always DRY'ing your code isn't obviously a good idea. Another example is to observe that the internet has many copies of the same libraries running at any given time. You can argue that they're on different machines, but then imagine a 'machine' consisting of multiple cores and private caches and non-uniform memory access and RAID-partitioned disks. Changing latency costs can make it reasonable to maintain multiple copies of some immutable data in a single 'machine'. Now consider that development is yet another cost that is open to variation. If (automatically) creating copies of something eases development, it's at least worth considering. For example, optimizing compilers can sometimes specialize a function differently for different callsites. That's duplication often inside a single binary, and it makes sense in some contexts.

Duplicating libraries isn't a problem: disk space for ease of development is an exchange which is getting more and more convenient.

Or, if it all needs to be bundled, you could have symlinks for the libraries you don't change.

If a name's already good, you're not going to change it; if it's bad, you should push that change upstream! (If the name's bad, it's likely that the original author didn't put much time into choosing the name, so I think it would be fairly straightforward to get that merged.)

But it's not just names per se. In that side-by-side comparison, they might also want to merge and branch parts of the code whose assumed invariants have now changed; invariants can act as Schelling points, like invisible names. Modifying code is something we do sometimes, and I think akkartik wants to see how much simplicity we'll get if everyone who wants a simpler system has the tooling support to modify the code and make it simpler themselves.

Personally, I find it fascinating how to design a language for multiple people to edit the code at the same time, a use case that can singlehandedly justify information hiding, modules, and versioning. But I think existing module systems enforce information hiding even more than they have to, so that in the cases where people do need to invade that hidden information, they face unnecessary difficulties. I think a good module system will support akkartik's way of pursuing simplicity.

But... my module system ideas aren't finished. At a high level:

- You can invade implementation details you already know. You can prove this by having their entire code as a first-class value with the expected hash.

Not necessarily. 'Good' and 'bad' are not absolute, they are extremely contextual. A name that is good for a general-use library might be sub-optimal for your application, or vice versa. Subjective taste is also a thing. So while you should certainly send out a pull request for the change, our model of the world shouldn't rely on the change actually getting pushed.

In general it is amazing to me how often a blindingly obvious Pull Request gets rejected or just sits in the queue, untouched. There's lots of different kinds of people out there. Which is why I tend to think more like a barbarian[1] about collaboration: think of other people as islands with whom you might collaborate if the stars align. But don't rely on the collaboration. Be self-sufficient.

[1] http://www.ribbonfarm.com/2011/03/10/the-return-of-the-barba...

---

"As much as I love this idea of implicit importing, I'm sure the explicit side -- which'll let you change whatever names you like -- will need to be there as well."

I actually interpreted your original post that kicked off this thread as implicit loading since Arc has no notion of modules or import. So the question of changing names did not arise. That seemed like a tangent to the original question.

These seem like separate questions:

1. Should Arc know how to react with implicit symbols?

2. Should Arc provide namespaces?

One the one hand, you can have implicit loading without needing a module/namespace system. On the other hand, I don't see how you can have implicit loading in the presence of namespaces. Without the "from..as" construct how would your system know which library to load a symbol from, if there's a collision?

For example:

string-to-number, or string->number, or string->num, or str->num, or ston, or ...

1. Whatever's chosen needs to be consistent (i.e. it should always be "str" -- or "string" -- but not a mix).

2. When choosing, readability and terseness need to be balanced.

(I have a feeling this is really a new discussion, and if this does start to kick off, I'll start a new thread.)

From what I can see, you have to install (npm's equivalent of) the standard libraries you want in to the node_modules dir. (So it's not much of a win over the boilerplate import code.)