I haven't read any more than a few papers on it, and maybe only one of those in depth (which I'll mention below). Mostly I'm going by forum threads, wiki articles, and the design choices certain languages make (like Inform's multimethods and Haskell's type classes).
As far as I understand the history, Philip Wadler's work basically defined the strict parameters of the expression problem and explored solutions for it. Separate compilation and the avoidance of dynamic casts were big deals for Wadler for some reason.
That work was focused on Java, where it's easy to define new classes that implement existing interfaces but impossible to implement new interfaces on existing classes.
The solution I'm most familiar with for Java-style languages is the use of object algebras, as described in Oliveira and Cook's "Extensibility for the Masses: Practical Extensibility with Object Algebras" (https://www.cs.utexas.edu/~wcook/Drafts/2012/ecoop2012.pdf). In this approach, when you extend the system with a new type, you define a new interface with a generic type parameter and a factory method for building that type, and you have that interface inherit all the existing factory methods. So you don't have to solve the unsolvable task of implementing a new interface for an existing class, because you're representing your types as type parameters and methods, not simply as classes.
So I think the main subject of research was how best to represent an extensible program's types and functions in a language like Java where the most obvious choices weren't expressive enough. I think it's more of a "how do we allow extensions to be made at all" problem than a "how do we make all the extensions maintainable" problem.
But then, I've really barely scratched the surface of the research, so I could easily be missing stuff like that.
>I suppose a more ideal form of the documentation would describe on the "Templates" page that template instances were a tagged type with a certain representation. It could describe the direct implications of that, but it would also have a link to the "Type operations" page for more background.
That would have been tremendously helpful, can we do that? Who actually maintains the documentation, can it be updated?
Ah, you're imagining us having to write and maintain 1000×1000 individual `defextend` forms someday? Yeah, that does seem like a problem that would not feel solved once we got to it. :-p
I don't think that aspect of the expression problem is solvable in a language design. Instead, it's an ongoing conversation in the community. Sometimes the intent of one feature and the intent of another feature interact, leading people to do a nonzero amount of work to figure out the intent of the two features put together. That work is an essential part of what the community is trying to accomplish together, so it's a cost that can't be eliminated. The intent has to be reflected in the code somewhere, so there will be a nonzero amount of code that serves feature-coordinating purposes.
Regardless, I'm optimistic that although the amount of code will be nonzero, it'll still have a manageable size. To the extent we have any kind of consistency around these feature interaction decisions, those consistent principles can develop into abstractions. The only way we'll have 1000×1000 individual intersections to maintain is if we as a community culture are already maintaining 1,000,000 compelling and distinct justifications for them. :)
Yeah, I suppose you could say the problem is 'solved'. I think of it as a trade-off with costs. We don't know how to achieve zero cost.
For example, I absolutely agree with you that 2 lines per method to extend every table method to some new type constitutes a solution for us. But if we had a thousand such types and a thousand such methods, it may seem like less of a solution. But then `defextend` would be the victim rather than cause of bloat.
Perhaps we should maintain the docs only in plain-text and only within the repo, just because of the operational overheads of managing multiple branches. At least that way they won't seem to lie to a newcomer. I'm curious to hear what others think.
You can get to it from the "Type operations" link in the table of contents. Of course, you might've been drawn to the "Templates" page instead.
And unfortunately, that documentation describes Anarki's stable branch. Making documentation of similar quality for Anarki's master branch might be quite a bit more work. So even if you were familiar with tagged types already, the knowledge that template instances were a tagged type on Anarki master might be hard to discover.
I suppose a more ideal form of the documentation would describe on the "Templates" page that template instances were a tagged type with a certain representation. It could describe the direct implications of that, but it would also have a link to the "Type operations" page for more background.
"But in general, having incompatible types easily share functions without sharing too much is an open problem: https://en.wikipedia.org/wiki/Expression_problem A language can easily add a method to many types, or add a new type to many methods. But we don't yet know how to achieve both sides."
I'm trying to follow, but I think you and I must have different understandings of the expression problem. That article lists several known solutions to the expression problem. The solution Anarki uses is `defextend`.
What do you mean by "sharing too much"?
Is Anarki's `defextend` technique already encouraging a bloated codebase, or is there some other technique you're thinking of that would do that?
I spend all my time with Arc working on the forum, so JSON is more important to me than it might be to someone else who primarily focuses on the language.
I was working on the data export feature, which before just dumped a user's posts and comments as a printed statement, but I wanted it to export JSON because that's more portable. Racket has a really nice JSON parser, unfortunately all the forum data is templated, so I can't just pass it to Racket because it doesn't understand the type. There may be a way to get it to work at that level but I don't know what it is.
So I had to get an old JSON parser for Arc and rewrite a bit of it to be able to generate JSON from forum data. I think it was zck's parser but I'm not entirely sure ATM. It works, but it's an awkward workaround. Any data coming from the forum has to go through one parser, and any other data, such as from a remote API would be going through another (Racket's.)
>Is making it work just a matter of calling the Racket primitive on `rep.template` instead of `template`?
Yes, I had to hunt around to even know that existed, because it's not in the docs.
> ... with language features like macros and templates that have become ubiquitous, I feel like it's kind of cheating not to just fold them into arc proper.
It's totally fine to move something into arc.arc if you want to do that. It's always felt like a non-existent distinction in my mind whether something is under arc.arc or libs/. Is Anarki all language or all standard library? Depends on how you look at it. Why does it matter?
> But that's just me... one thing I've learned being here is that I seem to flow against the culture more than with it, so I can just agree to disagree.
This doesn't feel like a disagreement, more like a language barrier. If I understood better I might know whether I agree or not.
Don't worry about sounding dismissive, I totally understand where the questions are coming from.
Tables and objects feel like separate concepts, and they have complementary strengths and weaknesses, and one doesn't subsume the other. To me it seems obvious that if we want to have both, we need them to have different types.
For example, sometimes you want the 'dynamic' ability to set arbitrary keys of metadata on a thing. Sometimes you want the same operation to be an error, by providing a schema. How would a single type do both? No language does so, to my knowledge.
Things should have the same type when they have compatible behavior. When they are incompatible, they shouldn't.
Supporting helpers like len and keys may well still make sense. And as the original story did, this is easy to do.
But in general, having incompatible types easily share functions without sharing too much is an open problem: https://en.wikipedia.org/wiki/Expression_problem A language can easily add a method to many types, or add a new type to many methods. But we don't yet know how to achieve both sides.
And honestly, I think the expression problem isn't important. It doesn't take too much code per method/type. And making it easier just encourages large, bloated codebases.
> ...given how often I and other people seem to run into issues with it.
One thing that might be useful here is a list of issues people have encountered with templates. Maybe we should create a wiki page on GitHub and add to it every time an issue comes up. Then we can have a big-picture view of them and a sense of how many are things people need to learn about Arc, and how many are bugs to be fixed.
I believe Anarki behaves exactly the same as Arc's intent when it comes to templates. The changes that I made here seemed strictly superior to the buggy implementation upstream. But if you disagree you should absolutely feel free to just revert the commits and go back to Arc behavior. I don't use Arc anymore, so my opinions are extremely weakly held, you don't have to bother persuading me. Or, if you have some other specific issue in mind, I'd be happy to be persuaded that I'm wrong.
I think it's a documentation issue. I think I had to search the forums to find out about it when I was playing with JSON interop. Nowhere on the actual template page in the Arc documentation does it tell you this is a thing.
Some things I've only been able to figure out by studying the compiler or arc source code itself. Granted, that's illuminating, but it's also sometimes really annoying.