There's a problem with using extend here: extend checks to see if it's going to override the original function first, before the original function has a chance to run. That's not what we want for defining custom callables such as for (my-custom-table 'key). The call in extend to (isa f 'table) would go into an infinite loop trying to call isa.
In my Penknife draft, there's a calling rulebook, extensible just like any other core rulebook. To resolve the paradox, two callable types are built in: Functions and rulebooks. The core calling behavior tries those two cases first, then finally calls the "rb-core-call" rulebook.
If someone replaces rb-core-call with something other than a core function or rulebook, they'll get an infinite loop, and it'll be their own fault.
I think Arc could easily be patched for something like this: Just change the "Function call on inappropriate object" error message so that it calls an Arc function that raises that error message instead. We could call it 'custom-call, or maybe 'defcall-missing. :-p
"Rulebook" may actually be a bad name for it, 'cause there are more complicated notions of rules. I chose the name because I considered it to be a simplified form of Inform 7's rulebooks, but I might have simplified the "rule" part right out of it.
In my Penknife draft, every function can either return a value, raise an exception, or fail with a failure message. Failure means the function doesn't meaningfully apply in this case, and it lets the caller try something else if they they happened to use failcall to provide an alternative. If they didn't use failcall, it just raises an exception.
A Penknife rulebook can be called as a function. It's represented by a sequence of functions that are called one at a time until one returns a value or raises an error. If they all fail instead, then the whole rulebook fails.
A rulebook is like a series of 'extend cases, but being its own type, it's less encapsulated. There could theoretically be rulebook utilities that let you count the number of extensions, etc. While an implementation of 'extend could do this too, it's a little unheard of.
The exact implementation of a Penknife rulebook is kind of up in the air at this point. I was just using a box with a list of functions inside, but recently I've been considering focusing all of a module's mutation onto one easily swappable place, rather than having it spread out over a namespace and an arbitrary number of rulebooks, global tables, and other such things.
So... that sounds an awful lot like message passing + prototypes. So much so that I'm fairly sure you could create rulebooks as syntatic sugar on top of message passing + prototypes.
And if rulebooks are sugar on top of prototypes, and the built-ins are implemented with message passing...
Okay, so my example truly is impossible in pgArc. That's what I already suspected, but it's nice to actually know. Using something like defcall would probably work, though.
A few random observations, in case they are helpful to you:
- I often don't get anyone agreeing with me in the forums (http://arclanguage.org/item?id=14263 notwithstanding :)... I notice that people often will spend their time instead to share their own ideas, concerns, and criticisms... or say nothing if they don't have anything additional to contribute. (Some of my best hacks have gotten little response in the forum when I first post about them). Thus I suspect you shouldn't necessarily take lack of positive agreement as evidence of disagreement.
- One pattern I've noticed on the forum is that someone will drop by and say "Arc would be a better solution for my problem if it had (immutable lists | software transactional memory | message passing | documentation | unit tests | libraries | ...)". It's more rare for this to be followed up by an actual runnable implementation of the feature.
- When someone does provide an implementation, they sometimes appear to be disappointed when their solution isn't adopted by other people. I suspect maybe there may be some confusion about the nature of problems and solutions: even if say X is an excellent and succinct solution to problem A, someone like myself for example may not take the trouble of adapting it if I happen not to have problem A. And... even if I do have problem A, X may or may not be a useful contribution to a solution for my problems A+B. When I'm trying to solve A+B, I may remember X and then try it out and see if it is useful for A+B, but it can be hard to know in advance whether I'll personally find X useful until I'm seriously looking at the details of A+B.
- I enjoy sharing solutions I've implemented to problems I've had, and I'm pleased when I learn that someone else has found them useful as well. But it can take a year or more after I first publish a hack for me to learn that someone is using it... which is fine for me since I'm aware that there can be a long feedback cycle, but I notice other people become discouraged when nothing seems to be happening in a shorter time frame.
To maximize the likelihood that other people will be able to make use of your solution, you may find this template helpful:
- explain the problem the solution solves. (People often omit this step: they immediately start off with describing the solution. But while it may be easy to see that a solution is elegant, it can be hard to see what it's a solution for).
- describe the solution. (Most people do this part).
- demonstrate how the implementation solves the problem. (This part is also often omitted, but a few concrete examples can do wonders to help people make the connection between the solution and a problem they have).
The forum itself tends to be a poor place to do this because of its modest formatting capabilities and the inability to edit posts later means that descriptions become disjointed, spread over multiple comments and posts.
I agree with most of what you're saying, but I will add a few things:
There are, generally speaking, two types of people who visit this forum. The first, type-A let's call them, are hardcore hackers, like yourself(ves), who are interested in the language design and the underbelly mechanics of language assembly. Then there are people, like myself, whom are only interested in using the language to build applications.
When I read: > "Arc would be a better solution for my problem if it had...", "It's more rare for this to be followed up by an actual runnable implementation of the feature." I will suggest the two types of people represent the divide within the pattern you're recognizing and I will going to even further to suggest this is not a bad thing. I would like to encourage more type-B user engagement - even, if only, so that I am not all alone :)
While I don't have the time or inclination to understand the full content/depth within these type-A posts, I do try to learn what I can even if only at a surface level. I really do enjoy reading these posts and do appreciate them so I hope they don't go to a back channel as akkartik has mentioned a possibility for.
Furthermore, I would like to highlight some of the points aw's message that I feel are really important (and with my interpretation):
* Inline comments are not the best way to present your ideas. I've struggled assembling all the bits and pieces and knowledge requirements from prior posts/threads. The order alone messes everything up and I simply do not have the time to do the paper chase.
* Writing an external blog post will likely force you to present a fuller picture, with its surrounding context, to an audience that can't always be on top of things.
All that said, please don't give up on the ideas, we need more of them - not less.
Oh, and by the way, there's an infinite amount of room available in the wiki for other things such as proposals, ideas, arguments, implementation approaches, and so on. So if you're looking for a place where you can write things out and edit them later, the wiki is great place to do that. (You can of course also write your own blog post if you want, and we can link to it from the wiki as well).
I took a break to write a blogging platform from scratch that includes my own custom markdown.
I even posted my first arc topic here: http://blackstag.com/blog.posting?id=2. I then quickly discovered the auto spam bots are moving from the arc forum to my blog site every night. So I took another break, lol, just to put some spam management code in. I'm glad had read enough of pg's code to implement some good techniques similar to news.arc.
Now I'm ready to do more blogging - I hope. I've accumulated a list of 9 or 10 posts and I plan to get some arc stuff in there as well.
I notice other people become discouraged when nothing seems to be happening in a shorter time frame.
I went through the five stages of grief after submitting http://arclanguage.org/item?id=12657. It took me literally weeks to get right, and it sat at I think 1 point for nearly a week (with just evanrmurphy's positive comment[1]).
But eventually I realized much of what you said :) I have more realistic expectations now. Points are nothing, and responses come in time.
[1] While evan was active you could tell when he'd seen something because it would go to 2 points :) (without suggesting that he upvoted everything)
Evan would often articulate agreement or admiration even if he had nothing to add, and it was useful. But I'm not sure we should all do it.
Perhaps all this is a sign that we need a back channel.
Yeah it seems people here don't express agreement unless they can add something useful.
Some of my best hacks have gotten little response in the forum when I first post about them
Because some of your best hacks left me speechless :)
Things I see here often take a while to digest, days or weeks. It's taken me over a year to fully appreciate extend, a road that took me through various dead-ends.
My http://awwx.ws/table-vivifier patch sounds like it may be close to what you want, though it goes too far: it extends tables so that if a key isn't found in the table, it calls a specified function to generate the value -- and then inserts the new value in the table.
If it would be useful, I could simplify it to leave off the last part, and then you could specify the action to take if the key isn't found (such as looking it up in a parent, for example).
Pretty cool. But with my proposal, you could write that in Arc itself.
My solution is general... it works on all compound data types: tables, input, lists, etc.
Imagine being able to write cons, car, cdr, instring, peekc, readc, and more, all in Arc! It drastically cuts down on the number of primitives Arc needs.
You might also find writing an Arc interpreter in Racket or even Arc3.1 a lot of fun:
- you'll get continuations, threads, and parameters that actually work
- you'll be able to easily implement things like fexpr's, first class macros, or serializable closures in your interpreter that Arc3.1 doesn't have (hard to do in a compiler)
- you can write parts of your program in Arc3.1 that you want to run fast (and doesn't need your extra interpreter features) and other parts using your Arc interpreter, and it can all work together
Yeah, I also considered that, or Chicken Scheme, or Common Lisp, but honestly... as much as I may like Lisp... Scheme and Common Lisp just feel... bloated. I love Arc's shortness and simplicity, and Python has it too, I just hate the way Python behaves sometimes (curse Python's lambdas and lack of proper closures).
I admit that would certainly be an easier choice, in some ways. I do like the idea of writing an Arc interpreter in Arc... but from my initial tests, Arc seems quite slow with string/list processing. Maybe it was just my program. Isn't that already being handled with arcc, though?
Ah, if by "arcc" you're referring to https://github.com/nex3/arc/tree/arcc, that's an Arc compiler (replacement for ac.scm) written in Arc. Thus while it produces code that runs faster than interpreting code would, it has the same limitations as ac.scm.
Arc seems quite slow with string/list processing
Do you have an example handy? ("Here's a program in Python, and here's the program in Arc, and see how Arc is slower"). ...There may be things we can do to speed Arc up, but having a runnable example is useful. I.e., I might do something that makes my Arc program run faster, but it might not necessarily make your Arc program run faster.
Oh, at first glance it seemed to behave somewhat like an interpreter, my mistake.
---
Yeah, I do, actually. I wrote a program that can take an (ad-hoc and simple) playlist format and convert it into .xspf. I then rewrote the program in Arc (which was much nicer than writing it in Python), but it ended up being drastically slower. Which means I'm not sure if it's a problem in Arc, or Racket, or my program! It could be any combination of those three.
The program itself should have O(n^2) complexity, due to error checking (which actually isn't even implemented in the Arc version...) If I got rid of error checking, I could get it down to O(n log n) but I don't want to do that.
In any case, the complexity should be the same for both Python and Arc. If I recall, the slowness was primarily caused by me searching for whether one string is a substring of another string. Python has string.find (which performs faster than regexps, when I tried it), but I'm guessing Arc's implementation is slow.
This is all whishy-washy guessing, though. I haven't done concrete tests or anything, so take it with a grain of salt. However, I'm vaguely interested in finding out what the performance bottleneck is, and possibly fixing it.
---
Edit: I just tried these:
# Python
for item in range(100000):
"foobarquxcorge".find("foobar") != -1
; Arc
(repeat 100000 (posmatch "foobar" "foobarquxcorge"))
And got some very weird results... the Python version consistently takes about 2 seconds. When I first tried the Arc version, it took something like a minute or two. Aha! So that's the bottleneck? Not so fast. I then tried it a second time, and it took ~5 seconds... slower than Python, but not by too much.
It seems pgArc's performance isn't very reliable. I've noticed sometimes at the REPL that running the same (or similar) code will sometimes be instantaneous, and other times it'll chug along for a few seconds. I don't think it should be taking 1-2 minutes for something that should be taking 5 seconds, though.
However, my program pretty consistently took much longer than it does in Python, so I think I can safely rule out posmatch, which actually seems quite fast (almost as fast as Python, anyways).
There's room for improvement in posmatch, but it doesn't seem to be the smoking gun (at least not yet). For fun, I tried this:
for item in range(100000):
re.search("foobar", "foobarquxcorge")
It took ~10 seconds, so as I said, regexps are slower than find, in Python. Possibly because it has to create a match object, rather than just returning a number? I don't know.
Times will be variable because of garbage collection, that's normal. But having your posmatch example take a minute is very weird though, I've never seen anything like that happen.
I'm actually surprised to hear that posmatch is almost as fast as Python's find. Python's find, after all, isn't written in Python (like how Arc's posmatch is written in Arc), it's written in C.
If you use a regex more than once you'll want to compile it. Both Python and Racket have this capability.
I've sometimes experienced extremely long delays with DrRacket (I use DrRacket as an editor; I use the command-line "racket" to interact with Arc, and haven't observed these delays with racket). Perhaps Pauan is using DrRacket? And as for why that happens... I think it was a) trying to complete a garbage collection with b) a large part of its memory put into swap space. I'm puzzled by how it could take so long to un-swap that much memory (max 360-ish MB)... perhaps it did it in an impressively inefficient way (like, load page 1 containing a certain cons cell, then seek the disk for page 25359 containing its car, then seek back to page 2 for the cdr, and so on). Also, perhaps displaying the "recycling" animation contributes to the problem. Hmm, perhaps I'll figure that out at some point.
If you're not using DrRacket, then I'm not sure what might cause it, other than some unlikely things (e.g. memory of "racket" was swapped to disk and you were moving massive files around).
Meanwhile, if you really care about string-searching, or find it to be a pain point, you may want to implement the awesome Boyer-Moore algorithm: http://en.wikipedia.org/wiki/Boyer-Moore
No, I'm just using Arc on the command line. Garbage collection was my bet, and you're right that it could have been swap as well.
---
Yeah, might actually be a good idea to integrate that into the base posmatch, but as I said, posmatch isn't actually that slow compared to Python's find, so the bottleneck is probably elsewhere.
But yes, obviously in production-level code you should be compiling/caching them yourself, if solely on a "just-in-case" basis.
---
Also, I would like to point out that in all my time using Python, it's been very consistent as far as speed goes (no 3 second pauses), so would that imply that Python's garbage collector is more incremental than Racket's?
One possibility is that Python uses reference counting, which immediately frees non-cyclic garbage as you go, and, iirc, an additional occasional garbage collection cycle to free the remaining cyclic garbage. So I'm just guessing, but if you're not creating much cyclic garbage, maybe that's an explanation for why you're not seeing many garbage collection pauses.
This is a forum for the Arc language. To request a feature for Hacker News, click on the "Feature Requests" link in the footer of the Hacker News page.
Ah, from my interpretation of your wording, it sounded like you wanted the bug fixed in Hacker News... which is why I replied with the link to the page for Hacker News feature requests.
Your contribution of a bug report for news.arc is valuable, and we thank you for it.
Actually I imagine that many people won't have a problem with running the connector, in much the same way that many people don't have a problem with downloading and running other kinds of software from the Internet on their personal computer.
And... naturally there will be other people who don't want to run the connector, and so the site will be less useful to them.
Since this is a new and different kind of system, I wanted to be clear about what's going on -- so no one is surprised, and can make their own informed decision about if and where they want to run the connector.
I also plan to eventually have an API so that, among other things, people can perform their own runs and post the results to the site.
