One option is to extend calling lists so that calling a list with a number would continue to do the same thing (return the item at that position), while calling a list with a non-number would treat the list as an association list and do a lookup on that key.

Then no special syntax is needed because the standard `alist!x` would work, as it expands into `(alist 'x)`.

Naturally, this means that you couldn't do a lookup in your alist with a number using the `!` syntax, but that might be OK for you if you aren't using numbers in keys in the alists that you want to use the `!` syntax with.

This option wouldn't return the alist value on an index access though, as it would continue to return the association pair.

testible.clef means look up the value stored at what clef evaluates to in testible

testible!clef means look up the value stored at clef in testible

testable.0 means look up the value stored (at what 0 evaluates to) in testible

testible!0 means look up the value stored at 0 in testible

atist.0 means look up the value stored at what 0 evaluates to in atist

atist!0 means look up the value stored at 0 in atist

0 of course being an atom evaluates to 0

but what if you want it to evalute to something like the key stored at 0 . . . . . . if quote means something like "don't evaluate" and unquote means something like "do evaluate" then one could reason that atist,0 means look up the value stored at what 0 evaluates to (do evaluate it) in atist

  arc> atist.0
    '(a "up")
  
  arc> atist!0
    '(a "up")
  
  arc> atist,0
    "up"

Is this reasonable?

Edit: this could work for alists and insertion-ordered tables since it's unobvious how testible!0 & testible.0 should behave, numbers can and should be able to be keys so one can imagine a situation where behavior would be like so:

  arc> (= oble {1 "dream" 2 "bigger" 0 "awake"})
    {1 "dream" 2 "bigger" 0 "awake"}
  arc> oble.0
    "awake"
  arc> oble!0
    "awake"
  arc> oble,0
    "dream"

"Immediate reaction: this is a bad idea. Commas mean something specific in Lisp. And having `0` mean different things in different contexts is a recipe for disaster.

If it's just to lookup an entry by the index position IMHO that's not really a good trade for poor efficiency memory laden tables.

Meant to operate as a generalized pipeline for items to move through.

Could function as a basis for a kanban or as a sales management system.

In which cases the "stacks" as I call the stages in a pipeline would be:

"todo" "fortnight" "week" "today" "done"

"attention" "interest" "decision" "action" "love" "refer"

Respectively.

I was working on converting the whole thing from lists (https://github.com/Kinnardian/pipeline/commits/templatize) when I realized that tables are not order preserving (I can't believe this never came up working with json).

Order definitely matters, and I'm envisioning functionality like repeating tasks that once moved to the "done" stack/marked done wrap around to `pipe.0`(Imagine something that must be done once a month or once a week). I suppose I could tag items as monthly or weekly in which case they'd get appended to pipe!month or pipe!week as soon as they're done.

Another example is the `(createItem)` function which should take an item and an optional "stack" parameter. If no stack parameter items should be appended to the zeroeth stack `pipe.0` identified by index because it will be named by different keys in different pipelines.

Another example is a `(push)` function which pushes an item from its current stage to the "next" stage. With order this is simple.

EDIT: Order even matter matters within a stack. In the current kanban system I have a bunch of things I need to do today, the top priorities are actually at the top. I wouldn't want them getting all disheveled.

I think order-preserving tables are pretty useful for JSON -- particularly to help ensure implementation details aren't leaking even in a distributed system where JSON is being passed through multiple parsers and serializers -- but it's worth noting that even JSON objects are specified to be unordered and are commonly treated as such. (RFC 8259: "An object is an unordered collection ...")

Even JavaScript's standard JSON.parse(...) doesn't preserve order the way you might like it to. Here's what I get in Firefox right now:

  > JSON.stringify(JSON.parse("{\"a\": 10, \"1\": 20}"))
  "{\"1\":20,\"a\":10}"

JavaScript Maps are a different story; they're a newer addition to the language, and they're strictly specified to preserve insertion order. Other languages use dictionaries which preserve order, including Groovy (where [a: 1, b: 2] is a LinkedHashMap literal), Ruby, and... you've already found details on Python.

Even in Groovy, JavaScript (Maps), and Ruby, the only way to observe the order of their collections is to iterate from the beginning, and the only way to influence it is to insert entries at the end. This means any substantial interactions with the order of these collections will be just about as inefficient and inconvenient as if we had converted the whole collection to an association list, processed it that way, and converted it back. Essentially, it seems the order is preserved only to prevent programs from having language-implementation-dependent bugs, and maybe to help with recognizing values in debug output, not because it's a useful for programming.

Every one of these languages makes it easy to communicate the intent of an ordered collection by using lists or arrays of some sort. I'm pretty sure I've seen this lead to association lists even in JSON, particularly for things like database query results:

  [
    {"id":..., "name": ..., "occupation": ...},
    {"id":..., "name": ..., "occupation": ...}
  ]

- If I'm not doing both kinds of lookup several times in the same part of the code, the lookup styles don't both need to be concise at the same time. Different parts of the code can convert to different representations that are concise to use in that context.

- If it's the root layer of a data structure, then I can simply use two different local variables to refer to that data, one of which always treats it as a list and one of which always treats it as a table.

- Unless I'm processing some specific indices, I'll usually want to iterate through the whole data structure, so I usually wouldn't be using any indexing operations in the first place.

- If I know what specific indexes I want at development time, then I would usually use an unordered table (for easy indexing) or a fixed-length list (for easy destrucuring).

- If I'm performing any two lookups for the same reason, they can usually use the same expression in the code.

So I would have to be writing some code where I'm performing several different lookups of specific ordered indexes and specific chosen keyed indexes, all of which are for distinct reasons I know at development time but few of which are under indexes I know at development time. Moreover, each lookup must be two or more layers deep into a nested data structure.

This seems to me like a very specific situation. I suppose maybe it might come up more often in data-driven applications that have many scripted extensions which are specialized to certain configurations of data, but it seems to me even those would rarely care about specific ordered indexes.

I think I'm just trying to encourage you to get to know other techniques because I think that's the easiest way to start. Building a custom data structure in Arc is not something many people go to the trouble to do, so it's a bit hard to come up with a good example.

I don't think you need any new ssyntaxes for this new data structure.

Say you have an ordered dictionary value `db` and you want convenient syntaxes to look up items by index (where 0 should give you the first item) and by key (where 0 should give you the item with key 0). One thing you might be able to do is this:

  db!by-key.0
  db!by-pos.0

  (defcall ordered-dict (self val)
    (case val
      by-key (fn (key) ...)
      by-pos (fn (pos) ...)
      (err "Expected by-key or by-pos")))

  (= db!by-key.0 "val")
  (= db!by-pos.0 "val")

  (sref db!by-key "val" 0)
  (sref db!by-pos "val" 0)

But we've defined db!by-key and db!by-pos to be procedures that only know how to get the value, not how to set it. We need them to return something that knows how to do both.

So let's define another type, 'getset, which contains a getter function and a setter function.

  (def make-getset (get set)
    (annotate 'getset (list get set)))
  
  (def getset-get (gs . args)
    (apply rep.gs.0 args))
  
  (def getset-set (gs val . args)
    (apply rep.gs.1 val args))

  (defcall getset (self . args)
    (apply getset-get self args))

  (defextend sref (self val . args) (isa self 'getset)
    (apply getset-set self val args)))

  (defcall ordered-dict (self val)
    (case val
      
      by-key
      (make-getset
        (fn (key) ...)
        (fn (key val) ...))
      
      by-pos
      (make-getset
        (fn (pos) ...)
        (fn (pos val) ...))
      
      (err "Expected by-key or by-pos")))

Note that we don't need to do (defextend sref ...) for ordered-dict values here because the `sref` calls don't ever get passed the table directly. You would only need that if you wanted (= db!by-key "val") or (= db.0 "val") to do something, but those look like buggy code to me.

So far, so good, but you probably want to do more with ordered dictionaries than getting and setting.

There's at least one more utility you might like to `defextend` in Anarki for use with ordered dictionaries: `walk`. A few Anarki utilities like `each` internally call `walk` to iterate over collections, so if you extend `walk`, you can get several Anarki utilities to work with ordered dictionaries all at once.

You may also want to make various other utilities for coercing between ordered dictionatires and other Anarki values, such as tables and alists. That way you can easily use Anarki's existing table and alist utilities when they work for your situation.

Note that this approach doesn't make it possible to use the {...} curly brace syntax you and shawn have been talking about. Since there are potentially many kinds of tables, I'm not sure whether giving them all read syntaxes really makes sense; we might start getting into some obscure punctuation. :-p

  db#0
  db?0

But I don't see the need to order data in your application equating to the need for tables that support constant insertion order.

I 'm not going to say what you're doing is wrong because it's not, but I am going to say it's non-standard and I think there other ways to manage your data that doesn't require ordered tables (which have downsides with data growth). It's just my opinion, but there's not much you can't do with the standard storing of table records and maintaining of indexes.

https://morepypy.blogspot.com/2015/01/faster-more-memory-eff...

It's only more memory efficient than its previous incarnation.

It's an interesting implementation though. The use of sparse arrays to index the entries is compelling. Measuring this kind of stuff can be non-trivial though as you also have to account for gc (and/or compaction) at different times within your application. There are always trade-offs.

Personally I haven't compared the various implementations (clojure vs. racket vs. python) such that I can give you any real insight. I know clojure's (or rather java's) array-maps are costly when performing iterative lookups within large data sets because I've benchmarked it.

Microbenchmarks against a previous version only means they've made a relative improvement.

The benchmarks that would matter to us (or at least me) are:

1. how does it compare to an equivalent implementation without having to maintain insertion order.

2. how does it hold up under stress (larger data sets, with heavy load where gc/compaction have to occur)

So I don't think this is something anybody supports:

  arc> (= grail (otable a "up" b "down"))
  arc> grail.0
  "up"

I think clojure's array-maps do it but with some limitations:

https://clojure.org/reference/data_structures#ArrayMaps

https://pymotw.com/2/collections/ordereddict.html

More serious suggestion: Journaling Dict. Maybe the Arc type could be `jtable`?

'jtable'

Journaling infers logging by insertion order.

so jtable, or log-table are good no?