Skip to content

Adopting astero

The tutorial builds a compiler from scratch. This is the other path: you have a compiler, it works, and you want the derivations without a rewrite.

That path has been walked three times: a Python-to-JavaScript compiler, a Python-to-WebAssembly compiler, and a typed-Python-to-native compiler. Some of what follows is advice; the rest is what went wrong.

Start with a shadow

Declare your grammar, derive the table you already have by hand, and assert they agree, without changing any behaviour.

def test_my_operand_table_matches_the_grammar():
    assert derived_from(MY_GRAMMAR) == MY_HAND_WRITTEN_TABLE

Two outcomes, both good. They agree, and you can now delete the hand-written one with a test that would have caught the deletion going wrong. Or they disagree. Every time this project ran that comparison, the disagreement was a defect in the hand-written side.

Adopt in this order

  1. coverage first. It is a test, changes nothing, and finds handlers you never wrote. Cheapest thing in the library.
  2. grammar queries next: replace one hand-written table with a derivation.
  3. rewriting where you have NodeTransformer passes.
  4. hygiene if you inline, substitute or rename. See below.
  5. emit_rules last, one production at a time; anything without a rule falls through to your existing generator.

Declaring your grammar

If your AST subclasses Python's, one line:

MY_LANG = lang_py.build(module=my_ast_module, name="mylang")

Roles come from Python's declaration; only the productions you actually have appear. If your IR is annotated dataclasses, Grammar.from_dataclasses reads names, sorts and shapes off the annotations and you author only roles.

The mistakes that were actually made

Asking the table before the scope. Two compilers looked up a name in a builtin table before asking whether anything bound it. In one, all 42 of its runtime-symbol names became unusable as variables: map = 3; return map + 1 emitted the source text of a helper function. Python's rule is that any binding beats a builtin, and a table consulted first overrides it. If you have a name table, ask the scope first.

Substituting at every Name. An inliner replaced every occurrence of a parameter, so a body containing [x for x in ...] produced [99 for 99 in ...]. A binding occurrence is not a use, and only roles tell them apart.

Capture nobody was looking for. The same inliner, after that was fixed, still turned g(k) with body [n + k for k in ...] into [k + k for k in ...]: the argument's free name captured by a binder in the body. A guard against the body rebinding a parameter does not catch it. hygiene.substitute with fresh= renames the binder.

Recovering structure from emitted text. A statement emitter built an elif chain by emitting the nested if to a string and cutting it up: strip a leading "if (", strip a trailing "}", splice. It produced JavaScript that would not parse the moment the inner test needed a declaration hoisted ahead of it. If you find yourself calling startswith on generated code, the answer is in the node.

Two shapes of failure to watch for

Drift is the loud one: two enumerations of the same thing, one of them stale.

Answering the wrong question is the quiet one, because nothing disagrees with anything. ident_slots() says where identifiers are; ident_slots(ns) says where variables are, and a renamer built on the first rewrites x.foo. When two questions are close enough to confuse, make the grammar answer both and name them differently.

What will not convert

Measured across the three compilers:

  • Lowering: turning a comprehension into a loop is a fold over a list whose length is the input's, and a fold is not a declaration. Moving it into a rewrite pass costs lines and, if you lower onto closures, about 11% of the generated code's speed.
  • Statement layout and control flow: no notation for it. Every compiler here writes its own.
  • Name resolution against an FFI or a module resolver: that is host policy. A guard that means "ask this other subsystem" is where the guard language stops being closed.
  • A cost model: "is this operand provably small?", "can this overflow?" decide a representation; no closed guard vocabulary reaches it.

A note on what this buys

Across three compilers, roughly nine defects were forced into view by a derivation disagreeing with a hand-written table. Rather more were found by differential testing against the source language, which needs no library at all.

The claim is narrower than "it finds bugs": of the defects a derivation forces, several become unwritable rather than fixed. You cannot write the wrong ctx against a grammar that derives ctx. If you want fewer bugs this quarter, write a differential harness first. If you want a class of bug to stop existing, this is the tool for it.