Skip to content

User guide

Each module answers one question. This is the question, the call, and the failure it prevents.

grammar: the declaration and its queries

A Grammar is productions of fields; each field has a shape, a sort and a role.

Static queries answer which fields, over every node of a production:

PY.positions(Kind.DEF, VARS)   # production -> field names that bind
PY.operands(VARS)              # ... that read
PY.definitions(VARS)           # ... that introduce
PY.ident_slots(VARS)           # ... that hold a bare identifier
PY.children("FunctionDef")     # fields a traversal descends into
PY.concrete("stmt")            # productions that can actually appear
PY.with_trait("pure")          # productions carrying a trait

Instance queries answer which values, for one node, applying conditions:

grammar.reads(node, ns)        # the contents of every slot that reads a name
grammar.binds(node, ns)        # ... that introduces one

The difference matters when a field's role depends on a sibling. postpile's AssignValue.target is a definition when declare is set and a use when it is not; a static table must report both, and only an instance can say which. Writing that walk inline is the sign you want the instance query.

Three front doors

lang_py.build(module=my_ast)                    # a module of ast-like classes
Grammar.from_dataclasses(name, classes, roles=…) # annotated dataclasses
GrammarBuilder()                                 # by hand

from_dataclasses reads names, sorts and shapes off the annotations, so only the roles are authored. postpile declares twelve SSA instructions in twelve lines that way.

rewriting: rules over trees

Pass("desugar", rules("pattern => result"), eliminates=(...), grammar=…, stop_at=(...))
  • A rule is pattern => result, or a decorated function for a fold.
  • @rewrite("pattern") puts a callable rule next to its pattern.
  • A callable returning None declines, and the next rule is tried.
  • A rule may return a list, which is spliced into a statement list.
  • stop_at=(ast.FunctionDef, ...) keeps a scope-local rewrite out of nested scopes.
  • eliminates= is a postcondition; fix_contexts recomputes ctx from roles.

Prevents: a rewrite that sets the wrong context, and one that fires nowhere.

scopes: the scope tree and what each block binds

scope_tree(tree, grammar)
bound_names(value, sort)

Two tables, deliberately distinct: SCOPES says what symtable calls a block, BINDING_SCOPES says what a name binds over. PEP 709 separated them at 3.12. Ask the one that matches your question.

Prevents: an enumeration of binding positions that misses for, with … as, except … as or the walrus.

hygiene: moving code without breaking names

substitute(node, mapping, grammar, ns, scopes=…, fresh=…)
rename(node, mapping, grammar, ns)
Fresh.avoiding(node, grammar)
free_names(node, grammar, ns)

substitute replaces only uses, refuses when the target rebinds a substituted name, and, given fresh, renames a binder that would capture.

Prevents: [99 for 99 in ...], and an argument's free name captured by a comprehension in the body.

coverage: gating a dispatch table against the language

cover = dispatch(PY, (gen_expr.registry, gen_stmt.registry),
                 bases=("stmt", "expr"), accounted={"desugared": {...}})
assert not cover.missing, cover.explain()

handlers() reads singledispatch registries; match_arms(fn) reads case Cls() patterns back out of source, so a match is measurable too. accounted names each deliberate omission, and absent/redundant fail when a reason goes stale.

Prevents: a production nobody handles, found by a user rather than a test. Its first run found dead code that lowered assert a second and wrong way.

emit_rules: emission as a declaration

js = Emitter(grammar, levels=LEVELS, typer=get_type, fallback=…)
js.projection("sym", lambda op: SYMBOL[type(op)])
js.rule("BinOp", "{left} + {right}", All((OpIs((ast.Add,)), Both(NUMERIC))))

Guards are closed on purpose: Is, Both, OpIs, Const, Has, joined by All, Either, Not. A guard that admits host code stops the rule set being readable as a table.

With levels, brackets are derived from binding powers. With levels=None and the operator last, template composition emits post-order, which is how the same notation reaches a stack machine and WebAssembly's flat form.

Where it stops: guards about a node's types convert; guards about an operand's provenance (is it provably a small constant? can this operation overflow?) are a cost model, and stay written.

emit: documents and derived brackets

Doc, Nest, Line, a renderer, and needs_parens from a precedence table. emit_py is a whole Python emitter built on it, checked by emit-then-reparse over the standard library.

tables: the tables a grammar cannot derive

Family.parse declares an operation-by-type matrix with the normalisation stated once and holes written as holes. Use it when one rule ("a Bool crosses as an Int64") would otherwise be spelled in every row.

generate: programs that exercise every declared position

missing_coverage turns "have I tested every role in the grammar?" into a checkable obligation.