astero¶

astero is a Python library for writing compilers, transpilers and code analysers. You declare your language's syntax tree once; astero computes the name handling and code generation that such a program otherwise writes, and maintains, by hand.

Python's grammar ships with the library, so for Python source there is nothing to declare.

pip install abilian-astero        # or: uv add abilian-astero

The distribution is called abilian-astero, because PyPI already had astero. The import name is astero:

from astero.lang_py import PY, VARS

Python 3.11 through 3.15. No runtime dependencies. Apache-2.0.

The problem, in one example¶

Rename a variable throughout a module. This is the version that gets written:

import ast

NEW = {"x": "y"}
SOURCE = "def f(x):\n    import os as x\n    try: pass\n    except E as x: g(x)\n    return x\n"


class Renamer(ast.NodeTransformer):
    def visit_Name(self, node):
        node.id = NEW.get(node.id, node.id)
        return node

    def visit_arg(self, node):
        node.arg = NEW.get(node.arg, node.arg)
        return node


print(ast.unparse(Renamer().visit(ast.parse(SOURCE))))

def f(y):
    import os as x          # not renamed
    try:
        pass
    except E as x:          # not renamed
        g(y)                # now reads a name nothing binds
    return y

Two of the four binding positions were missed. The result still parses. It means something else.

The same job with astero:

from astero.hygiene import rename
from astero.lang_py import PY, VARS

print(ast.unparse(rename(ast.parse(SOURCE), NEW, PY, VARS)))

def f(y):
    import os as y
    try:
        pass
    except E as y:
        g(y)
    return y

Fifteen positions in Python can hold a variable name. The set moves between releases: 3.12 added three. rename reaches all of them because a declaration lists them, not because anyone remembered to.

Why that generalises¶

Renaming is the smallest visible case. The same fact, which fields introduce a name, is needed in several other places. Hand-writing it means several enumerations that have to agree with each other and with the language:

The first three are defects that shipped, in three different compilers: **kwargs never bound, an inliner substituting at every Name, and an AugAssign rewrite setting the context by hand. None of them crashed.

In astero they are one query, positions(Kind.DEF, ns), read by four modules: scopes, hygiene, rewriting and generate. Add a binding form to the language, edit the declaration; the four follow.

That is the shape of the library. Structural facts are declared once and consumed many times. Names are the clearest case; the same holds for which fields a traversal must enter, which productions can appear in a tree, and how tightly each operator binds.

What a declaration buys¶

You declare a role for each field of each production: whether it introduces a name, refers to one, or merely holds a subtree. For Python that declaration ships with the library; for your own IR it is one table, read off your dataclasses.

What reads it:

Two worked compilers ship with the docs: TinyPy, a Python subset with two back ends, and PL/0, Wirth's textbook language, built on a tree of plain dataclasses with no relationship to ast.

What astero is not¶

• Not a parser. Bring your own AST. astero starts from a tree.
• Not a code generator you run. There is no astero build; nothing is generated on disk. It is a library you import; every entry point is a function call at run time.
• Not a framework. It does not own your pipeline, your IR, or your main. You can adopt one query in one function and leave everything else alone.

This is a narrower scope than compiler toolkits such as Cocktail or Eli, which generate a compiler's components from declarations ahead of time. astero answers questions about a declaration at run time, which is why adoption can be incremental.

Examples¶

Each of these runs as written. The outputs in the comments are asserted by tests/a_unit/test_tutorial.py.

Ask the grammar for a table¶

from astero.lang_py import PY, VARS

PY.ident_slots(VARS)["arg"]  # ('arg',)      — every slot a renamer must reach
PY.definitions(VARS)["For"]  # ('target',)   — positions that introduce a name
PY.definitions(VARS)["FunctionDef"]  # ('name',)

ident_slots(VARS) returns the full mapping: on Python 3.12 and later, 15 productions holding a variable name.

Rewrite a tree without touching ctx¶

import ast
from astero import Pass, rules
from astero.lang_py import PY

DESUGAR = Pass(
    "desugar",
    rules("ast.AugAssign(_t, _o, _v) => ast.Assign([_t], ast.BinOp(_t, _o, _v))"),
    eliminates=(ast.AugAssign,),
    grammar=PY,
)

ast.unparse(DESUGAR(ast.parse("b[1] += 5")))  # 'b[1] = b[1] + 5'

In Python's AST every Name and Subscript carries ctx=Load() or ctx=Store(). A rewrite that moves a node between positions has to update it. The rule above does not mention ctx: astero reads each field's role from the grammar and sets the context to match the position the node ends up in. The same rule therefore handles x += 5 and b[1] += 5, where the target must become Store in one place and Load in the other.

eliminates= is a postcondition. If any AugAssign survives the pass, it raises and will not pass a half-desugared tree downstream.

Substitute without capturing a name¶

from astero.hygiene import Fresh, substitute

body = ast.parse("[n + k for k in xs]", mode="eval").body
substitute(body, {"n": ast.parse("k", mode="eval").body}, PY, VARS, fresh=Fresh())
# [k + _t1 for _t1 in xs]

Replacing n with the expression k would put a free k inside a comprehension that binds its own k, silently changing what the code means. substitute detects the collision and, given a Fresh name supply, renames the comprehension's binder to _t1 first. It also replaces only uses of n: a position that binds n is left alone, because substituting there produces something that is not a program.

Write a code generator as a table¶

from astero.emit_rules import All, Both, Emitter, OpIs


def type_of(node):
    """Your type inference. A stub, so the example runs."""
    return int if isinstance(node, ast.Constant) else None


js = Emitter(PY, typer=type_of)
js.projection("name", str)
js.projection("lit", str)
js.rule("BinOp", "{left} + {right}", All((OpIs((ast.Add,)), Both((int,)))))
js.rule("BinOp", "add({left}, {right})", OpIs((ast.Add,)))
js.rule("Name", "{id:name}")
js.rule("Constant", "{value:lit}")

js.to_text(ast.parse("1 + 2", mode="eval").body)  # '1 + 2'
js.to_text(ast.parse("a + b", mode="eval").body)  # 'add(a, b)'

A rule is a production name, a template, and an optional guard. Rules are tried in order and the first whose guard holds wins, so the specialised case sits above the general one. {left} emits that field and brackets it if the precedence table says it needs brackets.

typer is how the emitter asks your type inference a question; it is the only hook into host analysis.

Fail the build when a production has no handler¶

from functools import singledispatch

from astero.coverage import dispatch


@singledispatch
def gen(node: ast.AST) -> str:
    raise NotImplementedError


@gen.register
def _(node: ast.Assign) -> str:
    return "assign"


@gen.register
def _(node: ast.Return) -> str:
    return "return"


cover = dispatch(
    PY,
    (gen.registry,),
    bases=("stmt",),
    accounted={"desugared": {"AugAssign", "AnnAssign"}},
)

cover.explain().splitlines()[0]  # '2 of 28 productions handled'
sorted(cover.missing)[:3]  # ['Assert', 'AsyncFor', 'AsyncFunctionDef']

dispatch compares the productions your singledispatch registries handle against the productions the grammar declares. In a real project you would assert not cover.missing, cover.explain() in a test.

accounted names the productions you deliberately do not handle, with the reason as the key. A stale reason, one whose production no longer exists or has since gained a handler, shows up in cover.absent and cover.redundant, not in cover.missing.

Where variable names appear in Python's AST¶

Take one concrete job: renaming a variable throughout a Python module. You need every position where a variable name can appear. On Python 3.12 and later there are fifteen:

AsyncFunctionDef.name    Global.names        Name.id            TypeVar.name
ClassDef.name            MatchAs.name        Nonlocal.names     TypeVarTuple.name
ExceptHandler.name       MatchMapping.rest   ParamSpec.name     alias.name, alias.asname
FunctionDef.name         MatchStar.name                         arg.arg

Three observations a hand-written list tends to miss:

• The list is version-dependent. Python 3.11 has twelve of these. ParamSpec, TypeVar and TypeVarTuple arrived with PEP 695 in 3.12. Code that hard-codes the list is correct for one interpreter.
• The unobvious entries are the ones omitted. ExceptHandler.name is the e in except E as e. alias.asname is the y in import x as y. Global.names holds a list of names. The four Match* patterns bind too.
• Namespaces matter. ident_slots() without one returns two more productions, Attribute.attr and keyword.arg. Those are identifiers, but renaming them changes x.foo into a different attribute and f(key=1) into a different call. Passing VARS asks the narrower question.

PY.ident_slots(VARS) computes the answer from the interpreter you are running on, so all three problems go away together.

How a grammar is declared¶

A grammar is a set of productions, each with named fields. Every field carries three things:

• a shape: ONE, OPT or SEQ, i.e. whether the slot holds one node, an optional node, or a list;
• a sort: what the slot holds, another production, a bare identifier, or plain data;
• a role: what the position does with what it holds.

Roles have no equivalent in ASDL, the notation Python's own AST definition is written in. Every query above is computed from them:

child       structural containment; a traversal descends into it
attr        plain data; never traversed
def(ns)     introduces a name in namespace `ns`
use(ns)     refers to a name in `ns`
defuse(ns)  both, for read-modify-write positions
del(ns)     removes a name
declare(ns) `global` and `nonlocal`: names a binding that lives elsewhere

Two rules govern how roles are written, both explained in Roles. A role belongs to the parent's field, describing a position the child node occupies. Namespaces then distinguish positions that hold the same type but mean different things.

You can declare a grammar three ways: lang_py.build() if your AST subclasses Python's, grammar.from_dataclasses() if your IR is annotated dataclasses, or GrammarBuilder by hand. All three are in the API reference.

How the derivations are checked¶

Each derivation is compared against an independent source of truth over the Python standard library:

These are integration tests, run on Python 3.11 through 3.15. The exact counts depend on which standard library you measure, so the tests hold the ratio. A regression fails; an improvement does not have to be chased.

Where to go next¶

• Getting started: install it and run three queries against real code.
• Tutorial: build a working compiler for a Python subset, with two back ends, in about 170 lines.
• User guide: every module, its API, and when to use it.
• Adopting astero: fitting it into a compiler you already have.