astero.python.rewriting¶
Rules over trees, with ctx recomputed from the grammar's roles.
astero.python.rewriting ¶
Rewrite rules over Python ASTs.
A pass is a named rule set plus a traversal strategy. A rule is one string, with
=> between the pattern and its replacement:
rule("hypot(_x, _y) => sqrt(_x ** 2 + _y ** 2)")
Several at once, one per line:
rules('''
exp(_x) => e ** _x
expm1(_x) => exp(_x) - 1
''')
Both sides are Python source. Identifiers spelled _[a-z]... are
metavariables: in expression position one binds a node, in an identifier field
(Attribute.attr, arg.arg, FunctionDef.name) it binds a string. A
metavariable used twice must bind equal subtrees. _ on its own matches
anything and binds nothing. *_xs binds the rest of an argument list and
**_kw the rest of a keyword list, with *_ and **_ for the anonymous
cases, so exp(_x, **_) reads as "exp of one argument, whatever the keywords".
Where concrete Python syntax cannot say what a rule means (an operator hole, a
node constrained on some of its fields), write abstract syntax with an explicit
ast. prefix, positionally in field order or by keyword:
rule("ast.AugAssign(_t, _o, _v) => ast.Assign([_t], ast.BinOp(_t, _o, _v))")
An abstract term constrains only the fields it names.
A rewrite no template can express, typically a fold over a list of arbitrary
length, carries its pattern on the function that performs it, and joins the
set through rules:
@rewrite("ast.BoolOp(_o, _v)")
def nest(m):
... # return the replacement, or None to decline
rules('''
-_x => 0 - _x
''', nest)
Rules never mention ctx or source positions. Both are derived from the shape
of the tree and reinstalled after rewriting.
print(some_pass) lists its rules. A rule that fails to compile names where it
was written, and each Rule keeps that location in origin.
Strategy ¶
Bases: Enum
Where in the traversal a rule set is applied.
INNERMOST: children first, then re-apply at the node until nothing fires. TOPDOWN: apply once at the node, then continue into the result. OUTERMOST: apply at the shallowest matching nodes and stop there.
Term
dataclass
¶
Abstract-syntax pattern: matches cls, constraining only fields.
Match
dataclass
¶
What a rule's right-hand side and guard get to see.
new ¶
Build a node of the grammar's class for cls's production.
Pass
dataclass
¶
A named rule set, a traversal strategy, and a postcondition.
key ¶
Structural key over a tree, skipping ignored fields.
match ¶
Match node against pattern pat; return bindings, or None.
copy_tree ¶
copy_tree(node: Any, grammar: Grammar | None = None) -> Any
Copy a node along its declared fields and source positions.
copy.deepcopy follows every attribute an object has, which is wrong for a
syntax tree that a host has decorated. prescrypt-ng's converter gives each
node a _parent back-pointer, so deep-copying any node copied the entire
module, and the whole enclosing CPython tree with it through _orig_node.
Desugaring one 400-line file took 1.6 seconds and grew faster than the
square of its size.
A node's parts are what the grammar says they are. Everything a host hangs off a node besides that is its own bookkeeping, and a pass that rebuilds the syntax is not entitled to guess whether it still applies.
Pass grammar for a tree that is not Python's. Without it the fields
come from node._fields, which only a Python AST has, so a node of any
other kind cannot be copied here and is refused by name rather than handed
back — returning it unchanged aliases the original, and a caller that then
mutates its "copy" has mutated the input. ast.walk in hygiene and
ast.iter_fields in scopes made the same mistake and answered emptily,
which is wrong but harmless; this one corrupted.
The two paths are not two thoroughnesses of one copy, and the parameter is
not an optimisation. They produce different trees on purpose. A
declared copy carries what the grammar declares, and the declaration omits
what is derived, so copy_tree(t, PY) returns a Name with no ctx and
a FunctionDef with no type_comment: correct for a pass that runs
fix_contexts afterwards, uncompilable for one that does not. Choose by
which tree you want, not by which is safer.
build ¶
build(
tpl: Any,
binds: dict[str, Any],
grammar: Grammar | None = None,
) -> Any
Instantiate template tpl with binds, in grammar's node classes.
rule ¶
rule(
text: str,
rhs: Rhs | None = None,
when: Callable[[Match], bool] | None = None,
origin: str = "",
) -> Rule
Compile a rule.
With one argument, text is "pattern => replacement". With two, text is
the pattern and rhs is DELETE or a function of a Match, the escape
hatch for rewrites a template cannot express.
origin names the rule in compile errors, and defaults to the call site.
rewrite ¶
rewrite(
pattern: str,
when: Callable[[Match], bool] | None = None,
) -> Callable[[Callable[[Match], Any]], Rule]
Declare a rule whose replacement is the function it decorates.
A rewrite a template cannot express still wants its pattern next to the code that carries it out, rather than a name pointing somewhere else:
@rewrite("ast.BoolOp(_o, _v)")
def nest(m):
"`a and b and c` becomes `a and (b and c)`."
values = list(m.node.values)
if len(values) <= 2:
return None
...
Returning None declines, and the next rule is tried, exactly as a guard
that returns False. So a condition the function has to compute anyway is
written once, inside it, and when is for a guard worth naming.
rules ¶
Compile one pattern => replacement per non-blank, non-# line.
also takes rules already compiled, which is how a @rewrite function
joins the set without the whole thing splitting into two shapes spliced
together. They are tried after the block, in the order given.
Each rule's origin is the call site plus its line within the block, since the absolute line of a string literal is not recoverable from the caller.
target_fields ¶
target_fields(
grammar: Grammar,
) -> dict[str, tuple[tuple[str, ...], type]]
Production name -> (fields holding a target, the context it implies).
Derived, never authored. This used to be a literal table, and it silently
fell behind Python 3.12: TypeAlias.name is a binding position, so
type X = int was given a Load context where CPython says Store.