UnifiedIR for Julia

Overview

WIP/RFC: Unify all our various IR data structures into one #62334. Draft. Keno wants to merge 45 commits into Draft.

UnifiedIR is one IR data structure intended to span the compiler pipeline and external compilers (design specification in UnifiedIR/docs/design.md):

• A flat statement table with hybrid regions over a shared storage core (AttrGraph: kind column + packed two-mode operand words + one tagged operand pool + open attribute-column universes)
• Layout states (builder/dense/editable/floating) with exactly two renaming points
• A namespaced kind registry with statically reserved dialect ids for the bootstrap stack
• A generic tree porcelain (Tree/NodeList cursors, construction, mapchildren/copy_ast, provenance walks over graph-qualified :source chains)
• compact!-as-GC (compact_graph!/collect_syntax!), verifier, printer/parser, and a reference interpreter

Zero dependencies.

Wiring: UnifiedIR joins TOP_LEVEL_PKGS (symlinked to the Julia share directory), is bootstrapped into Base immediately before JuliaSyntax (base/Base.jl) - JuliaSyntax's SyntaxGraph runs on this substrate - and its sources join COMPILER_FRONTEND_SRCS so sysbase rebuilds on change. The Base-baked instance keeps its kind registry clean of the test dialect (registration is gated to package mode).

Co-Authored-By: Claude Fable 5

Storage Core

SyntaxGraph wraps UnifiedIR.AttrGraph:

• kind is a real core column (the :kind attribute is a Dict-shaped view of it)
• Children are STMT-tagged words in the shared operand pool addressed by packed range words
• The historical edge_ranges/edges/attributes properties are preserved exactly (identity-stable views; is_compatible_graph semantics kept)

compact_graph! gives lowering the GC over syntax graphs it always wanted.

Porcelain

SyntaxTree{Attrs} = UnifiedIR.Tree{SyntaxGraph{Attrs}} and SyntaxList = UnifiedIR.NodeList:

• Child indexing, attribute properties, newnode/newleaf/mknode/mkleaf/mktree, copy_attrs!/mapchildren/copy_ast, provenance walks, structural isapprox and printing are UnifiedIR generics
• JuliaSyntax keeps only the genuinely syntax-specific parts (the Kind registry wrapper, SourceRef/source-text machinery, the leaf payload convention, parser integration, prune/unalias/@stm)

Kind Registry

The historical module-id mechanism re-plumbs into the shared UnifiedIR registry through register_kinds!:

• Kind modules become dialects claiming contiguous opcode blocks (range predicates and BEGIN_/END_ markers preserved)
• The bootstrap stack claims statically reserved dialect ids: JuliaSyntax=1, JuliaLowering=2, formatter=3; core=0 - so K"..." literals stay compile-time constants
• Syntax K"call" and core K"call" are distinct kinds sharing one numbering space
• Kinds re-register in __init__ (the registry is UnifiedIR session state); numbering is deterministic, keeping baked constants valid

Dual-Mode

Bootstrapped into Base it binds Base.UnifiedIR (included just before it); as a package it depends on the UnifiedIR package.

Co-Authored-By: Claude Fable 5

JuliaLowering.UnifiedBackend

Additive; zero changes to existing sources. Reuses the front half unchanged - macro expansion, desugaring, scope analysis, closure conversion - and emits structured UnifiedIR region form directly (if/loop/try region ops, cells for frame variables, sealed exit terminators) instead of goto linear IR, with graph-qualified provenance:

• Every emitted statement carries a :source column entry holding the originating syntax tree cursor, so the generic provenance walk crosses from optimized IR statements back to surface text (highlightable diagnostics)
• collect_syntax! can GC the lowering graph against live IR

Pure definitions; nothing calls it during bootstrap, and it bakes cleanly in the optional sys-JL sysimage stage. Bindings resolve through the enclosing JuliaLowering (same JuliaSyntax and hence same UnifiedIR instance in both Base-baked and package modes).

Co-Authored-By: Claude Fable 5

Compiler Integration

The provider side of the substrate (package-mode ONLY; the sysimage bootstrap never sees it):

• CodeInfo ↔ UnifiedIR boundary converters (cfg-wrap entry incl. exception handlers; goto/typed exits with phi synthesis)
• An inference port running natively on UnifiedIR (differentially validated at 100% equal-or-better return types against the stock pipeline on session MethodInstances)
• Optimizer passes (SROA, inlining via splice_body!, ADCE, structurization, cell promotion)
• The Queries API
• Activation through the ordinary compiler-replacement mechanism (Core.OptimizedGenerics.CompilerPlugins.typeinf owner hook + with_unified_compiler), with every cached result round-tripped through a verified UnifiedIR shadow

Loading: Compiler.load_unified!() includes the port on demand into a Main-rooted carrier module and binds it as Compiler.Unified. On demand because the stdlib stub fast-path bypasses module evaluation when the package matches the sysimage copy; Main-rooted because this baremodule rebinds getproperty to raw getfield for its nested tree, which would break property-forwarding types. The plugin overload pins invoke_in_world to an end-of-module sentinel world so the shadow hooks are visible under both pkgimage and interpreted loading.

Tests in Compiler/test/unified/.

Co-Authored-By: Claude Fable 5

Tests and Demo

• UnifiedIR/test wired into the test harness (choosetests + the runtests_vendored project-activation pattern of the other top-level packages): substrate, porcelain, kind registry, provenance walk, compact!/collect_syntax! GC, acceptance/fuzz/splice suites.
• UnifiedIR/demo/provenance_demo.jl - the one-stack demonstration under the built binary: parse → JuliaLowering front half → UnifiedBackend.lower_to_ir (graph-qualified :source cursors) → Compiler.Unified inference + UnifiedIR optimization → highlight() of the exact surface text an optimized statement came from → print_ir with per-statement source excerpts → collect_syntax! (conservative and prune policies) → the same highlight, byte-identical after AST GC.
• NEWS.md entry under compiler/runtime improvements.

Co-Authored-By: Claude Fable 5

Discussion

Member | I guess one thing. Does some form of structured control flow make sense here? Downstream having it has been quite useful. But the passes that structurize it are also quite complete and it's not super useful for lowering to LLVM

Member Author | Yes, the control flow is structured in this representation (with an escape hatch that claude is abusing a bit at the moment, but I'll clean that up).

Printing Improvements

typed_ir returned the IR handle but its REPL display was the compact one-liner, which is useless for exploration. IR now shows as the full print_ir listing under MIME text/plain by default; truncation is opt-in, globally via display_maxlines!(n) or per-stream via IOContext(io, :ir_maxlines => n), with a tail note. Core.Const lattice types render as Const(value) instead of the opaque lattice fallback.

@code_unified is the @code_typed-shaped entry point: argument values or bare ::T annotations, optimize=false for inference-only IR.

Co-Authored-By: Claude Fable 5

Member Author | Try at home example:
(Printing subject to improvement)

Yield → Result Rename

In Julia, yield reads as task suspension (Base.yield), and the name should stay available for genuine suspension points in the eventual await/continuation form (design doc §5.6). The region-value terminator is now result, pairing with region_arg: regions take region_args and feed their owner through result terminators. Mechanical rename across the core dialect, printer/parser, converters, optimizer, the lowering backend, and the design document; no semantic change.

Co-Authored-By: Claude Fable 5

Cell Promotion Passes

Promote Arm Cells

Add promote_arm_cells!, the if-join leg of §6 cell promotion, by store sinking: conditional stores in sibling arms become per-arm result values plus one unconditional post-join cell_set (extract-indexed when several cells tuple through one if), with the incoming value materialized as a cell_get before the if under definite assignment. Diverging arms contribute nothing; handler-observable, maybe-undef and token cells refuse per the §6 inviolables. In-arm reads reached by arm stores are rewritten through the reaching store, re-read at transform time so the rewrite is order-invariant across cells.

optimize_ir! now runs arm and loop promotion to a joint fixpoint: arm sinking produces exactly the unconditional store shapes the dominating and loop passes consume, so each round can expose new cases for the others. This makes the conditional-swap shape in Base.gcd (a, b = b, a under an if) promote fully: typed_ir(gcd, (Int,Int)) retains zero cell ops, and the corpus optimizer wall drops ~18% (the earlier promotion shrinks the IR for every later round). The passes record their join placements in PROMOTION_TRACE when the completeness harness asks for them.

Island Block Promotion

Add the fourth §6 join-point class: liveness-pruned, IDF-placed SSA construction for frame cells over cfg-op block graphs. Phis are block region args; incoming values ride the edge bundles of block terminators and of sealed exits of nested islands that land on our blocks mid-block (§5.5) - a mid-block edge exports the reaching definition at the sealed sub-op's member position, which stays well-defined through handler-nested exits because candidate stores are all direct block members and cannot execute mid-try. The incoming value is a cell_get inserted before the cfg op under definite assignment, folded by the dominating pass next round. Entry-leading cell_new (NewvarNode form) is honored by invalidating the incoming value.

Soundness rule found by the DF harness: when a loop body lies strictly between the cfg op and the cell declaration, cell memory is carried across iterations through the island's sealed continue/break exits, so deleting island stores would leave the next iteration's incoming read stale - such cells refuse unless the island is store-free for them. Dissolving that class needs exit-value threading through sealed exits (deferred; the dominant remaining stock-oracle exception).

promote_arm_cells! now also fires inside island blocks - its transform is local to the if's region subtree, and its post-join store is exactly what the island and block passes consume - and optimize_ir!'s promotion fixpoint includes the island pass. Stock-oracle violations on the Base/stdlib corpus drop from 38 bodies to 16, all in the documented exit-threading class; corpus cell-op elimination rises from 20% to 55%.

Verification Harness

Substantiate that the §6 join vocabulary - if results, loop region args, loop break values, island block args - projects the iterated dominance frontier onto the region tree, three ways:

1. classify_residual_cells: every cell surviving the isolated promotion fixpoint (promotion_fixpoint!, the five promotion passes to a joint fixpoint plus dead-cell dropping) carries a documented exception class (escape/token, throw-edge/handler, maybe-undef, island exit-threading, refused multi-level exit); UNCLASSIFIED is a completeness bug. The stock oracle demands the classes match stock's own outcome: where code_typed fully mem2reg's (slot-, Box-, and undef-guard-free), no stray residuals may remain.

2. CellFuzz: a structured fuzzer emitting interpretable core-dialect bodies with random nested if/loop/try, sibling-arm swaps, multi-level exits, guarded maybe-undef reads, and handler interactions; each case runs a semantic differential (same values, same thrown errors) across the fixpoint, plus classification totality. 10k cases clean.

3. df_correspondence: flatten the pre-promotion body through the exit converter, compute stock slot2ssa's liveness-pruned IDF per cell, and check the recorded promotion placements (PROMOTION_TRACE, id-translated round-by-round through the compaction RemapSets, region ids for island phis) land exactly on those blocks; statically dead code is excluded on both sides. Zero missing placements on the corpus and fuzz samples. The DF leg caught two real bugs during bring-up: an order-dependent stale-operand rewrite in the arm pass and the island backedge-staleness unsoundness - both fixed in the pass commits.

bench/unified_completeness.jl runs all three legs at scale (600-body corpus, 10k fuzz, seeded); test/unified/completeness.jl carries the regression shapes: the gcd swap, nested depth 3, all-diverging-but-one, maybe-undef and throw-edge refusals, island phi placement and the staleness refusal, and a seeded 800-case battery.

The claim, the five promotion passes and their IDF projection, the backedge-staleness rule, the machine-checkable exception classes, and the three-legged verification harness (classifier + stock oracle, structured differential fuzzing, dominance-frontier correspondence).

Indexing Fix

The extract index was 0-based (an LLVM extractvalue habit). This is Julia: the index is now exactly getfield's, so the extract → getfield mapping at every boundary (canonicalization, converters, tfunc) is the identity instead of a ±1 that each site had to remember. Producers, consumers, tests, and the design document flipped together.

Island Store Dominance

promote_cells! refused any cell touched inside a cfg island wholesale. The dominating case is region-local, so the refusal was stronger than needed: dominates_for_cell only walks region nesting, meaning a store proves dominance either from outside the island (it executes before the whole cfg) or from the same block subtree (blocks execute their members in order, and re-entry re-executes from the top, so the store re-executes before the get on every iteration). Cross-block flow never establishes region-static dominance, so nothing unsound gets through. The one genuine island hazard is a positionally-later store reaching an earlier get across island back edges - invisible to shares_loop, which only knows structured loop bodies. The blanket refusal was load-bearing for exactly that case; it is