Speeding Up a Python Service with CinderX: JIT and Static Typing

Overview

Every Python optimizer claims a speed number, measured on kernels, sorting, tree traversal, and arithmetic in a loop. This service works differently: the handler queries the database, performs calculations in NumPy, serializes the response, and the bytecode to which this number applies may be almost entirely absent from it. CinderX accelerates the bytecode, and this number does not extend beyond the bytecode. The proportion of bytecode in your service is a property of your code, not the extension, and until it is calculated, the decision of "whether or not to use it" is based on guesswork.

What Is CinderX?

CinderX is a CPython extension: a binary package that is installed into an existing interpreter. It replaces the frame evaluator (the function that CPython calls to execute each frame), and execution proceeds through it from that point on. Inside:

• JIT - compiles the entire object code into machine code.
• Static Python - a separate compiler for a typed subset of Python. It generates its own opcodes, which the JIT then translates into machine code.
• Parallel garbage collector - parallelizes the two garbage collection phases.
• Lightweight frames - in a compiled frame, only the subset of fields required by the machine code itself is filled in. The rest is completed if the runtime requests a full frame. Works in conjunction with the JIT.
• Library primitives, typed containers, and primitive equivalents of built-in functions.

JIT

The stock CPython 3.14 already includes a JIT (--enable-experimental-jit from PEP 744, also known as tier 2). So the question becomes: why do we need another one? These are two completely different compilers.

CinderX's Approach

CinderX is a binary package that sits inside an existing interpreter. It replaces the frame evaluator, and execution proceeds through it from that point onward. The key components are:

• JIT - compiles the entire object code (up to 800 UOPs) into machine code.
• Static Python - a separate compiler for a typed subset of Python that generates its own opcodes, which the JIT then translates.
• Parallel garbage collector - parallelizes the two garbage collection phases.
• Lightweight frames - only fill in the subset of fields required by the machine code itself; the rest is completed on demand if the runtime requests a full frame.
• Library primitives - typed containers and primitive equivalents of built-in functions.
• HIR - a linear sequence of UOPs (no intermediate values located in machine registers).
• LIR - a fine-grained abstraction layer above assembly, also in SSA.
• Machine code generation - via asmjit, using pre-assembled stencils and patches for hole handling.

How the JIT Works

During the CPython build phase, Clang compiles a stencil template once for each micro-operation (UOP). At runtime, _PyJIT_Compile receives a "track," copies the desired stencil into executable memory, and fills the holes with real values. The chaining is held together by tail calls. Each stencil ends with:

#define PATCH_JUMP(ALIAS)
do {
    PATCH_VALUE(jit_func_preserve_none, jump, ALIAS);
    __attribute__(("musttail")) return jump(frame, stack_pointer, tstate);
} while (0)

This creates a trace-a chain of functions that jump to the next without growing the stack. The call convention preserve_none allows frame, stack_pointer, and tstate to be stored in registers throughout the entire chain. Values are passed between UOPs via the frame value stack (in memory).

What CinderX Doesn't Do

Adjacent UOPs don't share a common register. Intermediate values are passed through the frame stack in memory. Operations on the reference counter aren't reordered based on liveliness. The abstract interpreter in optimizer_analysis.c runs data-flow analysis along the path and eliminates provably redundant, duplicate type version checks-but these limitations exist even with CinderX.

What You Need to Build It

To build CinderX, you need:

• LLVM 19
• Clang only (CPython)

The build involves copying and patching the CPython tree. The key artifact is Tools/jit/template.c, which contains the body of a single UOP (micro-operation) wrapped in a function with the signature (frame, stack_pointer, tstate). Runtime everything varies are declared as holes with descriptive names: _JIT_OPARG, _JIT_OPERAND0, _JIT_TARGET, _JIT_CONTINUE.

During the CPython build phase, Clang compiles this stencil once for each UOP and assembles the resulting pieces into a stencil table. This requires LLVM 19 with clang because the template relies on __musttail, which GCC does not support.

Copy-and-Patch Mechanism

The idea behind "copy-and-patch" is to avoid having a compiler at runtime altogether. The process works as follows:

1. Copy - Clone the CPython source tree.
2. Patch - Modify the relevant files (e.g., Tools/jit/template.c) to define a stencil for each UOP.
3. Build - Compile the stencil once for each UOP using Clang.
4. Assemble - Combine the resulting machine-code pieces into a stencil table.
5. Run - At runtime, _PyJIT_Compile receives a track, copies the desired stencil into executable memory, and fills the holes with real values.

This approach avoids parsing, analysis, and instruction selection at runtime-hence it is extremely cheap. The chaining is held together by tail calls, creating traces that are chains of functions jumping to the next without growing the stack.

Benefits Achieved

• Disappearance of the dispatch loop - The traditional Python dispatch loop with its switch and unpredictable branches is eliminated.
• Abstract interpreter optimizations - The analyzer runs data-flow analysis along the path and removes provably redundant, duplicate type version checks.

Limitations

• Adjacent UOPs don't share a common register.
• Intermediate values are passed through the frame stack in memory.
• Operations on the reference counter aren't reordered based on liveliness.

CinderX: A True Compiler Method

A unit in CinderX is an entire object code-a function, a method, or a lambda. Unlike traces, units are fully compiled, so there is no need to guess which branches to include.

Preloader Phase

This is a separate phase-the only place in the entire pipeline where Python execution is permitted. It resolves global names under LOAD_GLOBAL, determines types, and resolves call targets for Static Python opcodes in advance. Because any execution of Python during compilation can break the compiler's assumptions, this phase runs sequentially under the GIL, after which worker threads compile in parallel.

Bytecode → HIR

The CPython virtual machine is a stack-based machine, and its bytecode is written accordingly: an instruction takes its operands from the top of the stack and places the result there as well. To eliminate unnecessary work, the compiler must first prove that certain work is unnecessary-and any such proof ultimately boils down to the question, "Where did this value come from, and who is using it?"

The goal is to rewrite bytecode into a form where every value has a name, every instruction has explicit inputs and outputs, and every value also has a type. This is HIR (High-Level IR):

• Not built from pure bytecode - it combines bytecode with what the CPython adaptive interpreter left behind.
• Specialized opcodes - when the interpreter observes that + b always adds integers, it rewrites the opcode to a specialized BINARY_OP_ADD_INT.
• 22 specializations - arithmetic on int, float, and str; indexing of lists, tuples, and dictionaries; comparisons, unpacking sequences, and LOAD_ATTR_MODULE.

HIR Traversals

After construction, the HIR undergoes:

1. Conversion to SSA (Static Single Assignment) - each name is assigned exactly once, making "where the value came from" a single-link traversal rather than a graph search.
2. Type inference - determining the type of each variable.
3. Optimization pipeline - including simplification, elimination of dynamic comparisons, removal of redundant type checks, elimination of φ-nodes, inlining, and cleanup of the control flow graph.

The inliner operates on a budget: the cost of a function is the number of real opcodes, with a default limit of 2,000. Once the budget is exhausted, inlining stops.

Refcount Insertion

Runs near the very end because each previous pass moves and discards instructions. The compiler sets up reference counting based on metadata about how each HIR instruction handles references and memory. Incref and decref are treated as regular instructions and can be eliminated in pairs.

HIR → LIR → Machine Code

There is no direct mapping between HIR (Python operations) and machine code. The allocator needs to know machine constraints (register availability), so a form where instructions are already machine code but still have sufficient registers is necessary. This is LIR (Low-Level IR), a fine-grained abstraction layer above assembly, also in SSA. The register allocator distributes registers in a single linear pass through the function, splitting lifetimes of values if there aren't enough registers. Finally, x64 code is generated via asmjit.

Deoptimization

Since the code relies on security checks, a deoptimization path is needed. The interpreter's state is represented in HIR by objects like FrameState (offset of current opcode, operand stack contents, block stack). The Snapshot instruction locks the state at a point in the program. Guard takes a Boolean operand and inherits FrameState from the nearest dominant snapshot.

In machine code, Guard is a comparison followed by a jump to a placeholder. There are three levels of placeholders:

• One per check (stores metadata index)
• One per function (substitutes runtime address and address of actual epilogue)
• One jump to the runtime

When a safety check fails (from two instructions within a function to a return to the interpreter), the system falls back to the interpreter's state via resumeInInterpreter.

What Each Component Cleans Up

Performance Results

The author recorded 400 consecutive calls to a single function ("Life") on a 96×96 grid:

• Blue line (stock CPython 3.14): 41.9 ms after 250 calls, then compilation kicks in at call 251.
• Green line (stock CPython 3.14 with Tier 2 JIT): 10.46 ms total.
• Stock CPython 3.14 with PYTHON_JIT=0: 11.54 ms total.

On the same binary with PYTHON_JIT=0, the Tier 2 version runs at 10.46 ms (1.14-fold improvement over stock). With PYTHON_JIT=0, the Tier 2 version takes 11.54 ms (1.10× slower than stock).

These benchmarks show that CinderX provides meaningful speedups when the threshold is met, though the absolute gains depend on workload characteristics. The key insight is that the benefit is tied to the proportion of bytecode in the service-not the extension itself. Until the threshold is calculated, decisions about whether to use the JIT are based on guesswork.