Show HN: Lucen a Python compiler that parallelizes for-loops via comment pragmas

Lucen is a source-to-source compiler and automatic loop parallelizer for ordinary Python, driven by comment pragmas. Unlike existing Python parallel frameworks, it asks you to describe where parallelism is allowed rather than how to implement it, and it parallelizes only the loops it can prove are both safe and worthwhile. Its one guarantee has no tier and no opt-out: Lucen never produces an incorrect result.

Before, ordinary Python:

for i in range(len(records)):
    scores[i] = score(records[i])

After, still ordinary Python:

# LUCEN START
for i in range(len(records)):
    scores[i] = score(records[i])
# LUCEN END

3.8x faster on 12 cores (CPython 3.14, CPU-bound map, measured). Bit-identical output, floats included. No multiprocessing code. No pools, no locks, no pickling errors to debug. No risk of adopting it: a loop Lucen cannot prove safe runs exactly as the sequential Python you wrote, and a structured report tells you why.

Activation is one call at program start:

import lucen
lucen.activate()

The pragmas are ordinary comments. A file with Lucen removed, uninstalled, deactivated, or never present runs identically to one where it never existed. We call this the Comment Invariant, and it is load-bearing: the worst case of adopting Lucen is the program you already had. It covers the other case: the loop already sitting in a codebase that nobody wants to restructure. No worker functions, no pool lifecycle, no serialization plumbing, no rewrite into a framework's shape. Two comments slide into existing code without a hiccup, and adoption is reversible by deleting them.

• Contents: Guarantees | Install | Tutorial | Beyond the basics | Expert guide | How it works | Performance | Limitations | The honesty contract | Documentation | Contributing | License

Guarantees

• Never an incorrect result. Chunks write private slabs, audited for disjointness at join and committed in chunk order. Dict insertion order, float reduction bits, and mid-error container state are identical to sequential execution, bit for bit. A write conflict discards the parallel attempt and transparently re-runs your loop sequentially.
• Never disruptive. Anything Lucen cannot prove safe runs as the sequential Python you wrote, and the reason lands in a structured fallback report instead of your stderr. Exceptions keep their type, their message, and the exact sequential-prefix state of your containers.
• Never silently pointless. A profitability gate (static pre-screen plus a runtime probe that does real work while measuring) refuses to parallelize loops that would lose to dispatch overhead, and reports that too. Parallelism you cannot observe is a bug here, not a shrug.

These are not aspirations. They are enforced by construction and verified by a cross-version test matrix: 7 interpreters x 8 workloads x 4 execution pathways, every cell bit-identical to plain Python. See BENCHMARK.md.

Installation

pip install lucen

Python 3.9 or later. No required dependencies on 3.11+; on 3.9 and 3.10 the TOML parser dependency (tomli) installs automatically.

From source (optional Rust acceleration core, needs a Rust toolchain):

git clone https://github.com/fcmv/lucen
cd lucen
pip install -e ".[dev]"
pytest

On GIL builds 3.9 through 3.14, pip installs a native core (Rust, abi3, one binary per platform) that runs the orchestration hot loops (the write-set audit and the by-reference reduction folds). On free-threaded builds, where the abi3 core cannot load, pip installs a pure-Python wheel instead, so the install always succeeds; Lucen then runs its pure-Python fallback, which is fully supported and passes the identical test suite.

Tutorial

This walkthrough assumes nothing beyond basic Python. Every step shows real commands and real output shapes.

pip install lucen

Save this as work.py. It scores 20,000 records with a CPU-heavy function, plain Python, no Lucen anywhere yet:

import math

def score(x):
    acc = 0.0
    for k in range(400):
        acc += math.sin(x * 0.001 + k) * math.cos(k * 0.5)
    return acc

def main():
    records = list(range(20_000))
    scores = [0.0] * len(records)
    for i in range(len(records)):
        scores[i] = score(records[i])
    print(f"checksum: {sum(scores):.6f}")

if __name__ == "__main__":
    main()

python work.py    # takes roughly a second of pure compute

Add two comment lines around the loop. Nothing else changes:

# LUCEN START
for i in range(len(records)):
    scores[i] = score(records[i])
# LUCEN END

Run it again. Nothing happens. That is the point: pragmas are comments, and you have not activated anything. Your program is exactly as safe as before.

Run the file with lucen run. It rewrites the marked loops in the script you point at and then executes it, so the loop you just marked runs in parallel:

lucen run work.py

That is the whole story for a script you launch directly. When Lucen is instead embedded in a larger application you start yourself, activate the import hook once at startup. Activation installs the hook, so it must run before the module holding your marked loop is imported, which is why that loop lives in an imported module here:

# app.py
import lucen
lucen.activate()
import work
work.main()

python app.py

On a multi-core machine the loop now runs about 3 to 4 times faster, and the checksum is identical to the digit. Not approximately identical. Identical.

Two things to know, either way:

• The if __name__ == "__main__": guard in work.py matters on Windows and macOS. Lucen uses process workers there, and Python re-imports the entry module inside each worker. Lucen detects a missing guard and falls back to sequential with a message telling you to add it, so the failure mode is slowness, not breakage.
• activate() is idempotent and safe to call once at program start.

lucen explain work.py

work.py: 1 marked block(s) [gil interpreter assumed]
Block 1 (line 12)
  + Parallelized
    Backend: PROCESS (THREAD needs a free-threaded interpreter) (GIL interpreter assumed)
Runtime-dependent (never reported statically): argument picklability, custom-callable well-formedness, pool availability -- see `lucen profile`.

explain is static and honest: facts are reported as facts, and anything knowable only at call time is never reported as a yes or no.

Change the loop body to depend on the previous element:

# LUCEN START
for i in range(1, len(scores)):
    scores[i] = scores[i - 1] + score(records[i])
# LUCEN END

lucen explain work.py

Block 1 (line 12)
  - Sequential
    Reason: cross-iteration dependency 'scores[i - 1]' (monotonic chain); ...

Your program still runs and still produces the correct answer. Lucen just refuses to parallelize what it cannot prove, and says so. This is guarantee number two working as designed.

Mark a trivial loop:

# LUCEN START
for i in range(len(xs)):
    ys[i] = xs[i] * 2 + 1
# LUCEN END

At runtime Lucen probes the first chunk, measures roughly 50 nanoseconds per iteration, computes that process dispatch would cost more than it saves, and runs the whole thing sequentially at full speed. The fallback report says:

lucen fallback: PARALLEL_UNPROFITABLE (work.py:12): measured ~50 ns/iteration loses to dispatch overhead; ran SEQUENTIAL (calibrate=false overrides, spec 5.17)

If you believe the gate is wrong for your case, override it per block:

# LUCEN START calibrate=false

import lucen
lucen.activate()
import work
work.main()
for record in lucen.get_fallback_report():
    print(record.error, record.file, record.line, record.message)

Nothing Lucen decides is hidden. Every downgrade has a reason string and a location, and lucen profile script.py shows what actually ran, per block, with timings. That is the whole naive workflow: mark, activate, read what it tells you. You never need to know what a slab or a wavefront is to get correct parallelism.

Pragma Clauses

Tuning happens through pragma clauses. Every clause only ever trades a Lucen-held proof for a user-held assertion, or exactness for speed, never a different answer. A malformed clause is a loud import-time error with a did-you-mean suggestion, never a silent ignore.

# LUCEN START calibrate=false, timeout=5.0, on_error=collect

The full surface is fifteen clauses on # LUCEN START and two on # LUCEN TRUST. The complete reference, with every accepted form, is docs/pragmas.md; the summary:

On # LUCEN TRUST above a helper def:

• args= (how its arguments are trusted)
• qualname= (which callable the trust applies to)

Project-wide defaults and hard ceilings live in lucen.toml at your project root: pool sizes, timeout ceilings, an experimental-features veto, error verbosity. Degenerate values are rejected at load with the same strictness as pragma clauses.

CI integration: lucen explain --strict --baseline baseline.json fails the build if any block's classification regressed relative to a committed baseline, so a refactor that silently de-parallelizes a hot loop is caught in review, not in production.

Expert Surface

The expert surface is the trust system. Lucen proves what it can and trusts you for the rest, one explicit assertion at a time.

Assert a helper is parallel-safe. Lucen statically proves helper purity where it can read the source. A helper it proves stateful (mutates a module global, consumes random state, performs I/O) makes the block run sequentially, with a report naming the helper. Three overrides, narrowest first:

# LUCEN TRUST
def my_helper(x):
    ...  # you assert this def is safe under parallelism

# LUCEN START trust=callables (this block trusts all its helpers)

[trust]  # lucen.toml, project-wide
callables = ["mylib.fast_path"]

Assert independence the analyzer cannot see. depend=none asserts your writes are disjoint. The runtime write-set audit still runs and catches you if the assertion is false. Adding skip_runtime_check=true disables that audit too. It takes both assertions, two explicit lies, to make Lucen produce a wrong result; red-team testing confirmed one lie is always caught.

Assert pickle fidelity. The process backend verifies that the first chunk's argument bundle survives serialization at a byte-stable fixed point, which catches value-shifting __reduce__/__getstate__ implementations. trust=pickle waives the check.

Experimental Features

Experimental features, off by default, enabled per process:

lucen.activate(experimental=["early_exit", "typed_buffers"])

A [limits] allow_experimental = false in lucen.toml vetoes all of them, for fleet operators.

How It Works

One pipeline, five stages, each with a written decision record:

scanner -> rewriter -> selector -> codegen -> dispatch
(pragmas) (classify) (route)   (twins)  (execute + audit + commit)

The rewriter classifies every name in the block (loop-local, read-only, reduction accumulator, indexed write, cross-iteration read) and recognizes dependency shapes analytically, including results[i // 2]-style DAGs. The selector routes each block: parallel-eligible shapes to a backend, everything else to sequential, with reasons. Codegen emits two functions per block, a chunk function for workers and a sequential twin that is also the fallback path, so the sequential behavior is your original loop. Dispatch runs chunks over persistent pools, audits write disjointness at join, folds reductions in exact element order, and commits in chunk order.

The orchestration hot loops run in the Rust core where measurement says they should: the write-set audit is one native call over all chunks (4.5x the Python loop) and reduction folds run natively by reference through CPython's own number protocol, so bignums, float bits, and user-defined operators behave exactly as in the sequential loop (5x). The element-wise commit was ported, measured slower than CPython's specialized list stores, and deliberately kept in Python; the loop bodies themselves are always your Python.

Compiling eligible bodies to native kernels is the flagship roadmap item, not a current feature.

Backend selection is static and interpreter-independent: maps and reductions run on PROCESS on both GIL and free-threaded builds (measured: shared-object reference counting makes threads lose on shared-container workloads even without a GIL), THREAD serves by-reference blocks and free-threaded heavy compute, and everything lighter runs sequentially. Full details in the technical specification.

Performance

Summary across seven interpreters (CPython 3.9, 3.10, 3.11, 3.12, 3.13, 3.14, and 3.14 free-threaded), medians of [...]