Bit-Reproducible AI Inference: Why “Close Enough” Isn’t Always Enough

Most AI inference systems are built around a reasonable assumption: If two outputs are numerically close enough, they are effectively the same. For many workloads, that is absolutely fine.

But there are cases where “close enough” creates a different kind of engineering problem. What if you need to know whether two inference runs were exactly the same computation?

That question matters for regression testing, debugging, distributed execution, auditing, and verification. And it gets especially interesting when one model runs across multiple machines.

Floating-point inference is usually approximate by design

Modern AI inference relies heavily on floating-point arithmetic. That gives us excellent performance on CPUs, GPUs, and accelerators. But floating-point arithmetic has an important property: the order of operations can affect the final bits.

For example, mathematically:

(a + b) + c and:

a + (b + c) represent the same addition.

In floating-point arithmetic, however, they are not guaranteed to produce exactly the same binary result.

That becomes relevant in highly parallel systems. Different hardware, kernels, thread schedules, reduction orders, and execution paths can produce tiny numerical differences. Usually those differences do not matter. But sometimes they do.

The problem becomes harder in distributed inference

Now imagine a model that does not fit on one machine. Instead of loading the entire model onto a single server, you split it across several machines. Machine A holds one part. Machine B holds another. Machine C holds the next. Together, they execute one model and one inference process.

This solves a capacity problem. But now another question appears: Can the distributed run be reproduced exactly?

If intermediate numerical values drift across devices or execution paths, comparing two distributed runs becomes harder. Was the difference caused by:

• an actual code regression?
• a model update?
• a different kernel?
• a changed execution path?
• or ordinary floating-point variation?

For debugging, that ambiguity is expensive.

Exact reproducibility changes regression testing

Suppose you change one component of your inference system. You run the same test again. The outputs differ slightly. Now you need to determine whether the change is meaningful.

With bit-reproducible inference, the test becomes much cleaner. Either:

hash(run_A) == hash(run_B)

or it does not.

That does not eliminate the need for deeper testing. But it removes an entire class of numerical ambiguity.

Determinism is a different optimization target

AI infrastructure is usually optimized around:

• tokens per second
• latency
• throughput
• accelerator utilization
• cost

Those are important. But exact reproducibility is another possible optimization target.

At Pelloran, we have been exploring that trade-off with an open-source project called tritify. tritify uses deterministic integer execution for its reproducible inference path.

The goal is: Same model. Same input. Same execution schedule. Same bits. Not approximately the same output. The same bits.

One model across multiple machines

This also applies to distributed execution. A model can be partitioned across several machines. Each machine stores and computes only its part. They are not running several independent AIs. They are cooperating as one system to execute one inference process.

The distributed output can then be compared against a reference run exactly. That is useful for systems where reproducibility matters as much as capacity.

A large-model experiment

We tested this approach with GLM-4.5-Air. The model has roughly 106.9 billion parameters. The tested integer representation involved approximately 427 GB of integer codes. The model was distributed across three machines. No individual machine needed to hold the complete integer representation in memory.

For the tested run, the distributed execution matched the streamed reference run bit for bit, including matching generated/logit hashes and per-layer hidden-state hashes.

The important part is not that distributed inference exists. It already does. The interesting part is combining:

distributed execution + deterministic computation + exact verification

This is not a speed claim

There is an obvious trade-off. Highly optimized floating-point inference is built for performance. Exact deterministic execution has a different goal.

tritify is not claiming that deterministic integer inference automatically beats optimized floating-point inference in raw throughput. It is optimizing for something else:

• reproducibility
• auditability
• debugging
• regression testing
• verification

For some workloads, speed wins. For others, exact reproducibility may be worth the trade-off.

Where could this matter?

A few obvious cases:

Regression testing
A numerical difference is more likely to represent a real change instead of floating-point noise.

Distributed inference
Machines can cooperate while still producing a result that can be checked against a reference run.

Reproducible research
A result can be rerun and compared exactly.

Audit-heavy systems
The execution can leave behind hashes and deterministic outputs that are easier to inspect.

Verification
Deterministic execution creates a stable computation that other verification mechanisms can reason about.

Determinism does not solve every trust problem

It is also important not to overstate what reproducibility provides. Deterministic execution does not automatically provide:

• privacy
• confidentiality
• remote-machine honesty
• cryptographic attestation

Those are separate problems. But deterministic inference can provide a useful foundation underneath them. Before proving that a computation was executed correctly, it helps if there is an exact reference computation to compare against.

The bigger question

As AI infrastructure matures, we may need more than one optimization mode. For some systems: maximum throughput wins. For others: exact reproducibility wins. And some systems may use both - fast inference for normal operation and deterministic execution for testing, auditing, or critical checkpoints.

That is the trade-off we are exploring with tritify. The project is open source under Apache 2.0.

GitHub: https://github.com/pelloran-inc/tritify
Paper: https://doi.org/10.5281/zenodo.23986737

I’m especially interested in how other engineers think about this: Where would you personally trade some inference throughput for exact reproducibility?