Store Load Reordering: x86 vs ARM64, and the Bug Intel Was Hiding

Store Load Reordering: x86 vs ARM64, and the Bug Intel Was Hiding

Here is a bug report that keeps recurring, in different words, every time a team moves to ARM: "the same code works on our Intel CI and on the developers' older MacBooks, but it corrupts data / deadlocks / returns impossible values on Graviton." No source change, no compiler change, no new code. The bug was always there. x86 was hiding it. The mechanism underneath is store→load reordering, and it's one of the few places where two mainstream CPU architectures genuinely disagree about what your program means.

This post takes it apart with a small litmus test you can run yourself, real numbers from running it on ARM64, and the reason a memory fence turns an "impossible" outcome back into an impossible one.

The Test That Shouldn't Be Able to Fail

Two threads, two shared variables, both starting at zero:

// Thread 1
// Thread 2
X = 1; Y = 1; r1 = Y; r2 = X;

Reason about it in program order and r1 == 0 && r2 == 0 looks impossible. Walk the interleavings:

• Thread 1 finishes first → X=1, Y=0 →r1=0, r2=1
• Thread 2 finishes first → Y=1, X=0 →r1=1, r2=0
• They interleave after at least one store lands → r1=1, r2=1

There is no ordering of these four operations in which both loads read zero. For both to read zero, each thread's load would have to run before the other thread's store - but also before its own store, which comes first in the source. Under a sequentially consistent machine, that outcome cannot happen. Real hardware produces it anyway.

Why the CPU Makes the Impossible Happen

The CPU is allowed to reorder each thread into this:

// Thread 1 (as executed)
// Thread 2 (as executed)
r1 = Y; // load first
r2 = X; // load first
X = 1; Y = 1;

Now the timeline that was forbidden is trivial: Thread 1 loads Y (still 0), Thread 2 loads X (still 0), then both stores land. r1 == 0 && r2 == 0.

Why would a CPU do this? A store isn't finished when the instruction retires - it goes into the store buffer and drains to cache later. A load to a different address has no visible dependency on that pending store, so the core is free to let the load pass the buffered store and keep the pipeline busy. From a single thread's point of view nothing changed; X and Y are independent, so reordering X=1 and r1=Y is invisible to that thread. The other core is where the reordering becomes observable. That's the key idea: the reordering is legal precisely because it's invisible to the thread doing it. It only leaks through a second observer.

The Litmus Test, in Real Code

This is the harness from CoreTracer, trimmed to the essentials:

volatile int X = 0, Y = 0, r1 = -1, r2 = -1;

// Thread 1: X = 1; r1 = Y;
void *thread1_func(void *p) {
    X = 1;
    if (use_fence) memory_barrier(); // real CPU fence
    else compiler_barrier(); // stops the compiler only
    r1 = Y;
    ...
}

// Thread 2 is the mirror image: Y = 1; ...; r2 = X;
// After both threads finish an iteration:
if (r1 == 0 && r2 == 0) reorder_detected++;

Two details matter. First, the barrier:

void memory_barrier() {
    #if defined(__x86_64__)
        __asm__ volatile("mfence" ::: "memory"); // x86
    #elif defined(__aarch64__)
        __asm__ volatile("dsb sy" ::: "memory"); // ARM64
    #endif
}

void compiler_barrier() {
    __asm__ volatile("" ::: "memory"); // compiler fence, no CPU effect
}

Second, the volatile and the compiler barrier are deliberate. This is a litmus test, not production code - volatile isn't the right tool for real concurrency (that's what C11 atomics are for). Here it exists to stop the compiler from optimizing the accesses away or reordering them itself, so that the only reordering left to observe is the hardware's. That separation is the whole point: it lets you tell a compiler reorder apart from a CPU reorder.

What the Numbers Actually Say

Run it on ARM64 across a million iterations and the three cases separate cleanly:

Read those three rows carefully, because they tell the whole story:

• 2.3% with nothing - the reordering is not exotic. It's happening on roughly one in forty iterations.
• 1.8% with a compiler barrier - stopping the compiler from reordering barely moves the number. That's the proof that the CPU, not the compiler, is doing this. A compiler_barrier() (or Go's //go:nosplit -style tricks, or marking things volatile) does nothing about it.
• 0.0% with a real fence - one dsb sy and the outcome is gone entirely.

On x86 the same test behaves very differently. Store→load is the only reordering TSO permits, and the store buffer drains aggressively, so on a short run you often see zero hits and have to push iterations way up to catch any at all. The test harness even says as much when it comes back empty: "try more iterations or a different CPU." The video runs both sides live and prints the counts if you want to watch the gap open up rather than take the table's word for it.

x86 TSO vs ARM64: What Each One Lets Slide

The reason the same binary behaves differently is that the two architectures publish different rules for which reorderings are allowed: