The Test That Passes in Staging But Fails When a Customer Runs It

You have been here. The test suite is green. The deployment pipeline reports all checks passed. Then a customer opens a ticket with a screenshot that shows something your test never caught. The test passed in staging. It fails in production. And you cannot reproduce it locally. This is not a flaky test problem. It is a fidelity problem. Your test environment and your production environment are not the same thing. The gap between them is where real bugs live. Let me walk through one concrete example, the fix, and what it teaches about writing tests that survive the handoff to a real user.

The Problem: Environment Drift

A fintech team I worked with had a checkout flow. The test clicked "Pay Now", waited for a success message, and asserted the text "Payment successful" appeared on screen. It passed every time in staging. Customers reported that after paying, they saw a blank white page for several seconds before the success message appeared. Some of them closed the tab during that blank period, thinking the payment failed. The transaction went through. The customer never saw the confirmation. Support tickets piled up.

The test never caught this because the staging environment served the success page in under 200 milliseconds. The blank period did not exist there. Production had a slower downstream service that introduced a three-second delay between the payment confirmation and the page render. The test was correct in what it checked. It was wrong in what it assumed about timing and state.

The Fix: Test the Experience, Not Just the Outcome

The fix was not to add a longer wait. The fix was to test what the user actually experiences during that gap. Here is a minimal Playwright test in TypeScript that catches this class of problem:

import { test, expect } from '@playwright/test';

test('checkout shows loading state before success', async ({ page }) => {
  await page.goto('/checkout');
  await page.fill('#card-number', '4111111111111111');
  await page.fill('#expiry', '12/28');
  await page.fill('#cvc', '123');
  await page.click('button[type="submit"]');

  // The user sees a loading indicator before the success message
  // This assertion catches the blank-page gap
  await expect(page.locator('[data-testid="loading-spinner"]')).toBeVisible({ timeout: 5000 });

  // Then the success message appears
  await expect(page.locator('text=Payment successful')).toBeVisible({ timeout: 10000 });
});

The key line is the loading spinner assertion. It forces the test to observe the intermediate state. If the spinner never appears because the page goes blank, the test fails. That failure tells you something about the real user experience that a simple text assertion never would.

In Python with Playwright, the same logic looks like this:

from playwright.sync_api import sync_playwright

def test_checkout_loading_state():
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page()
        page.goto("/checkout")
        page.fill("#card-number", "4111111111111111")
        page.fill("#expiry", "12/28")
        page.fill("#cvc", "123")
        page.click('button[type="submit"]')

        # Assert the loading state appears
        spinner = page.locator('[data-testid="loading-spinner"]')
        assert spinner.is_visible(timeout=5000)

        # Assert the final success state
        success = page.locator("text=Payment successful")
        assert success.is_visible(timeout=10000)

        browser.close()

This is not about adding more assertions. It is about asserting the right things at the right moments.

Technical Detail: Why This Works

The blank page problem happens because the browser receives the HTTP response for the success page before the JavaScript that renders the loading state has finished executing. In staging, the response arrives so fast that the JavaScript finishes before the browser paints. In production, the response arrives, the browser paints a blank page, and then the JavaScript catches up.

A standard waitForSelector or toBeVisible on the success text will pass in staging because the text appears within the default timeout. In production, the same assertion might pass after a longer wait, but it never observes the blank period. The test reports green. The user reports a broken experience.

By asserting the loading state, you force the test to observe the transition. If the loading state never appears, you know the page went through an unexpected state. That is the signal you need.

What This Teaches

Three things.

• First, your test environment is a lie. It is a controlled simulation that hides timing differences, network latency, and service degradation. The only way to catch environment-specific bugs is to write tests that assume the environment is hostile. Assert intermediate states. Assert loading indicators. Assert error boundaries. Do not just assert the happy path endpoint.
• Second, a passing test is not proof of quality. It is proof that your test and your environment agree on what should happen. That agreement can be wrong. The customer is the only judge that matters.
• Third, the most valuable tests are the ones that fail in staging. A test that fails in staging tells you something about your system before it reaches a customer. A test that passes in staging and fails in production tells you that your test was not testing the right thing.

The Call to Action

Look at your test suite right now. Find one test that only asserts a final state. A success message. A redirect. A database row. Add one assertion for an intermediate state. A loading spinner. A disabled button. A progress bar. Run it in staging. Then ask yourself: if this test ran in production, would it still tell the truth?

The gap between staging and production is not going away. But your tests can learn to see it. What is one test in your suite that you suspect is lying to you?