DEV Community 12h ago

System Design - Building an Experiment Service

Designing an experiment service: API design, data modeling, and how to scale it.

Before any architecture, a shared vocabulary. An experiment is the thing under test - "does the new signup page convert better than the old one." Each version shown to visitors is a variant; the version you'd show if there were no experiment running at all is usually the control. Traffic doesn't have to split evenly across variants - each variant carries a weight, its configured share of traffic. When a visitor sees a variant, that's an exposure. When they complete the goal action - signing up, buying something - that's a conversion. Conversion rate - conversions divided by exposures, per variant - is the number that eventually tells you which variant won.

That vocabulary is simple. Building the service that makes it happen correctly, at real traffic volume, without slowing down the page it's embedded in, is where the actual design work lives.

Requirements

Good system design starts with requirements, not architecture. Here's what this service has to do - and, just as importantly, what it has to guarantee.

Functional requirements

A client (a business - this is a multi-tenant service) can sign up and get credentials.
A client can define an experiment: its variants and their weights.
Given a visitor and an experiment, the system returns which variant they should see.
Exposures and conversions are recorded as they happen.
A client can view exposures, conversions, and conversion rate per variant.

Non-functional requirements

Low latency on the read path. The variant-assignment call sits directly between a visitor and their page rendering. It has to respond in single-digit-to-low-double-digit milliseconds - a real browser is blocked on it.
Consistency. The same visitor, on the same experiment, must always get the same variant, for as long as the experiment runs.
Data integrity. Every exposure and conversion must be counted exactly once, even under retries and duplicate requests.
Multi-tenancy. One client's experiments must never be visible to, or affected by, another client's.
Scalability. Traffic should be able to grow by orders of magnitude without a redesign - by adding capacity, not by rewriting.
Graceful degradation. If any dependency misbehaves, the system degrades to a safe default rather than failing visibly for a real visitor.

Every design decision below traces back to one of these - mostly the first three.

API design

`POST /clients` - sign up

No auth here - you can't require a key to get your first key.

curl -X POST https://api.yourexperimentservice.com/clients \
  -H "Content-Type: application/json" \
  -d '{"name": "Acme Labs"}'

Response:

{
  "id": "cl_9f2a",
  "name": "Acme Labs",
  "api_key": "sk_live_..."
}

`POST /experiments` - configure an experiment

Requires the API key from signup.

curl -X POST https://api.yourexperimentservice.com/experiments \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk_live_..." \
  -d '{
    "key": "signup_test",
    "status": "live",
    "variants": [
      { "key": "control", "weight": 70, "static_copy": "Welcome!" },
      { "key": "variant_b", "weight": 30, "ai_prompt": "Write a short signup headline" }
    ]
  }'

Response: the persisted experiment, with every variant's static_copy filled in - generated ones included.

`GET /assignment` - the hot path

curl "https://api.yourexperimentservice.com/assignment?user_id=visitor_123&experiment_key=signup_test" \
  -H "X-API-Key: sk_live_..."

Response:

{
  "experiment_key": "signup_test",
  "assigned_variant": "control",
  "static_copy": "Welcome!"
}

Same user_id + experiment_key always returns the identical result, indefinitely.

`POST /events` - record an exposure or conversion

curl -X POST https://api.yourexperimentservice.com/events \
  -H "Content-Type: application/json" \
  -H "X-API-Key: sk_live_..." \
  -d '{
    "visitor_id": "visitor_123",
    "experiment_key": "signup_test",
    "variant_key": "control",
    "event_type": "exposure"
  }'

Response:

{
  "status": "success",
  "event_id": "ev_88b1"
}

A repeated identical exposure returns this same shape with the same event_id - not an error.

`GET /results` - reporting

curl "https://api.yourexperimentservice.com/results?experiment_key=signup_test" \
  -H "X-API-Key: sk_live_..."

Response:

{
  "experiment_key": "signup_test",
  "results": [
    {
      "variant_key": "control",
      "exposures": 812,
      "conversions": 94,
      "conversion_rate": 0.116
    }
  ]
}

The same data also powers a tenant-scoped dashboard view, so a client can check results without writing a script.

High-level design

flowchart TD
  A[Signup API] --> E[(Database)]
  B[Config API] --> E
  C[Assignment API]:::hot --> E
  D[Event API] --> E
  E --> F[Dashboard]
  classDef hot stroke-width:3px

Four independent write paths - signup, configuration, assignment, and events - all persist to one database. Reads for reporting go through the same store, for now.

Of these four, only the assignment path carries a strict latency budget: it's invoked once per page view, synchronously, with a real visitor's browser waiting on the response. That single asymmetry - one latency-critical path, three that can afford to be slower - drives most of the design decisions below.

Design deep dive: keep expensive work off the hot path

The experiment API has one interesting requirement: a variant's copy can be written by hand, or generated by an AI model. The naive design calls the model at assignment time - whenever a visitor is due to see an AI-authored variant, generate fresh copy right then. That breaks the latency requirement immediately. A third-party model call can take seconds, time out, or simply be unavailable - none of which the assignment path can tolerate.

The general principle here shows up across most systems with a strict-latency read path: push expensive computation to write time, and serve reads from something already computed. If a value doesn't need to be fresh at read time - and nobody actually needs bespoke copy generated in the few milliseconds between a page loading and a response - compute it once when it's cheap to be slow (at configuration time), persist the result, and let every subsequent read be a plain lookup. The assignment path never talks to the model at all. If generation fails outright, the system falls back to plain, default copy rather than blocking the one call that isn't allowed to fail.

This is the same idea behind materialized views, precomputed caches, and read-model patterns generally: decouple when a value is computed from when it's read.

Design deep dive: the assignment API - sticky and deterministic

This is the most interesting endpoint in the system, and it's worth separating the requirement from the mechanism, because the two get conflated easily.

The requirement is stickiness. A visitor must see the same variant on every visit, for the life of the experiment. If you've worked with load balancers, this should sound familiar - it's the same property that session affinity ("sticky sessions") provides: routing a client's subsequent requests to the same backend server it hit the first time.

The traditional way to get stickiness is stateful: the first time you see a visitor, make a decision and remember it - in a session store, a cookie plus a server-side lookup, or a load balancer's affinity table. Every later request has to consult that stored state.

The mechanism is determinism instead of memory. There's a cheaper way to get the same property, if one condition holds: if the decision is a pure function of information already available on every request, nothing needs to be stored - the identical answer can be recomputed every time. That's the route taken here.

Concretely: hash the visitor ID and the experiment key together, producing a number. Reduce it to a fixed range - say 0 to 99 - and partition that range according to the configured weights: the first 70 numbers to variant A on a 70/30 split, the rest to variant B. The same two inputs always hash to the same number, so the same visitor lands in the same partition, for the same experiment, forever - nothing written to disk, no state anywhere.

Two properties of the hash function are doing the actual work:

Determinism - the same input always produces the same output. This is what gives stickiness without storage. Most hash functions are deterministic by construction; this isn't something you engineer, but it is the property being relied on here.
Uniform distribution - across many different inputs, outputs spread evenly across the output range, without clustering. This is what makes a configured 70/30 split actually land close to 70/30 in practice, instead of skewing toward one variant because of some quirk in how visitor IDs happen to be generated.

Why hash the visitor ID and the experiment key together, rather than the visitor ID alone. This detail matters more than it looks. Hash the visitor ID alone, and a given visitor lands in the same percentile bucket for every experiment ever run - someone in the "top 30%" bucket for one experiment is in the top 30% for all of them. That's an unwanted correlation: it means experiments aren't actually independent of each other, and a visitor who happens to convert unusually well (or badly) skews every test in the same direction. Hashing the experiment key in as well gives each experiment its own, effectively independent bucketing for the same visitor - which experiment and which visitor stop leaking into each other.

A clarifying note, since the terminology overlaps. This is not the same thing as consistent hashing, a related but distinct technique for distributing data across a changing set of nodes - think shards in a distributed cache or database - while minimizing how much gets remapped when a node is added or removed. What's happening here is simpler: one hash function mapping (visitor, experiment) to a fixed bucket range that never changes shape. Worth knowing the distinction, since "hashing for consistent assignment" and "consistent hashing" sound alike but solve different problems.

One more subtlety worth internalizing. This design produces two different guarantees, and verifying one doesn't verify the other. For any single visitor, the assignment is completely deterministic - no randomness in an individual decision at all. Across a population of visitors, the outputs merely look randomly distributed, and it's that population-level randomness that makes a configured split hold up in aggregate. That second property is statistical, not deterministic, and it has to be checked empirically, with a genuinely large sample - a handful of test visitors can look meaningfully skewed purely by chance in a perfectly correct system, and can just as easily look fine when something's actually broken.

Design deep dive: idempotency and duplicate prevention

Recording an exposure or conversion looks trivial: a visitor saw a variant, write a row. It stops being trivial the moment real traffic replaces clean test data - double-clicks, retried page loads, back-and-forward navigation all produce the same logical event more than once. If every one of those writes a row, every conversion-rate number downstream is silently inflated.

The naive fix, and why it isn't enough. Check whether this exact event has already been recorded before writing a new one. The problem: "check" and "then write" are two separate operations with a gap between them. Under concurrent requests, two duplicate calls can both pass the check before either has written anything. This is a classic check-then-act race condition - sometimes called TOCTOU, time-of-check to time-of-use - and it shows up anywhere a read-then-write isn't atomic.

The actual fix: idempotency, enforced structurally. An operation is idempotent if performing it once and performing it five times leave the system in the same state. The reliable way to get there isn't detecting duplicates in application code - it's making a duplicate physically impossible to write, by pushing a uniqueness constraint down to the storage layer itself: here, a unique constraint on (visitor, experiment, variant, event type). A duplicate write is no longer something application code has to notice - the write itself is rejected by the database, no matter how many concurrent requests arrive. The calling code treats "rejected as duplicate" as a success, not an error, since the thing it wanted recorded already exists.

This generalizes well beyond one system - it's the same principle behind idempotency keys in payment APIs, exactly-once processing in message queues, and safe retries in any at-least-once delivery system. Wherever possible, move "detect and reject a duplicate" down into "make the duplicate structurally impossible," as close to the data as the storage layer allows.

Data model (low-level design)

erDiagram
  CLIENT ||--o{ EXPERIMENT : owns
  EXPERIMENT ||--o{ VARIANT : has
  EXPERIMENT ||--o{ TRACKING_EVENT : logs
  VARIANT ||--o{ TRACKING_EVENT : recorded_under

  CLIENT {
    uuid id PK
    string name
    string api_key
  }

  EXPERIMENT {
    uuid id PK
    uuid client_id FK
    string key
    string status
  }

  VARIANT {
    uuid id PK
    uuid experiment_id FK
    string key
    int weight
    string static_copy
  }

  TRACKING_EVENT {
    uuid id PK
    uuid experiment_id FK
    uuid variant_id FK
    string visitor_id
    string event_type
    timestamp created_at
  }

Four entities, and the relationships between them are doing real work, not just organizing storage:

Client: the tenant. Everything else exists only in the context of one of these.
Experiment: belongs to exactly one client. Its name only has to be unique within that client, not globally - which is exactly what makes "scope every lookup to the tenant" a hard requirement rather than a nice-to-have. Two clients can legitimately name an experiment the same thing; any lookup that isn't scoped by client risks serving one tenant's config to another tenant's request.
Variant: belongs to an experiment, carries a weight and either literal or AI-generated content.
TrackingEvent: one row per exposure or conversion, referencing both the experiment and the specific variant.

Read on DEV Community ↗ ← Back to News

System Design - Building an Experiment Service