Simulating Traffic for an OCR Service: Part 1

TL;DR

• For OCR-like services, docs per second, rather than RPS is a key metrics to plan capacity for.
• Look at the hours when users are busiest - in our example morning p99 is twice the all-day p99.
• The simulation methods like Monte Carlo can give a workload we can employ to see how those peaks affect waiting time.
• Use the worker estimate to decide what to benchmark. Micro-batching could roughly halve the worker count (as in our case) in larger scenarios depending on a setup parameters.

When it comes to system design of your services, it's a good idea to start with understanding your traffic. For an invoice OCR service, one HTTP request might contain a single JPEG or a whole folder of invoices collected during a year.

The business wants users to get the value from the service promptly, even during the highload hours. Engineering has to figure out how much capacity that takes and how long a user can afford to wait while a batch fills.

Let's build a small capacity calculator, then simulate a day of incoming traffic. All numbers below are assumptions for this example. The Python code is in the research notebook.

Step 1. Estimate the incoming load

Suppose our service extracts text from PDF and JPEG invoices. A typical invoice has one page.

The first calculation fits on paper.

daily_uploads = dau * avg_requests_per_user
mean_rps = daily_uploads / 86400 # seconds in a day
morning_rps = mean_rps * peak_factor_morning
morning_docs_per_second = morning_rps * avg_docs_per_request

The baseline gives us 5,000 uploads and 35,000 documents per day. That's 0.05787 uploads/s on average and 0.17361 uploads/s during the morning peak. At seven documents per upload, the morning rate is 1.215 documents/s.

Step 2. Compare online and micro-batch processing

An online worker processes documents one at a time.

• Suppose one document takes 650 ms.
• A continuously busy worker can then process $1/0.65 = 1.54$ documents/s.
• Assumption: for micro-batching, suppose the same worker can process eight documents together in 2.6 seconds.
• That gives $8/2.6 = 3.08$ documents/s.

On a real deployment, we would benchmark both times on the same hardware. Here's the capacity calculation from compare_methods, reduced to the part that determines the worker count.

from math import ceil

def planned_workers(peak_docs_per_second, docs_per_batch, batch_seconds, utilization):
    docs_per_second_per_worker = docs_per_batch / batch_seconds
    return ceil(peak_docs_per_second / (docs_per_second_per_worker * utilization))

With a target worker utilization of 70%, use (docs_per_batch=1, batch_seconds=0.65) for online and (8, 2.6) for micro-batching.

Main takeaways here: micro-batching strategy makes more sense with a larger traffic - documents assemble the batch quicker, the potential worker savings come from processing several documents more efficiently together.

Step 3. Give the traffic a daily profile

Let's say the morning peak runs from 08:00 to 12:00. To keep the expected total at 5,000 uploads per day, we need to balance those four busy hours against the remaining 20.

If k is the off-peak multiplier, the average multiplier over 24 hours must equal one.

$$\frac{4 \cdot 3 + 20 \cdot k}{24} = 1 \Rightarrow k = 0.6$$

What would the upload counter actually show at 08:00:00? Let $N_t$ be the upload count in second $t$. We draw it from a Poisson distribution with mean $\mu_t$, the expected count for that second. We assume independent counts across seconds given this daily profile.

$$P(N_t = k) = \frac{e^{-\mu_t} \mu_t^k}{k!}$$

For a morning second, $\mu_t=0.17361$. The probability of an empty second is $e^{-0.17361} \approx 84.06%$.

With a baseline parameters, even during the busiest part of the day most seconds still empty (84%). For this first simulation, we fix the upload size = 7, so a second with two uploads brings 14 documents.

Step 4. Simulate 100 possible days

We can now give each second its expected count and use SciPy to draw the arrivals. Each row is one simulated day and each column - the same second of the day across all runs.

import numpy as np
from scipy import stats

def simulate_uploads(dau, uploads_per_user, days=100, seed=42):
    mean_rps = dau * uploads_per_user / 86400
    hours = np.arange(86400) / 3600
    morning = (hours >= 8) & (hours < 12)
    expected_counts = mean_rps * np.where(morning, 3.0, 0.6)
    requests = stats.poisson.rvs(
        mu=expected_counts,
        size=(days, 86400),
        random_state=np.random.default_rng(seed),
    )
    return expected_counts, requests, morning

expected_counts, requests, morning = simulate_uploads(1000, 5)
t = 8 * 3600 # 08:00:00
print(expected_counts[t]) # 0.17361... expected uploads
print(requests[:10, t]) # [0 0 0 0 0 0 0 1 0 0]
print(requests[:, t].mean()) # 0.22 uploads, averaged over 100 days

The model gives 08:00:00 the same expectation every day, and the counter shows a different possible outcome each day. If we increase the number of users to 10,000 users and 10 uploads each, the expected count at 08:00:00 rises to 3.472. The first ten draws are now: [5 2 3 3 4 1 0 4 6 2].

To see the wider pattern, pool the one-second counts across all 100 days and calculate the quantiles. Then repeat for the morning alone.

def quantiles(counts):
    return np.quantile(counts, [0.50, 0.95, 0.99], method="inverted_cdf")

for users, uploads in [(1_000, 5), (10_000, 10)]:
    _, counts, morning = simulate_uploads(users, uploads)
    print(users, quantiles(counts), quantiles(counts[:, morning]))

In Part 2, I'll send the same arrivals through online and micro-batch workers and measure how long each user waits for the last document in their upload.