31.8x Speedup by Changing One File: Async Embedding Calls on AWS Bedrock

Benchmark Setup

• Model: Amazon Titan Text Embeddings V2 (AWS Bedrock)
• Dataset: 33 text chunks from a single document
• Region: us-east-1
• Test: Sequential blocking requests vs. concurrent asynchronous requests

Results

Why the Gap Was So Massive

Sequential (Blocking I/O)

Each request must complete before the next one starts.

chunk 1 ──> request ──> ⏳ wait (~1.5s) ──> response
chunk 2 ──> request ──> ⏳ wait (~1.5s) ──> response
chunk 3 ──> request ──> ⏳ wait (~1.5s) ──> response
...

Total Time ≈ (33 chunks × ~1.5s) ≈ 49.6s

Since network latency dominates the runtime, the event loop sits completely unused.

Concurrent (Non-blocking Async)

Requests are dispatched concurrently; total time collapses down to the duration of the slowest single request.

chunk 1 ──> request ──────────────> response
chunk 2 ──> request ──────────────> response
chunk 3 ──> request ──────────────> response
... (in flight concurrently)

Total Time ≈ slowest single request ≈ 1.56s

The Code Change

Before: Sequential Execution

import requests

def generate_embeddings(texts: list[str]) -> list[list[float]]:
    all_embeddings = []
    for text in texts:
        response = requests.post(url, headers=headers, json={"inputText": text})
        vector = response.json().get("embedding", [])
        all_embeddings.append(vector)
    return all_embeddings

After: Concurrent Execution

import asyncio
import aiohttp

async def fetch_embedding(session: aiohttp.ClientSession, text: str) -> list[float]:
    async with session.post(url, headers=headers, json={"inputText": text}) as res:
        data = await res.json()
    return data.get("embedding", [])

async def generate_embeddings_async(texts: list[str]) -> list[list[float]]:
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_embedding(session, text) for text in texts]
        return await asyncio.gather(*tasks)

Note: If you're using boto3 instead of raw HTTP requests, look at aioboto3, or offload the blocking SDK calls with asyncio.to_thread() to get similar non-blocking behaviour.

How It Scales

The more chunks you ingest, the more the concurrent approach pays off - batch ingestion jobs that took half an hour now finish in seconds.

Key Takeaways

• Check for idle waiting first. Before scaling up infrastructure or adding complex worker queues, verify whether your pipeline is simply blocked on I/O.
• Mind the service quotas. Bedrock has per-region request-rate limits - as chunk counts grow into the thousands, cap concurrency (e.g. asyncio.Semaphore) before you hit them.
• Low effort, high impact. Changing just one file (embedder.py) removed the embedding stage as a bottleneck in the pipeline.