Beyond the LLM Call: Anatomy of a Production AI Application

Most AI demos have the same shape: User input -> LLM API -> response And for a demo, that is enough. But the moment an AI system handles real documents, multiple tenants, uneven traffic, expensive model calls, retries, and uptime expectations, the LLM becomes only one part of the problem. A production AI application is a distributed system with an LLM inside it. The engineering work is not just asking a model a question. It is designing a system that can: - ingest large and unpredictable workloads, - retrieve the right context safely, - control latency and token cost, - survive duplicate events and partial failures, - isolate one tenant from another, - observe what happened after an answer is returned, - and degrade predictably when a dependency is unavailable. This article walks through a reusable architecture for a production AI API on AWS. The example is a multi-tenant retrieval-augmented generation system, but the underlying lessons apply to document intelligence, AI agents, support copilots, internal search systems, and many other AI workloads. The core principle is simple: Keep user-facing AI requests bounded and synchronous. Move expensive, variable, failure-prone preparation work into durable asynchronous pipelines. The Engineering Problem A knowledge-grounded AI API usually needs to do two things: Ingest information Accept files, extract content, split it into chunks, create embeddings, and index those chunks for retrieval.Answer questions Retrieve relevant chunks, assemble a prompt, call a model, validate the result, and return a response. These two workloads look related, but they behave very differently. | Workload | Typical behavior | Main concern | |---|---|---| | Query request | Small, interactive, latency-sensitive | Fast and predictable response | | Document ingestion | Large, bursty, long-running, failure-prone | Durable processing and recovery | | Embedding | Batch-friendly, provider-limited | Throughput and cost | | Vector retrieval | Low latency, filter-sensitive | Relevance and tenant isolation | | LLM generation | Variable latency and cost | Timeout, quality, and token control | A common mistake is trying to process everything inside one web request. POST /documents -> upload file -> extract text -> chunk text -> generate embeddings -> index vectors -> return success This feels simple until the first real workload appears: - a tenant uploads 5,000 files, - an OCR step takes 30 seconds, - the embedding provider throttles, - the process crashes after indexing half the chunks, - interactive query traffic competes with ingestion workers, - the API starts timing out. The architecture fails because it treats fundamentally different workloads as if they have the same runtime requirements. They do not. Why Naive AI Architectures Fail Let us look at the synchronous-everything design more closely. flowchart LR C[Client] --> API[API Service] API --> P[Parse Document] P --> CH[Chunk Content] CH --> E[Generate Embeddings] E --> V[Write Vector Index] V --> R[Return HTTP Response] At low traffic, it works. At production traffic, it produces several failure modes. 1. Large files block small requests A 5 KB text file and a 200-page scanned PDF pass through the same service and consume the same worker pool. That means a slow document-processing request can occupy capacity needed for a fast query request. This is called head-of-line blocking. Fast query arrives -> waits behind slow OCR job -> latency rises -> client retries -> load increases further The problem is not that OCR is slow. The problem is that slow work shares the same execution path as latency-sensitive work. 2. A traffic spike becomes an outage Imagine the system can process 100 documents per minute. Then one tenant uploads 10,000 documents. Without a durable buffer, the API must immediately absorb work it cannot complete. xychart-beta title "No Queue: Burst Traffic Overwhelms Workers" x-axis [0, 1, 2, 3, 4, 5] y-axis "Documents per minute" 0 --> 1200 line [100, 100, 1000, 900, 500, 150] line [100, 100, 100, 100, 100, 100] The first line represents incoming documents. The second line represents processing capacity. The difference becomes timeouts, failed requests, memory pressure, connection exhaustion, and eventually cascading failure. 3. Retries create duplicate work Distributed systems rarely guarantee exactly-once execution. A worker may successfully write vector records and then crash before it acknowledges the message that triggered the work. The queue sends the message again. If the system assumes the message is unique, the retry creates: - duplicate vectors, - duplicate model calls, - duplicate cost, - inconsistent metadata, - confusing retrieval results. The fix is not “make retries impossible.” The fix is designing side effects to be idempotent. If the same work runs twice, the final system state should be equivalent to running it once. 4. Unbounded context creates unbounded cost RAG systems often fail in a quieter way. The system retrieves more chunks as the corpus grows. More chunks become more input tokens. More input tokens become higher latency and higher cost. more documents -> more retrieved chunks -> larger prompt -> more tokens -> slower model response -> higher cost per request A production system needs hard boundaries: - maximum number of retrieved chunks, - relevance thresholds, - maximum tokens per chunk, - maximum prompt budget, - maximum output tokens, - per-tenant rate limits. Without those controls, “better retrieval” can quietly become “unpredictable spending.” The Core Theory: Bounded Work and Unbounded Work The architecture starts with one question: Is this work bounded enough to run inside a user-facing request? A query request should be bounded. For example: Maximum retrieval results: 8 Maximum context tokens: 8,000 Maximum model output tokens: 1,000 Maximum Bedrock timeout: 8 seconds Maximum retry attempts: 1 These boundaries give the request a predictable latency and cost envelope. Document ingestion is different. A document might be: - a short Markdown file, - a large PDF, - a scanned file requiring OCR, - a spreadsheet with multiple sheets, - a ZIP archive containing nested files, - malformed or malicious content. You cannot reliably promise that this work will finish inside a short HTTP request. That makes document ingestion unbounded work. The correct architecture is to accept the work durably, place it behind a queue, and process it asynchronously. flowchart TB subgraph Synchronous["Synchronous query path: bounded work"] Q[Question] --> Auth[Auth and tenant policy] Auth --> Retrieve[Retrieve bounded context] Retrieve --> LLM[Invoke model with deadline] LLM --> Response[Return response] end flowchart TB subgraph Async["Asynchronous ingestion path: unbounded work"] Upload[Document upload] --> Queue[Durable queue] Queue --> Extract[Extract] Extract --> Chunk[Chunk] Chunk --> Embed[Embed] Embed --> Index[Index] Index --> Ready[Mark document READY] end This split does not eliminate complexity. It puts complexity where it belongs. The Architecture Pattern A production AI API benefits from two independently scalable planes. Query plane The query plane serves interactive requests. Its job is to: - authenticate the caller, - resolve tenant policy, - enforce rate limits, - retrieve safe and relevant context, - invoke a model within a deadline, - validate the output, - return traceable metadata. Client -> API Gateway -> Query service -> Cache -> Vector retrieval -> LLM invocation -> Response The query path should optimize for: - low p95 latency, - predictable model usage, - tenant isolation, - graceful failure, - controlled cost. Ingestion plane The ingestion plane prepares knowledge for retrieval. Its job is to: - accept uploaded documents, - extract and normalize content, - create stable chunks, - create embeddings, - index vectors, - track document state, - retry recoverable failures, - route terminal failures for investigation. S3 upload -> event -> queue -> extraction worker -> chunking worker -> embedding worker -> vector index -> metadata state update The ingestion path should optimize for: - throughput, - durable acceptance, - retryability, - idempotency, - cost-efficient batching, - visibility into backlog and failures. Production AI API on AWS The following architecture uses AWS services deliberately. Each service exists to support a system property, not because it is a familiar logo on an architecture diagram. flowchart TB Client[Client Application] subgraph Edge["Edge and security boundary"] WAF[AWS WAF] APIGW[Amazon API Gateway] Auth[JWT/OIDC Authentication] end subgraph QueryPlane["Query Plane"] Query[ECS Fargate Query Service] Redis[ElastiCache Redis] DDB[(DynamoDB Metadata)] OS[(OpenSearch Serverless)] Bedrock[Amazon Bedrock] end subgraph IngestionPlane["Ingestion Plane"] S3[(Amazon S3)] EB[Amazon EventBridge] SQS[SQS Ingestion Queue] DLQ[SQS Dead-Letter Queue] SFN[Step Functions] Worker[ECS Fargate Workers] end subgraph Operations["Operations plane"] CW[CloudWatch and OpenTelemetry] KMS[AWS KMS] IAM[IAM] SM[Secrets Manager] end Client --> WAF --> APIGW --> Auth --> Query Query --> Redis Query --> DDB Query --> OS Query --> Bedrock Client --> S3 S3 --> EB --> SQS --> SFN --> Worker SQS -. terminal failure .-> DLQ Worker --> S3 Worker --> DDB Worker --> OS Worker --> Bedrock Query --> CW Worker --> CW API Gateway: the controlled public boundary Amazon API Gateway is the public entry point for HTTP requests. Its responsibilities include: - request routing, - throttling, - request-size limits, - authentication integration, - API versioning, - WAF integration, - request identifiers. The main architectural benefit is that the application service does not become the first line of defense against abusive or malformed traffic. Alternative An Application Load Balancer can be a good option for containerized services, especially when you need lower-level HTTP control or WebSockets. API Gateway is attractive w