From Prompt to Platform: How I Would Architect a Production-Grade GPT Application

Building a GPT application looks deceptively easy. The first version might be twenty lines of code: user input → prompt → LLM → response. That's technically an application. But now imagine the requirements grow: answer questions from private data, search documents, call internal APIs, update records, remember previous conversations, choose between multiple models, execute multi-step tasks, pause for human approval, resume hours later, recover from partial failure, stream responses, support thousands of organizations without leaking data between them, explain exactly why an action happened, and keep the infrastructure bill under control. Suddenly this is no longer "call GPT and return the response." It's closer to building a distributed workflow engine with a probabilistic decision-maker sitting inside the execution loop. That is the architecture I want to explore - not the demo, the system behind the demo. One caveat before I go further, because it's the kind of thing I've watched teams get wrong in both directions: nobody should build all of this on day one. Everything below is a destination, not a starting point. A team of four shipping their first internal agent should build the twenty-line version, ship it, and only bolt on a piece of this architecture when a specific failure mode actually bites them - a duplicate refund, an unexplainable account deletion, a runaway retry loop. The value of laying out the whole map up front isn't "go build all of this." It's so that when you do hit one of these failure modes, you recognize it instantly instead of reinventing a worse version of the fix under incident pressure.

Sequencing Near the End

Start With the Mental Model

A conventional request-driven application often looks like:

• Request → Business Logic → Database → Response

A production GPT application is fundamentally different. It looks closer to this:

• The execution path is dynamic.
• The application doesn't necessarily know beforehand whether a request needs 1 model call, or 4 model calls plus 2 database queries plus a vector search plus 3 API calls plus an approval plus a retry.

That uncertainty changes the system-design problem: the system must control a computation whose path, duration, resource consumption, and external side effects are only partially predictable. A REST endpoint has a bounded, mostly-known cost envelope - you can load-test it and know roughly what you're getting. An agent loop's cost envelope is a distribution, and a long tail of it involves the model deciding to do something you didn't anticipate. Every design decision below is really a decision about how to bound that distribution without also bounding away the thing that makes the system useful.

The Architecture I'd Start With

At a high level, I'd divide the platform into four planes. There's also another split that matters more than the diagram suggests: control plane vs. data plane. The control plane defines what agents are allowed to be; the data plane executes what agents actually do. In practice the control plane is owned jointly by platform and security - they decide what an agent is permitted to do - while the data plane is owned by the product teams building specific agents on top of it. Skipping this split and you get the failure mode I've seen most often: every product team hand-rolls its own policy checks inline in application code, security has no single place to audit what agents can do across the company, and a change to "who can approve a refund" requires a code change and a redeploy in six different services instead of one config change in one place.

The Orchestrator

The orchestrator shouldn't contain the intelligence itself - its job is to control execution:

while task not terminal :
    state = load_state()
    context = construct_context(state)
    decision = model(context)
    validate(decision)

    if decision == TOOL_CALL :
        result = execute_tool()
        append_observation(result)
    elif decision == NEEDS_APPROVAL :
        checkpoint()
        suspend()
    elif decision == FINAL :
        persist()
        return checkpoint()

That looks simple. The production implementation is not - it has to handle retries, timeouts, concurrent workers, partial failures, and resumption: this is effectively a workflow engine, which is why I'd model the execution explicitly as a state machine, not "messages in an array." An agent needs answerable state: can a task be resumed? can an approval expire? is another worker already executing it? was the previous tool action committed? That's much safer than reconstructing execution state from chat history - and it doubles as your incident-response tool. When something breaks at 2 a.m., the question you need answered fastest is "what state is this workflow in, and is it safe to just re-run it?" If your only record of execution is a chat transcript, someone has to infer the state from prose. If it's an enum in a row, they can query it.

Why not just use an existing durable-execution engine - Temporal, AWS Step Functions, or similar - instead of building this loop by hand? For many teams, you should; reinventing leases, retries, and checkpointing is often wasted effort. Where I've seen teams need a custom orchestrator anyway is when the "steps" of the workflow aren't known in advance - the model is choosing the next step at runtime, not walking a pre-declared DAG. In that case you often end up using Temporal as the durability layer underneath a thinner custom loop, rather than replacing it outright.

Prompt Construction Is Infrastructure, Not String Concatenation

A production system shouldn't have prompt = system_prompt + history + user_message. Context construction should be deterministic infrastructure - a compiler that decides how much conversation history to include, which memories matter, which documents belong in context, which tool schemas are necessary, what must never be truncated. That leads naturally to context budgeting: treat the context window like memory, not like an infinite string buffer. The failure mode I'd flag here: teams that skip this almost always find out the hard way, in production, that "just include everything, the context window is huge now" degrades quality long before it hits the token limit. Models get measurably worse at precise instruction-following and tool-argument accuracy as irrelevant context grows, even well under the stated window size. Budgeting isn't just token accounting - it's a quality lever.

Retrieval Authorization

Happens Before the Model, Not After One retrieval mechanism is rarely optimal for everything - I prefer hybrid retrieval (vector + keyword + structured queries) with a reranker. But the detail that matters most is where authorization sits: permission filtering happens before the content reaches the model, not afterward. The model cannot leak information it never received. I've seen this get built backwards more than once: teams build the retrieval pipeline first, get it working end to end, and only then bolt permission filtering onto the output - "we'll just strip out anything the user can't see before we show the answer." That's the wrong side of the boundary. If unauthorized content ever reaches the prompt, it's already been read by the model, and a sufficiently adversarial follow-up question or a prompt injection in a retrieved document can surface fragments of it regardless of what you do to the final response. Filter before the model sees it, not after it answers.

Memory Is an Architecture, Not a Feature

When products advertise "memory," multiple different systems are usually hiding underneath. I'd separate at least four categories: Workflow state, conversation, semantic memory, and audit history each have different latency and consistency needs, and different natural storage (Redis/DB, OLTP, vector store, append-only log respectively). Trying to solve all four with a single vector database is a design smell. If I had to rank these by how often teams get them wrong, it's semantic memory, by a wide margin. "Remember what the user told us" sounds like a retrieval problem, so it gets implemented as one - embed everything, retrieve by similarity, done. But facts have a lifecycle that similarity search doesn't model at all: they get superseded, contradicted, or scoped to a time window. A pure vector store will happily retrieve the stale fact alongside the current one, ranked by embedding distance rather than recency or validity. If semantic memory matters to your product, it needs fact versioning and supersession sitting on top of the vector index, not instead of it. Build a semantic memory layer on top of the vector store, not replace it.

Applications Shouldn't Scatter Direct Model Calls

Applications shouldn't scatter direct model-provider calls throughout the codebase. I'd introduce a model gateway responsible for provider abstraction, routing, fallbacks, timeouts, retries, token accounting, and safety configuration:

The orchestrator asks for a capability (execute(capability="complex_reasoning", latency_class="interactive", max_cost=X)) instead of hardcoding one specific model everywhere.

The return on this is almost entirely deferred, which is exactly why teams skip it under deadline pressure - and exactly why it's worth the small up-front cost anyway. The gateway pays for itself the first time a provider has a bad day, a model gets deprecated, or finance asks "which feature is burning our token budget" and you can answer from one place instead of grepping through a dozen services. Routing doesn't need to happen once per request, either - different steps of the same workflow can use different models (cheap model for intent classification and extraction, reasoning model for root-cause analysis, cheap model again for formatting). That can meaningfully change the economics of the whole platform, though it's worth naming the trap: every routing tweak is a behavior change that needs the same evaluation rigor as a prompt change. "We saved 40% on token spend" isn't a complete sentence if nobody checked whether task success rate moved too.

Treat Model Output as Untrusted Input

An LLM output is not executable truth. If the model proposes transfer_money(amount: "ONE MILLION DOLLARS!"), the system shouldn't casually convert that into an API call. Model output needs the same pipeline as any other untrusted input:

1. Model Output → Schema Validation
2. Semantic Validation
3. Authorization
4. Risk Classification
5. Human Approval?
6. Execution

The LLM's role is to propose an action. The platform's role is to determine whether that proposal is valid, authorized, safe, and executable. Those stay separate - which is also why tool descriptions are part of your security surface, not just your prompt-engineering surface. The tool executor itself should own credentials, retries, idempotency, timeouts, concurrency limits, and audit logging - the model should own none of it.

Idempotency Becomes Critical the Moment Agents Can Write

Idempotency becomes critical the moment agents can write. Consider this sequence:

• T0Agent decides to issue refund
• T1Tool executor sends request
• T2Payment processor commits refund
• T3Network connection fails
• T4Orchestrator sees timeout → retries

Without protection, that's two ₹5,000 refunds for one request. The idempotency key belongs to the logical action, not the network request - generated once, stored durably alongside workflow state, and reused on every retry of that same action. The subtlety that bites people: if the retry path generates a new key because "well, this is technically a new attempt," you've reintroduced the exact bug idempotency was supposed to close. The key's identity is "refund attempt #1 for ticket #482," not "this POST request." A related but distinct problem is the dual-write failure: the database updates but the event that should have fired never publishes, or the tool action completes but the workflow checkpoint never persists. Where the platform owns both pieces of state, a transactional outbox - writing the event durably in the same transaction as the state change, then relaying it later - turns "did I lose the event?" into "the event is durably pending delivery." It's worth knowing which of the two you're solving at any given seam: idempotency for the external side effect, an outbox for the internal dual write.

Human Approval Should Be a Durable Pause

For high-risk actions ("delete all inactive customer environments"), the workflow should checkpoint, persist as WAITING_FOR_APPROVAL, and go fully dormant - no thread waiting, no pod alive, no six-hour-open HTTP request. An approval event wakes it back up. That's the difference between implementing approval as a modal dialog and designing it as infrastructure. One decision this forces you to make explicitly: what happens if nobody approves within a reasonable window? "Wait forever" is rarely right for anything customer-facing. An expiring approval with a defined fallback - auto-deny, escalate, re-prompt - is part of the state machine, not an afterthought bolted on later. If WAITING_FOR_APPROVAL has no TTL, you will eventually find a workflow that's been sitting there for four months. The same discipline extends to checkpointing every meaningful transition (intent parsed, documents retrieved, plan generated, approval received, mutation completed) so a recovering worker restarts from the last safe point instead of from step one - and to concurrency control, so two workers can't both advance the same workflow. A lease (owner, lease_expires_at, version) or simple optimistic concurrency (UPDATE ... WHERE version = 84) handles this; if you already have a Postgres-backed workflow table, SELECT ... FOR UPDATE SKIP LOCKED or a version-column compare-and-swap gets you most of the way there without a separate lease service.

Failure Needs Classification, Not a Blanket Retry

"Retry three times" is not a strategy. A transient network failure wants different treatment than a logic error. The orchestration layer should classify failures and respond appropriately - exponential backoff with jitter, circuit breakers for downstream dependencies, dead-letter queues for persistent failures. Each category deserves its own handling policy, and the documentation of those policies should live alongside the code, not in a separate wiki page that gets outdated during incidents.