Infrastructure requirements for running agents in production · TokenGO

What agents lack in production isn't the model. It's four foundational primitives: timeouts, retries and circuit breaking, persisted state, and tracing. What agents lack in production isn't the model; it's four foundational primitives. Many teams moving agents from demo to production find that the first hurdle isn't model performance, but infrastructure. Running successfully once in a demo doesn't mean it can run day in and day out. We've helped many teams navigate this pitfall; what's usually missing are a few foundational setups. Timeouts, retries, state, and tracing as the production-ready toolkit Timeouts: don't let a slow API drag down the entire chain When an agent calls a tool and the other side doesn't respond, it can't wait indefinitely. Every call needs a timeout. When the time is up, it must either switch paths or throw an error. We once saw an agent where a half-dead third-party API stalled the main chain for 40 minutes. The user thought the system had crashed, but it was just stuck on a single call without a timeout. Retries and circuit breaking: back off on failure, trip the circuit on continuous failure Occasional tool failures are normal, and a few automatic retries are fine. But if the same API fails continuously, you have to break the circuit. Don't stubbornly hammer a dead service. We've seen an agent exhaust its quota with dozens of retries just because a downstream service blipped for a second. The right approach is retrying with backoff, and tripping the circuit after a certain number of consecutive failures, then probing again later. State: resuming from breakpoints after a restart If a machine restarts halfway through a long-running task, can it resume from the breakpoint instead of starting over? This requires the state of each step to be persisted, not just held in memory. If the state isn't persisted to disk, a restart means a complete rerun, and you simply cannot risk putting long-running tasks into production. Tracing: pulling up the entire chain when one step is slow When an agent executes over a dozen steps, you need to be able to pull up the entire execution trace to see exactly which step was slow or threw an error. Without tracing, debugging production issues is just guesswork. When onboarding customers to production environments, the very first thing we usually do is implement tracing. Once, a customer complained about a particularly slow response. We pulled up the trace and instantly saw that a retrieval tool was taking eight seconds. It wasn't a model issue, and troubleshooting took all of ten seconds. We are rolling out our free agent observability feature in waves now. TokenGo Agent Tracing. Architecture: stateful orchestration layer + stateless tool layer None of these are difficult on their own. The hard part is that they must exist as a cohesive whole, ideally provided by the platform so that every business team doesn't have to reinvent the wheel. We favor a two-layer architecture: a stateful orchestration layer that remembers where the task is and can resume after restarts, and a stateless tool layer that can be spun up or swapped out at any time. If one dies, just swap it. The two layers communicate using the primitives mentioned above. If a tool goes down, the orchestration layer switches to another one and keeps running; the user on the other end might only wait an extra two seconds. We built a reconciliation agent for a financial client where the tool layer went down twice; both times, the orchestration layer switched to backups in seconds, and the business side was completely oblivious. Stateful orchestration remembers the job. Stateless tools can be swapped. Taking agents to production is about whether it can pick itself back up after crashing. With a complete set of primitives, it can recover from crashes invisibly to the user. How this plays out in a real outage Let's look at a concrete example. In that reconciliation agent, the retrieval service in the tool layer crashed. The orchestration layer had a timeout set before the call; when it didn't get a response in eight seconds, it marked it as a failure and triggered the circuit breaker, stopping further requests to the dead service. The orchestration layer then swapped to a backup retrieval instance via an alternative path, recovering within ten seconds. The reconciliation task wasn't interrupted at all, and the business side noticed nothing. Looking at the trace afterward, the only anomaly in the entire chain was that the retrieval step jumped from its usual two seconds to an eight-second timeout while the rest proceeded normally. With the timeout + circuit breaker together, these two primitives turned an outage into a seamless switch. Timeout, circuit open, backup path: the task never stopped When we set up production environments for customers, the first thing we do is enable these four primitives by default. We don't leave it as an optional add-on. You don't think about these things during the demo phase, but if you wait until things break in production to patch them, the cost is immense. Internally, we call these four points the "Production-Ready Toolkit." When a new customer onboards, we first run a set of inspection scripts across their agent. Any missing primitive is flagged in red and prioritized for fixing, so they aren't left flying blind. Only when all four are green do we confidently say this agent is ready for production. The difference between running in a demo and running day in and day out is whether these four primitives are fully in place. For another client, before going live, our scan revealed they were missing state persistence and tracing. After patching those, their average alert troubleshooting time dropped from two hours to ten minutes. Alert troubleshooting dropped from two hours to ten minutes We can perform production readiness checks for every business client we onboard. Whether they are missing timeouts, retries, state, or tracing, we fix it first. We also document the mechanics of these primitives for the customers to read. Here at TokenGo we believe trust comes from transparency. Top comments (1) Wow very cool dude! I'm gonna head over to tokengo.com right now and get me some inference!