Switchboard: building a tool router so your AI agent stops drowning in MCP tools

Keyword search picks the right tool for an AI agent 21% of the time. The router we built picks it 88% of the time, while cutting the tokens spent describing those tools by 99.6%. This is how we got there, including the parts we got wrong first. Architecture: Link The problem nobody talks about until the bill shows up Model Context Protocol (MCP) solved a real problem. It gave every AI agent a standard way to talk to external tools: CRMs, observability stacks, email, vector databases, internal APIs. It created a quieter problem in the process. Once you connect more than a handful of MCP servers to a single agent, you are no longer describing a tool catalog to the model. You are dumping an entire warehouse inventory into every prompt. In our own environment, three backends exposed 142 tools. Every one of those schemas, with its names, descriptions, and parameter shapes, gets tokenized and sent to the LLM on every turn, whether the user's question needs one of them or none of them. That has three costs, and only one of them shows up on an invoice. The first is token cost. You are paying to re-describe N tools you will never call, every single time. The second is selection accuracy. The more tools an LLM has to choose from in a single context, the more often it picks the wrong one, hallucinates parameters, or gets confused by two similarly-named tools from different servers. The third is operational fragility. Every new MCP server you connect makes the prompt bigger and the agent's job harder, so scaling tool count and scaling reliability end up pulling in opposite directions. We built Switchboard to decouple those three curves. The name is the metaphor: a telephone switchboard operator connects your call to the right line so you never need to know the number. Switchboard does that for tools. The host asks for what it wants in plain language, and the router works out which of N tool to patch it through to. Two tools instead of two hundred Switchboard sits between your AI host (Claude Code, a custom chat agent, an IDE, anything that speaks MCP) and every backend MCP server you own. From the host's point of view Switchboard is an MCP server, but instead of exposing your full tool catalog it exposes exactly two meta-tools. find_tools(request) takes the user's request in plain language and returns a small, dynamically-sized set of relevant tools: zero, one, or a handful, never a fixed top-k and never the whole catalog. invoke(tool_id, args) then calls the selected tool against whichever backend actually owns it. Everything else is the router's problem rather than the host's: which backend hosts which tool, how many servers are connected, how the catalog changes over time. How find_tools actually decides This is the part that took the most iteration, because search alone is not enough. A naive nearest-neighbor lookup either returns too much, which defeats the purpose, or misses the right tool because of vocabulary mismatch between how a user asks and how a tool is described. The retrieval pipeline runs in four stages. It starts with concurrent dense and sparse search against a Pinecone vector registry. Dense embeddings catch semantic similarity, so "send the report to finance" finds send_email_with_attachment . Sparse keyword-style search catches the exact-term matches dense embeddings sometimes miss: API names, service identifiers, error codes. Both index round-trips are independent I/O, so they run in parallel and cost one round-trip of wall-clock rather than two. Next, a cosine-similarity gate filters out anything too far from the query's intent before it ever reaches the LLM. It is cheap and deterministic, and it keeps obviously irrelevant tools from wasting judge tokens. Then an LLM judge does the expensive part properly: capability-fit selection, de-duplication across near-identical tools from different backends, ordering by execution sequence, and deciding to ask a clarifying question instead of guessing when the request is genuinely ambiguous. Finally, the result count is fully dynamic. Most routing systems force a fixed top-k. We don't. Some queries need zero tools, some need exactly one, and a few legitimately need several, so the judge decides rather than a hardcoded number. If find_tools comes back empty there is a fallback, find_more_tools , that relaxes the gate before giving up and clarifying. It is a second chance before the decision gets punted back to the user. How we measured accuracy, and why the grader is strict The headline is 85 to 90% accuracy on a held-out suite of 70 realistic multi-tool queries, against a roughly 21% naive keyword-search baseline on the same suite. The baseline matters more than the headline, because it is the difference between "the pipeline works" and "an embedding lookup would have done fine." The grader is deliberately unforgiving. A case passes only if every required tool appears in the result and no forbidden tool appears, where forbidden: means the correct answer is no tools at all. For ["*"]allow_any_of groups, which are sets of interchangeable tools, exactly one member must be present. Returning two valid alternatives is a failure rather than a hedge, because it pushes the choice back onto the model we are trying to protect. When an order is specified, the returned tools have to contain it as a subsequence, with relative order preserved and unrelated tools allowed to interleave. And when a case is marked should_clarify , the router must return zero tools and set the clarify flag, which alone decides the case. That last rule is the one I would defend hardest. An accuracy metric that does not reward asking instead of guessing quietly incentivizes overconfident wrong answers, because the model learns that any answer beats admitting ambiguity. Ours treats correct abstention as a pass, which means the 88% includes the system knowing what it does not know. One honest limitation: cases requiring the same tool to be called multiple times are structurally unsatisfiable, because route() emits each tool id once. We count those as real misses rather than excluding them, so the reported number is a floor rather than a flattered figure. The decisions that mattered, and the alternatives we rejected Most of the interesting engineering here is not in what the pipeline does. It is in the four or five places where we deliberately chose the harder option. Dynamic-K over fixed top-k The obvious design is "return the top 5 matches." We rejected it because a fixed k is wrong in both directions at once. For a query that needs one tool, top-5 injects four irrelevant schemas and reintroduces exactly the selection-confusion problem the router exists to solve. For a genuine multi-step request, top-5 might truncate a plan that needed six. And for an out-of-scope question, top-5 confidently returns five wrong tools. Making k dynamic means the judge has to answer "how many?" as well as "which?", which is a harder prompt and a harder thing to evaluate. It was still the right trade. Our measured average is 1.2 tools per call out of 142, which no fixed k would have produced. One hard-coded rule, and only one There is a real temptation to encode catalog-specific heuristics, something like "queries mentioning 'log' should prefer the observability server." We kept exactly one code-side rule: an absolute cosine floor for out-of-scope detection. The reasoning is a division of labor. That floor is the one judgment the LLM cannot make cheaply. To know that nothing in the catalog fits, a judge would have to see the entire catalog, which is precisely the cost we are eliminating. A cosine threshold answers it in one vector op. Everything else (which tools, dedup, ambiguity, ordering) is semantic work and gets delegated to the judge. Rules that encode catalog specifics would need rewriting every time someone connects a new backend, which defeats the pluggability goal. Judge order, not score order Early on we sorted the returned tools by cosine score. That is wrong for multi-step requests. "Pull last week's errors and open a ticket for the worst one" has an inherent execution order that has nothing to do with which tool embeds closer to the query. The judge reasons about sequence, so the judge's output order is the router's output order, and we explicitly do not re-sort. Filter unhealthy backends before the judge, not after The intuitive design for backend health is retry-on-failure: select a tool, call it, and handle the error if the backend is down. We invert it. A backend marked down has its tools excluded from the candidate set before the judge sees them, re-evaluated on every call. This matters because of the specific failure it prevents. The judge picks the perfect tool, explains its reasoning, and then invoke fails, so the user gets a wrong-looking answer for a right-looking decision. Filtering early means the judge selects the best reachable tool instead, possibly a second-choice tool on a healthy backend, which is the correct behavior. We also chose reactive health detection over a heartbeat loop. A backend gets marked down on an actual failed call and retried on next use. A separate polling process is one more thing to keep in sync with reality, and health checks that themselves flake produce false "down" states. Deprecated tools stay fetchable Our first filter dropped every tool marked deprecated. We removed that clause deliberately, because a request like "export it in the legacy format" specifically needs the deprecated version. Governance filtering still applies, with PII-touching tools gated behind an explicit flag and destructive tools behind another, but deprecation is metadata for the judge to weigh rather than a hard exclusion for code to enforce. Keeping the registry honest A tool router is only as good as its index, and tool catalogs are not static. Backends add tools, deprecate others, and change descriptions. Switchboard runs a background ingestion pipeline, independent of request-time tr