Routing Workflow with Spring AI: Classify First, Then Answer
What routing means in practice
Most LLM applications start with one prompt that tries to do everything. It classifies the request, answers it, and follows a policy, all in the same instruction. That works for a demo. In production, it becomes hard to tune, hard to test, and hard to explain when one kind of request gets a poor answer because the prompt was written for another kind.
The Routing Workflow pattern takes a different approach. Instead of one general prompt, you first ask the model to decide what kind of input it is looking at. Then you send the input to a prompt written specifically for that kind of input.
Anthropic's article "Building Effective Agents" describes routing as one of the core workflow patterns for LLM systems, and the Spring AI team has published a compact example of it in the spring-ai-examples repository, under agentic-patterns/routing-workflow.
In this article I walk through the pattern, the code in the example, and the practical issues you will hit when you use it: how the classifier is built, why the route key has to match exactly, what happens when the model picks something unexpected, how to test the router, and where the pattern stops being the right tool. I will keep the code close to the example so you can follow along with the repository open.
Routing separates two decisions that a single prompt would otherwise make at once. The first decision is classification: what kind of request is this? Is it a billing question, a technical problem, an account security issue, or a product question? The second decision is handling: given that it is a billing question, what should the response look like?
Separating them has three practical benefits.
- Specialized prompts. A billing prompt can focus on charges, refunds, and payment options without carrying instructions about password resets or feature education. Each prompt stays short and specific.
- Testability. You can evaluate the classifier on a labeled set of tickets and measure how often it picks the right route, separately from measuring the quality of the answers.
- Explainability. When an answer is wrong, you can ask whether the route was wrong or whether the route's prompt was wrong. Those are different problems with different fixes.
The cost is an extra model call per request, and a new failure mode: the classifier can pick the wrong route. The rest of this article is about managing both.
The pattern in one picture
The flow has three steps:
- The input arrives, along with a map of route names to specialized prompts.
- The model reads the input and selects one route, returning a short reasoning and the chosen route name.
- The input is sent to the selected route's prompt, and that response is returned to the caller.
Notice that the router makes two model calls per input: one to classify, and one to answer. That is the main trade-off. If you route cheap, simple requests to a smaller model, the classification step can be a meaningful part of the cost savings. If every request needs the most capable model anyway, routing adds latency and money without a clear gain.
The Spring AI example
The example's main class is RoutingWorkflow, in the package com.example.agentic. It takes a ChatClient in its constructor and exposes one public method:
public String route(String input, Map<String, String> routes)
The method takes the raw input and a map from route names to prompts. It returns the response from the selected route. The core logic is short:
public String route(String input, Map<String, String> routes) {
Assert.notNull(input, "Input text cannot be null");
Assert.notEmpty(routes, "Routes map cannot be null or empty");
String routeKey = determineRoute(input, routes.keySet());
String selectedPrompt = routes.get(routeKey);
if (selectedPrompt == null) {
throw new IllegalArgumentException("Selected route '" + routeKey + "' not found in routes map");
}
return chatClient.prompt(selectedPrompt + "\nInput: " + input).call().content();
}
Three details are worth noticing.
First, the route key returned by the classifier is looked up directly in the map. If the model returns a name that is not a key, the method fails with an IllegalArgumentException. That is a deliberate guard, and it matters a lot, as I will show later.
Second, the selected prompt is concatenated with the input using the same "\nInput: " separator used in the parallelization example. The route prompt ends with Input:, so the input follows it cleanly.
Third, the method does not return the classifier's reasoning. It only returns the final answer. If you need the reasoning for logging or auditing, you have to change the design so that it is returned or recorded.
The classifier
The interesting part is determineRoute:
private String determineRoute(String input, Iterable<String> availableRoutes) {
String selectorPrompt = String.format("""
Analyze the input and select the most appropriate support team from these options: %s
First explain your reasoning, then provide your selection in this JSON format:
\\{ "reasoning": "Brief explanation of why this ticket should be routed to a specific team. Consider key terms, user intent, and urgency level.", "selection": "The chosen team name" \\}
Input: %s""", availableRoutes, input);
RoutingResponse routingResponse = chatClient.prompt(selectorPrompt).call().entity(RoutingResponse.class);
return routingResponse.selection();
}
This method does three things.
It builds a prompt that lists the available route names, asks for reasoning first, and asks for a JSON object with two fields. The order matters: asking for reasoning before the selection tends to make the selection more consistent with the explanation, because the model has written its argument before committing to the answer.
It asks Spring AI to convert the response into a RoutingResponse record, using the entity method:
public record RoutingResponse(String reasoning, String selection) { }
With entity(RoutingResponse.class), Spring AI's structured output support sends a format description to the model and parses the reply into the record. You do not write the JSON parsing yourself, and you get a typed object back.
It returns only the selection. The reasoning is printed to the console in the example, which is useful for development but not something you would ship as-is.
There is a small detail in the template. The JSON braces are escaped as \\{ because the prompt is built with String.format, and the text block would otherwise interpret the braces in a way that causes trouble. If you rewrite this prompt, keep that in mind; it is an easy place to introduce a formatting bug.
Defining the routes
The example's Application class defines four routes as a map of names to prompts:
Map<String, String> supportRoutes = Map.of(
"billing", """
You are a billing support specialist. Follow these guidelines:
1. Always start with "Billing Support Response:"
2. First acknowledge the specific billing issue
3. Explain any charges or discrepancies clearly
4. List concrete next steps with timeline
5. End with payment options if relevant
Keep responses professional but friendly.
Input: """,
"technical", """
You are a technical support engineer. Follow these guidelines:
1. Always start with "Technical Support Response:"
2. List exact steps to resolve the issue
3. Include system requirements if relevant
4. Provide workarounds for common problems
5. End with escalation path if needed
Use clear, numbered steps and technical details.
Input: """,
"account", """
You are an account security specialist. Follow these guidelines: ... Input: """,
"product", """
You are a product specialist. Follow these guidelines: ... Input: """
);
Each route prompt does four things well. It names a specific persona, sets a required opening line, lists the content the response should include, and ends with Input: so the route can append the ticket.
The required opening line is a useful trick. It gives you a cheap way to check, in tests or in monitoring, whether the right route handled the input. If a billing ticket comes back starting with "Technical Support Response:", something went wrong before the answer was written.
The example runs three tickets through the router: a login problem routed to account security, a small unexpected charge routed to billing, and a question about exporting data routed to the product route. The point is not that the classifier is perfect, but that each ticket goes to a prompt that matches its intent.
Why the route key has to match exactly
The most common failure in this pattern is a mismatch between what the model returns and what the map contains. The classifier is given a list of route names and asked to pick one. The model may return Billing instead of billing, or billing team instead of billing, or Technical Support instead of technical. Any of those will fail the map lookup.
In the example, the lookup returns null, and the method throws an exception that names the route it could not find.
You have three ways to handle this, from least to most robust.
- Fail fast. This is what the example does. It is correct for development, because a mismatch is a bug you want to see. In production, it means a failed request, and you need a plan for it.
- Normalize. Before the lookup, trim the string and compare it case-insensitively against the keys. This handles casing and whitespace, but not paraphrases like "billing team."
- Constrain the output. Ask for an enum, or give the model an explicit list of allowed values and validate the result against it. Spring AI's structured output can be combined with an enum-typed field, which narrows the set of possible answers. The model can still pick a wrong route, but it cannot invent a new one.
In practice, I would combine the second and third approaches: constrain the output where possible, normalize the rest, and keep the fail-fast check as a final guard. A route that does not exist should never reach the handling step silently.
Handling unexpected or ambiguous inputs
Routing assumes that every input belongs to one of the known categories. Real inputs do not always cooperate. A ticket may mention a billing issue and a technical problem in the same message. Another may be spam, a test, or a request that none of the routes covers.
The example does not handle these cases, so you need to decide what the system should do. A few options are common.
- Add a fallback route. A route named
generalorotherwith a prompt that asks the model to clarify or escalate catches anything the specialized routes do not cover. This is the simplest safety net, and it is the one that the README's own example uses. - Ask for a confidence value. Extend the classifier's record with a confidence field. If the confidence is below a threshold, send the input to a human queue or to the fallback route. The model's self-reported confidence is not a calibrated probability, so treat the threshold as a tuning parameter, not a guarantee.
- Allow multiple routes. If the input touches several categories, the router can return a list of routes and run each handler, then merge the results. That turns the pattern into a small fan-out, which starts to resemble the parallelization workflow. It is a reasonable extension, but it changes the cost profile.
Whichever you choose, write it down as part of the design. An unhandled ambiguous input is a product decision, not a technical accident.
Testing the router
Routing lends itself well to testing, because there are two separate things to test and they can be tested separately.
The first is the classifier. Build a small labeled set of tickets, each with the route you expect, and measure how often the classifier picks that route. Run it whenever you change the route names, the classifier prompt, or the model. A small table of expected routes is more informative than a handful of spot checks.
The second is the handler. For each route, check that the response starts with the required opening line and follows the instructions in that route's prompt. Because the handler is a plain prompt, you can test it with a stubbed ChatClient that returns a fixed string, which checks the plumbing, and separately with a real model on a small set of inputs, which checks the content.
Here is a unit test for the plumbing, using a fake client that always selects the route you tell it to:
@Test
void routesToTheSelectedPrompt() {
ChatClient client = mock(ChatClient.class, RETURNS_DEEP_STUBS);
// The classifier call returns a RoutingResponse
when(client.prompt(contains("select the most appropriate support team")).call().entity(RoutingResponse.class)).thenReturn(new RoutingResponse("Mentions a charge", "billing"));
// The handler call returns the answer for the selected prompt
when(client.prompt(contains("You are a billing support specialist")).call().content()).thenReturn("Billing Support Response: ...");
RoutingWorkflow workflow = new RoutingWorkflow(client);
String result = workflow.route("I was charged twice", Map.of("billing", "You are a billing support specialist. Input: ", "technical", "You are a technical support engineer. Input: "));
assertTrue(result.startsWith("Billing Support Response:"));
}
This test checks that the router sends the input to the prompt the classifier selected. It does not check whether the classifier is right, which is a question for the labeled evaluation set.
A test for the failure path matters just as much:
@Test
void failsWhenTheClassifierReturnsAnUnknownRoute() {
ChatClient client = mock(ChatClient.class, RETURNS_DEEP_STUBS);
when(client.prompt(anyString()).call().entity(RoutingResponse.class)).thenReturn(new RoutingResponse("Unclear", "Billing Team"));
RoutingWorkflow workflow = new RoutingWorkflow(client);
assertThrows(IllegalArgumentException.
Comments
No comments yet. Start the discussion.