Implementing Structured Output Retries Without Infinite Agent Loops
Structured outputs are one of the most useful capabilities in modern LLM applications. Instead of returning unstructured text, a model can generate JSON objects that applications use for classification, data extraction, tool execution, and workflow decisions.
But structured output does not automatically mean reliable output. A model may return valid JSON with missing fields, unsupported values, incorrect data types, or arguments that violate business rules.
The usual solution is to retry the model with an explanation of what went wrong. That approach works until the retry logic becomes another source of failure. An agent may repeatedly generate invalid responses, retry with nearly identical instructions, and continue consuming tokens without making progress.
Reliable structured output handling requires more than schema validation. It needs bounded retries, explicit failure states, and a clear distinction between recoverable formatting errors and application-level failures.
In this guide, we'll build a lightweight structured output retry mechanism using Python and Pydantic. We'll also add regression tests, retry budgets, failure classification, and safeguards against infinite agent loops.
Why Structured Output Retries Become Dangerous
Consider an AI agent that extracts invoice information from documents. The application expects:
{ "invoice_id" : "INV-1042" , "amount" : 1250.50 , "currency" : "USD" , "approved" : false }
The response must satisfy several requirements:
invoice_idmust be a non-empty string.amountmust be greater than zero.currencymust use a supported currency code.approvedmust be a Boolean.
Now imagine the model returns:
{ "invoice_id" : "INV-1042" , "amount" : "unknown" , "currency" : "USD" , "approved" : "maybe" }
The JSON syntax is valid, but the data violates the expected schema. A developer might implement:
while True :
response = call_model ( )
try :
return validate ( response )
except Exception :
continue
This code has several problems. First, the loop has no termination condition. Second, it retries on every exception, including programming errors that another model call cannot fix. Third, it provides no feedback explaining why validation failed. Fourth, it may repeatedly generate the same invalid response.
In a production agent, this can increase cost, latency, and infrastructure usage while preventing the workflow from completing. The solution is to make retries an explicit part of the application's control flow.
Separate JSON Parsing From Schema Validation
Structured output failures occur at different levels.
JSON syntax errors
The response cannot be parsed as JSON. Example:
{ "invoice_id" : "INV-1042" , "amount" : 1250.50 , }The trailing comma makes this invalid JSON.
Schema validation errors
The JSON parses successfully but does not match the expected structure. Example:
{ "invoice_id" : "INV-1042" , "amount" : "unknown" }Required fields are missing or contain invalid values.
Business-rule violations
The output matches the schema but violates an application-specific requirement. For example, an invoice may have a valid positive amount but exceed the amount the user is authorized to approve.
Tool execution failures
The structured output may be correct, but the external API rejects the operation or becomes unavailable.
These failures should not share the same retry policy. A formatting problem may justify asking the model to regenerate its response. An authorization failure should generally stop execution. A network timeout may require an infrastructure-level retry rather than another model generation.
The first design rule is to classify failures before deciding whether another LLM call is appropriate.
Project Structure
We'll build a small example with the following files:
structured-output-retries/
|-- models.py
|-- retry_handler.py
|-- test_retry_handler.py
|-- requirements.txt
-- .github/
-- workflows/
-- tests.yml`
The implementation uses Python 3.11 or later. Install the dependencies:
python -m pip install "pydantic>=2,<3" pytest
Create requirements.txt:
pydantic>=2,<3
pytest>=8,<10
The retry handler will use a callable model interface rather than depending on a specific LLM provider. That makes the implementation easier to test.
Define a Strict Structured Output Schema
Create models.py:
from typing import Literal
from pydantic import (
BaseModel,
ConfigDict,
Field
)
class InvoiceExtraction(BaseModel):
model_config = ConfigDict(
extra="forbid",
strict=True
)
invoice_id: str = Field( min_length=1 )
amount: float = Field( gt=0 )
currency: Literal[ "USD", "EUR", "GBP" ]
approved: bool
This schema enforces several constraints. The extra="forbid" configuration rejects unexpected fields. Strict validation prevents implicit type coercion where supported by the field type. The amount must be positive. The currency must match one of the allowed values. The approved field must be a Boolean.
Why strict validation matters
Without strict validation, some frameworks may automatically convert values. For example, a string containing a number might be accepted as a numeric field. That behavior can be convenient for ordinary form processing. For AI-generated outputs, however, it may hide inconsistent model behavior.
Strict validation makes failures more visible and easier to evaluate.
Schema validation is not authorization
The approved field is included to demonstrate Boolean validation. It must not be treated as proof that a real invoice has been authorized. A production payment or approval system should obtain authorization from trusted application state and enforce it independently of the model-generated output.
Define Explicit Retry Limits
The retry mechanism needs a clear execution budget. We'll use three controls:
- Maximum model attempts.
- Maximum repeated identical invalid responses.
- A wall-clock deadline checked between attempts.
These controls address different failure patterns. The attempt limit prevents unlimited model calls. The repeated-response limit detects situations where the model produces the same invalid result without improvement. The deadline prevents the retry handler from continuing after the allotted execution window.
A deadline check between calls cannot interrupt a model request that is already running. The model client must enforce its own request timeout.
Define retry-specific exceptions. Create retry_handler.py:
import json
import time
from collections.abc import Callable
from dataclasses import dataclass
from pydantic import ValidationError
from models import InvoiceExtraction
class StructuredOutputError(Exception):
pass
class RetryBudgetExceeded( StructuredOutputError ):
pass
class RepeatedInvalidOutput( StructuredOutputError ):
pass
class RetryDeadlineExceeded( StructuredOutputError ):
pass
@dataclass(frozen=True)
class RetryPolicy:
max_attempts: int = 3
max_identical_failures: int = 2
timeout_seconds: float = 30.0
def __post_init__(self):
if self.max_attempts < 1:
raise ValueError( "max_attempts must be positive" )
if self.max_identical_failures < 1:
raise ValueError( "max_identical_failures must be positive" )
if self.timeout_seconds <= 0:
raise ValueError( "timeout_seconds must be positive" )
The policy is immutable. This helps prevent retry limits from being modified accidentally during execution.
Implement Structured Output Validation
Next, create a function that parses and validates the model response. Add this to retry_handler.py:
def parse_structured_output( raw_response: str ) -> InvoiceExtraction:
try:
payload = json.loads(raw_response)
except json.JSONDecodeError as exc:
raise StructuredOutputError( f"Invalid JSON: {exc.msg}" ) from exc
try:
return InvoiceExtraction.model_validate( payload )
except ValidationError as exc:
errors = exc.errors( include_url=False, include_input=False )
raise StructuredOutputError( f"Schema validation failed: {errors}" ) from exc
This function handles two specific failure categories. If JSON parsing fails, it raises a structured output error. If Pydantic validation fails, it raises a structured output error containing validation details.
Other unexpected exceptions are not converted into retryable errors. That distinction is important. A programming error should not trigger repeated model calls.
Implement the Bounded Retry Handler
Now we can build the retry loop. The handler accepts a callable that generates a model response. The callable receives optional validation feedback from the previous attempt. Add this to retry_handler.py:
def generate_with_retries( model_call: Callable[ [str | None], str ], policy: RetryPolicy ) -> InvoiceExtraction:
started_at = time.monotonic()
previous_invalid_output = None
identical_failures = 0
validation_feedback = None
for attempt in range( 1, policy.max_attempts + 1 ):
elapsed = ( time.monotonic() - started_at )
if elapsed >= policy.timeout_seconds:
raise RetryDeadlineExceeded( "Retry deadline exceeded" )
raw_response = model_call( validation_feedback )
try:
return parse_structured_output( raw_response )
except StructuredOutputError as exc:
if raw_response == previous_invalid_output:
identical_failures += 1
else:
identical_failures = 1
previous_invalid_output = raw_response
if ( identical_failures >= policy.max_identical_failures ):
raise RepeatedInvalidOutput( "Model repeatedly returned " "the same invalid output" ) from exc
validation_feedback = str(exc)
raise RetryBudgetExceeded( "Maximum structured output " "attempts exceeded" )
This implementation guarantees that the handler initiates no more than the configured number of model calls. It also detects repeated identical invalid responses.
The retry loop terminates when:
- The model produces valid structured output.
- The attempt budget is exhausted.
- The repeated-invalid-output limit is reached.
- The deadline is exceeded before an attempt.
Why the loop is safer
There is no while True. The number of attempts is explicit. Only structured output validation errors are retried. Unexpected model-client exceptions propagate to the caller. The handler does not execute any external business action. It only produces validated data.
That separation reduces the risk of repeating side effects during output repair.
Provide Validation Feedback to the Model
Retries are more useful when the model knows what failed. Instead of repeating the original prompt unchanged, provide concise validation feedback. For example:
Your previous response failed validation.
Errors:
amount must be a positive number
approved must be a Boolean
currency must be USD, EUR, or GBP
Return only a valid JSON object matching the schema.
The retry handler already passes validation feedback to the model callable. A provider-specific adapter can use that feedback to construct the next request. Conceptually:
def model_call( validation_feedback: str | None ) -> str:
messages = [
{
"role": "system",
"content": (
"Extract invoice information "
"as valid JSON."
)
}
]
if validation_feedback:
messages.append({
"role": "user",
"content": (
"The previous output failed "
"validation:\n" + validation_feedback
)
})
return call_llm_provider(messages)
This is integration pseudocode. The call_llm_provider function must be implemented using the selected model provider's SDK.
Avoid feeding unlimited error history back to the model. Some retry implementations append every failed response and validation error to the conversation. Over time, the prompt becomes larger and more confusing.
A better approach is to keep only the information needed for the next repair attempt. For example:
- The original task.
- The required schema.
- The latest validation errors.
- Relevant source data.
This keeps the repair process focused. Validation feedback should also be sanitized and bounded before being included in a prompt.
Test Successful Recovery After an Invalid Response
We can test the retry handler without calling a real LLM. Create test_retry_handler.py:
from retry_handler import (
RetryPolicy,
generate_with_retries,
RetryBudgetExceeded,
RepeatedInvalidOutput
)
def test_recovers_after_invalid_json():
responses = iter([
'{"invoice_id": "INV-1",',
(
'{"invoice_id": "INV-1", '
'"amount": 100.0, '
'"currency": "USD", '
'"approved": false}'
)
])
feedback_received = []
def fake_model(feedback):
feedback_received.append( feedback )
return next(responses)
result = generate_with_retries(
fake_model,
RetryPolicy( max_attempts=3, max_identical_failures=2 )
)
assert result.invoice_id == "INV-1"
assert result.amount == 100.0
assert result.approved is False
assert feedback_received[0] is None
assert feedback_received[1] is not None
The first response contains invalid JSON. The second response is valid. The test verifies that the retry handler recovers successfully and provides feedback after the first failure.
Test Retry Budget Exhaustion
A reliable retry mechanism must stop when the model continues producing invalid output. Add this test:
def test_retry_budget_exceeded():
responses = iter([
'{"amount": "invalid"}',
'{"currency": "unknown"}',
'{"approved": "maybe"}'
])
calls = 0
def fake_model(feedback):
nonlocal calls
calls += 1
return next(responses)
try:
generate_with_retries(
fake_model,
RetryPolicy( max_attempts=3, max_identical_failures=3 )
)
except RetryBudgetExceeded:
pass
else:
raise AssertionError( "Expected retry budget failure" )
assert calls == 3
The model produces three different invalid responses. The handler stops after the configured maximum of three attempts. This prevents an infinite loop even when the model never returns valid data.
Detect Repeated Invalid Responses
Some models repeatedly produce the same malformed structure. For example:
{ "invoice_id": "INV-1042", "amount": "unknown" }
If the model returns this exact response multiple times, another retry may not be useful. Add the following test:
def test_repeated_invalid_output():
calls = 0
def fake_model(feedback):
nonlocal calls
calls += 1
re
Comments
No comments yet. Start the discussion.