DEV Community

Implementing Structured Output Retries Without Infinite Agent Loops

Structured outputs are one of the most useful capabilities in modern LLM applications. Instead of returning unstructured text, a model can generate JSON objects that applications use for classification, data extraction, tool execution, and workflow decisions.

But structured output does not automatically mean reliable output. A model may return valid JSON with missing fields, unsupported values, incorrect data types, or arguments that violate business rules.

The usual solution is to retry the model with an explanation of what went wrong. That approach works until the retry logic becomes another source of failure. An agent may repeatedly generate invalid responses, retry with nearly identical instructions, and continue consuming tokens without making progress.

Reliable structured output handling requires more than schema validation. It needs bounded retries, explicit failure states, and a clear distinction between recoverable formatting errors and application-level failures.

In this guide, we'll build a lightweight structured output retry mechanism using Python and Pydantic. We'll also add regression tests, retry budgets, failure classification, and safeguards against infinite agent loops.

Why Structured Output Retries Become Dangerous

Consider an AI agent that extracts invoice information from documents. The application expects:

{ "invoice_id" : "INV-1042" , "amount" : 1250.50 , "currency" : "USD" , "approved" : false }

The response must satisfy several requirements:

  • invoice_id must be a non-empty string.
  • amount must be greater than zero.
  • currency must use a supported currency code.
  • approved must be a Boolean.

Now imagine the model returns:

{ "invoice_id" : "INV-1042" , "amount" : "unknown" , "currency" : "USD" , "approved" : "maybe" }

The JSON syntax is valid, but the data violates the expected schema. A developer might implement:

while True :
    response = call_model ( )
    try :
        return validate ( response )
    except Exception :
        continue

This code has several problems. First, the loop has no termination condition. Second, it retries on every exception, including programming errors that another model call cannot fix. Third, it provides no feedback explaining why validation failed. Fourth, it may repeatedly generate the same invalid response.

In a production agent, this can increase cost, latency, and infrastructure usage while preventing the workflow from completing. The solution is to make retries an explicit part of the application's control flow.

Separate JSON Parsing From Schema Validation

Structured output failures occur at different levels.

  • JSON syntax errors

    The response cannot be parsed as JSON. Example:

    { "invoice_id" : "INV-1042" , "amount" : 1250.50 , }
    

    The trailing comma makes this invalid JSON.

  • Schema validation errors

    The JSON parses successfully but does not match the expected structure. Example:

    { "invoice_id" : "INV-1042" , "amount" : "unknown" }
    

    Required fields are missing or contain invalid values.

  • Business-rule violations

    The output matches the schema but violates an application-specific requirement. For example, an invoice may have a valid positive amount but exceed the amount the user is authorized to approve.

  • Tool execution failures

    The structured output may be correct, but the external API rejects the operation or becomes unavailable.

These failures should not share the same retry policy. A formatting problem may justify asking the model to regenerate its response. An authorization failure should generally stop execution. A network timeout may require an infrastructure-level retry rather than another model generation.

The first design rule is to classify failures before deciding whether another LLM call is appropriate.

Project Structure

We'll build a small example with the following files:

structured-output-retries/
  |-- models.py
  |-- retry_handler.py
  |-- test_retry_handler.py
  |-- requirements.txt
  -- .github/
    -- workflows/
      -- tests.yml`

The implementation uses Python 3.11 or later. Install the dependencies:

python -m pip install "pydantic>=2,<3" pytest

Create requirements.txt:

pydantic>=2,<3
pytest>=8,<10

The retry handler will use a callable model interface rather than depending on a specific LLM provider. That makes the implementation easier to test.

Define a Strict Structured Output Schema

Create models.py:

from typing import Literal
from pydantic import (
    BaseModel,
    ConfigDict,
    Field
)

class InvoiceExtraction(BaseModel):
    model_config = ConfigDict(
        extra="forbid",
        strict=True
    )
    invoice_id: str = Field( min_length=1 )
    amount: float = Field( gt=0 )
    currency: Literal[ "USD", "EUR", "GBP" ]
    approved: bool

This schema enforces several constraints. The extra="forbid" configuration rejects unexpected fields. Strict validation prevents implicit type coercion where supported by the field type. The amount must be positive. The currency must match one of the allowed values. The approved field must be a Boolean.

Why strict validation matters

Without strict validation, some frameworks may automatically convert values. For example, a string containing a number might be accepted as a numeric field. That behavior can be convenient for ordinary form processing. For AI-generated outputs, however, it may hide inconsistent model behavior.

Strict validation makes failures more visible and easier to evaluate.

Schema validation is not authorization

The approved field is included to demonstrate Boolean validation. It must not be treated as proof that a real invoice has been authorized. A production payment or approval system should obtain authorization from trusted application state and enforce it independently of the model-generated output.

Define Explicit Retry Limits

The retry mechanism needs a clear execution budget. We'll use three controls:

  • Maximum model attempts.
  • Maximum repeated identical invalid responses.
  • A wall-clock deadline checked between attempts.

These controls address different failure patterns. The attempt limit prevents unlimited model calls. The repeated-response limit detects situations where the model produces the same invalid result without improvement. The deadline prevents the retry handler from continuing after the allotted execution window.

A deadline check between calls cannot interrupt a model request that is already running. The model client must enforce its own request timeout.

Define retry-specific exceptions. Create retry_handler.py:

import json
import time
from collections.abc import Callable
from dataclasses import dataclass
from pydantic import ValidationError
from models import InvoiceExtraction

class StructuredOutputError(Exception):
    pass

class RetryBudgetExceeded( StructuredOutputError ):
    pass

class RepeatedInvalidOutput( StructuredOutputError ):
    pass

class RetryDeadlineExceeded( StructuredOutputError ):
    pass

@dataclass(frozen=True)
class RetryPolicy:
    max_attempts: int = 3
    max_identical_failures: int = 2
    timeout_seconds: float = 30.0
    def __post_init__(self):
        if self.max_attempts < 1:
            raise ValueError( "max_attempts must be positive" )
        if self.max_identical_failures < 1:
            raise ValueError( "max_identical_failures must be positive" )
        if self.timeout_seconds <= 0:
            raise ValueError( "timeout_seconds must be positive" )

The policy is immutable. This helps prevent retry limits from being modified accidentally during execution.

Implement Structured Output Validation

Next, create a function that parses and validates the model response. Add this to retry_handler.py:

def parse_structured_output( raw_response: str ) -> InvoiceExtraction:
    try:
        payload = json.loads(raw_response)
    except json.JSONDecodeError as exc:
        raise StructuredOutputError( f"Invalid JSON: {exc.msg}" ) from exc
    try:
        return InvoiceExtraction.model_validate( payload )
    except ValidationError as exc:
        errors = exc.errors( include_url=False, include_input=False )
        raise StructuredOutputError( f"Schema validation failed: {errors}" ) from exc

This function handles two specific failure categories. If JSON parsing fails, it raises a structured output error. If Pydantic validation fails, it raises a structured output error containing validation details.

Other unexpected exceptions are not converted into retryable errors. That distinction is important. A programming error should not trigger repeated model calls.

Implement the Bounded Retry Handler

Now we can build the retry loop. The handler accepts a callable that generates a model response. The callable receives optional validation feedback from the previous attempt. Add this to retry_handler.py:

def generate_with_retries( model_call: Callable[ [str | None], str ], policy: RetryPolicy ) -> InvoiceExtraction:
    started_at = time.monotonic()
    previous_invalid_output = None
    identical_failures = 0
    validation_feedback = None
    for attempt in range( 1, policy.max_attempts + 1 ):
        elapsed = ( time.monotonic() - started_at )
        if elapsed >= policy.timeout_seconds:
            raise RetryDeadlineExceeded( "Retry deadline exceeded" )
        raw_response = model_call( validation_feedback )
        try:
            return parse_structured_output( raw_response )
        except StructuredOutputError as exc:
            if raw_response == previous_invalid_output:
                identical_failures += 1
            else:
                identical_failures = 1
            previous_invalid_output = raw_response
            if ( identical_failures >= policy.max_identical_failures ):
                raise RepeatedInvalidOutput( "Model repeatedly returned " "the same invalid output" ) from exc
            validation_feedback = str(exc)
    raise RetryBudgetExceeded( "Maximum structured output " "attempts exceeded" )

This implementation guarantees that the handler initiates no more than the configured number of model calls. It also detects repeated identical invalid responses.

The retry loop terminates when:

  • The model produces valid structured output.
  • The attempt budget is exhausted.
  • The repeated-invalid-output limit is reached.
  • The deadline is exceeded before an attempt.

Why the loop is safer

There is no while True. The number of attempts is explicit. Only structured output validation errors are retried. Unexpected model-client exceptions propagate to the caller. The handler does not execute any external business action. It only produces validated data.

That separation reduces the risk of repeating side effects during output repair.

Provide Validation Feedback to the Model

Retries are more useful when the model knows what failed. Instead of repeating the original prompt unchanged, provide concise validation feedback. For example:

Your previous response failed validation.
Errors:
amount must be a positive number
approved must be a Boolean
currency must be USD, EUR, or GBP
Return only a valid JSON object matching the schema.

The retry handler already passes validation feedback to the model callable. A provider-specific adapter can use that feedback to construct the next request. Conceptually:

def model_call( validation_feedback: str | None ) -> str:
    messages = [
        {
            "role": "system",
            "content": (
                "Extract invoice information "
                "as valid JSON."
            )
        }
    ]
    if validation_feedback:
        messages.append({
            "role": "user",
            "content": (
                "The previous output failed "
                "validation:\n" + validation_feedback
            )
        })
    return call_llm_provider(messages)

This is integration pseudocode. The call_llm_provider function must be implemented using the selected model provider's SDK.

Avoid feeding unlimited error history back to the model. Some retry implementations append every failed response and validation error to the conversation. Over time, the prompt becomes larger and more confusing.

A better approach is to keep only the information needed for the next repair attempt. For example:

  • The original task.
  • The required schema.
  • The latest validation errors.
  • Relevant source data.

This keeps the repair process focused. Validation feedback should also be sanitized and bounded before being included in a prompt.

Test Successful Recovery After an Invalid Response

We can test the retry handler without calling a real LLM. Create test_retry_handler.py:

from retry_handler import (
    RetryPolicy,
    generate_with_retries,
    RetryBudgetExceeded,
    RepeatedInvalidOutput
)

def test_recovers_after_invalid_json():
    responses = iter([
        '{"invoice_id": "INV-1",',
        (
            '{"invoice_id": "INV-1", '
            '"amount": 100.0, '
            '"currency": "USD", '
            '"approved": false}'
        )
    ])
    feedback_received = []
    def fake_model(feedback):
        feedback_received.append( feedback )
        return next(responses)
    result = generate_with_retries(
        fake_model,
        RetryPolicy( max_attempts=3, max_identical_failures=2 )
    )
    assert result.invoice_id == "INV-1"
    assert result.amount == 100.0
    assert result.approved is False
    assert feedback_received[0] is None
    assert feedback_received[1] is not None

The first response contains invalid JSON. The second response is valid. The test verifies that the retry handler recovers successfully and provides feedback after the first failure.

Test Retry Budget Exhaustion

A reliable retry mechanism must stop when the model continues producing invalid output. Add this test:

def test_retry_budget_exceeded():
    responses = iter([
        '{"amount": "invalid"}',
        '{"currency": "unknown"}',
        '{"approved": "maybe"}'
    ])
    calls = 0
    def fake_model(feedback):
        nonlocal calls
        calls += 1
        return next(responses)
    try:
        generate_with_retries(
            fake_model,
            RetryPolicy( max_attempts=3, max_identical_failures=3 )
        )
    except RetryBudgetExceeded:
        pass
    else:
        raise AssertionError( "Expected retry budget failure" )
    assert calls == 3

The model produces three different invalid responses. The handler stops after the configured maximum of three attempts. This prevents an infinite loop even when the model never returns valid data.

Detect Repeated Invalid Responses

Some models repeatedly produce the same malformed structure. For example:

{ "invoice_id": "INV-1042", "amount": "unknown" }

If the model returns this exact response multiple times, another retry may not be useful. Add the following test:

def test_repeated_invalid_output():
    calls = 0
    def fake_model(feedback):
        nonlocal calls
        calls += 1
        re
Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.