Tracing RAG Failures Across Retrieval, Reranking, and Generation

A RAG application can return an incorrect answer even when its vector database, reranker, and language model are functioning normally. The problem is that each component optimizes a different objective.

Retrieval identifies potentially relevant documents, reranking prioritizes candidates, and generation produces an answer from the selected context. A failure at any stage can affect the final response.

The most reliable debugging approach is to trace each stage independently while preserving document identities, rankings, scores, model configurations, and the context actually passed to the LLM. With structured traces and stage-specific evaluations, engineers can distinguish missing evidence from incorrect ranking and generation errors.

This article explains how to implement that tracing, reproduce common failures, and build an observability workflow suitable for production RAG systems.

Why RAG Failures Are Difficult to Diagnose

Consider an enterprise assistant answering questions about internal infrastructure policies. A user asks: What is the approved process for rotating production database credentials?

The application retrieves ten documents, reranks them, selects the top four, and sends those passages to an LLM. The generated answer recommends rotating credentials manually through the database administration console. However, the organization's current policy requires using an automated secrets-management workflow.

The response is incorrect, but where did the failure occur? There are three possibilities.

• Retrieval failure: The correct policy never appeared among the retrieved candidates.
• Reranking failure: The correct policy was retrieved but ranked below less relevant documents and excluded from the final context.
• Generation failure: The correct policy reached the LLM, but the model ignored, misinterpreted, or contradicted it.

These failures require different solutions. Increasing the retrieval candidate count may help the first problem but cannot guarantee that the reranker selects the correct evidence. Changing the reranker may address the second problem but will not fix a generation model that disregards explicit instructions. Prompt engineering may help the third problem, but it cannot recover evidence that was never retrieved.

Without stage-level observability, engineers often change multiple components simultaneously and lose the ability to identify which modification improved or degraded the system.

Understanding the Three Stages of a RAG Pipeline

A common production RAG workflow looks like this:

User Question | v Query Processing | v Document Retrieval | |Top 20 candidates v Reranking | |Top 5 passages v Context Assembly | v LLM Generation | v Answer + Citations

Each stage has its own inputs, outputs, configuration, and potential failure modes.

Retrieval: Finding potentially relevant evidence

Retrieval searches a document collection using techniques such as dense vector similarity, keyword search, or hybrid retrieval. Its primary responsibility is to produce a candidate set containing information that could answer the question.

A retrieval system may fail because of:

• Poor document chunking.
• Incorrect query embeddings.
• Missing or outdated indexed content.
• Inappropriate metadata filters.
• Weak keyword matching.
• Insufficient candidate depth.

A document can be highly relevant to the user's question but absent from the candidate set. Once that happens, downstream reranking cannot recover it unless another retrieval operation is performed.

Reranking: Selecting the strongest candidates

Reranking evaluates retrieved candidates and attempts to order them by relevance to the query. For example, a cross-encoder can jointly process a query and candidate passage to estimate their relevance.

Reranking may fail when:

• Relevant passages receive unexpectedly low scores.
• Candidate text is truncated.
• The reranker receives a different query from the retriever.
• Important passages are excluded by the final top-k cutoff.
• Model behavior differs from the target domain.

Microsoft's RAG architecture guidance distinguishes broad candidate retrieval from reranking, which aims to improve the precision of the context selected for generation. Reference: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-information-retrieval

Generation: Producing an answer from evidence

The generation stage combines retrieved context with system instructions, the user's question, and potentially conversation history.

Failures include:

• Unsupported claims.
• Incorrect interpretation of evidence.
• Missing citations.
• Contradictions between the answer and context.
• Excessive reliance on model knowledge.
• Failure to acknowledge insufficient evidence.

An LLM can produce a confident answer even when the supplied context does not support it. This makes generation evaluation fundamentally different from retrieval evaluation.

Why Request-Level Logging Is Not Enough

A typical application log might contain:

{ "request_id": "req_1042", "status": "success", "latency_ms": 1840, "retrieved_documents": 20, "response_generated": true }

This tells us that the request completed. It does not explain whether the correct document was retrieved, how reranking changed its position, or what evidence reached the LLM.

For useful debugging, a trace must preserve the relationship between stages. A more informative trace structure is:

rag.request
| +-- query.process
| +-- retrieval.search
| candidate_count: 20
| index_version: index_v12
| +-- reranking.rank
| input_count: 20
| output_count: 5
| model: reranker_v2
| +-- context.assemble
| selected_chunks: 5
| +-- generation.invoke
model: configured_llm
output_tokens: 320

Each operation becomes a span within the same trace. A trace identifier connects the operations belonging to one request, while individual span identifiers represent the separate processing steps. This allows engineers to inspect how information moves through the complete pipeline.

What Should You Capture in Each RAG Trace?

A useful trace should capture enough information to reconstruct decisions without unnecessarily exposing sensitive enterprise data.

Retrieval span

Capture:

• Query identifier or protected query reference.
• Retrieval strategy.
• Index and embedding model versions.
• Requested and returned candidate counts.
• Candidate document and chunk identifiers.
• Candidate rankings and scores.
• Applied filters.
• Retrieval latency.

Reranking span

Capture:

• Reranker model and configuration.
• Input candidate identifiers.
• Original retrieval ranks.
• New reranking positions.
• Reranker scores.
• Selected candidate identifiers.
• Reranking latency.

Generation span

Capture:

• Generation model identifier.
• Prompt template version.
• Selected context identifiers.
• Input and output token counts when available.
• Generation parameters.
• Response identifier.
• Latency and error status.
• Evaluation results associated with the answer.

Avoid recording raw prompts, confidential document passages, or generated responses by default.

Implementing RAG Tracing With OpenTelemetry

OpenTelemetry provides APIs for collecting traces, metrics, and logs across distributed systems. It can be used to instrument RAG stages independently of the chosen vector database or LLM provider.

For a minimal Python demonstration, install the tracing packages:

pip install opentelemetry-api opentelemetry-sdk

The following example uses a console exporter so traces can be inspected locally without configuring an external observability backend.

from opentelemetry import trace
from opentelemetry . sdk . trace import TracerProvider
from opentelemetry . sdk . trace . export import ( BatchSpanProcessor , ConsoleSpanExporter , )
provider = TracerProvider ( )
provider . add_span_processor ( BatchSpanProcessor ( ConsoleSpanExporter ( ) ) )
trace . set_tracer_provider ( provider )
tracer = trace . get_tracer ( "rag.pipeline" )

This creates a tracer capable of recording nested operations. Next, define a small RAG pipeline with three stages.

def retrieve ( query ) :
 with tracer . start_as_current_span ( "rag.retrieval" ) as span :
  documents = [ { "id" : "doc_1" , "text" : "Credential rotation policy" } , { "id" : "doc_2" , "text" : "Database access guidelines" } ]
  span . set_attribute ( "rag.retrieval.candidate_count" , len ( documents ) )
  return documents
def rerank ( query , documents ) :
 with tracer . start_as_current_span ( "rag.reranking" ) as span :
  ranked = sorted ( documents , key = lambda item : item [ "id" ] )
  span . set_attribute ( "rag.reranking.input_count" , len ( documents ) )
  span . set_attribute ( "rag.reranking.output_count" , len ( ranked ) )
  return ranked
def generate ( query , documents ) :
 with tracer . start_as_current_span ( "rag.generation" ) as span :
  span . set_attribute ( "rag.generation.context_count" , len ( documents ) )
  return "Demonstration response"
def run_rag ( query ) :
 with tracer . start_as_current_span ( "rag.request" ) :
  documents = retrieve ( query )
  ranked = rerank ( query , documents )
  return generate ( query , ranked )
print ( run_rag ( "How are database credentials rotated?" ) )
provider . force_flush ( )
provider . shutdown ( )

This example demonstrates nested tracing rather than actual semantic retrieval, learned reranking, or LLM inference. The document list, sorting operation, and generated response are intentionally deterministic placeholders. It provides a starting point for instrumenting real components without depending on a particular AI framework.

Tracing Retrieval Failures

Retrieval debugging starts with a simple question: Did the correct evidence appear in the candidate set?

Suppose the expected document is security_policy_v3. The retrieval output contains:

• RankDocument 1 security_policy_v1
• 2 database_manual
• 3 access_guidelines
• 4 security_faq
• 5 credential_overview

The current policy is missing. This suggests that the problem occurred before reranking. Possible causes include stale indexing, weak embeddings, missing keywords, or overly restrictive metadata filters.

Measure Recall@k

Recall@k measures how many known relevant items appear among the first k retrieved results. For a query with one known relevant document, the calculation is straightforward. If the correct document appears within the first ten candidates, Recall@10 is 1 for that query. Otherwise, it is 0.

For multiple relevant documents:

def recall_at_k ( retrieved_ids , relevant_ids , k ) :
 if k <= 0 :
  raise ValueError ( "k must be positive" )
 relevant = set ( relevant_ids )
 if not relevant :
  raise ValueError ( "Relevant document set cannot be empty" )
 retrieved = set ( retrieved_ids [ : k ] )
 return len ( retrieved . intersection ( relevant ) ) / len ( relevant )

Example:

retrieved = [ "doc_7" , "doc_3" , "doc_9" , "doc_2" ]
relevant = { "doc_3" , "doc_2" }
score = recall_at_k ( retrieved , relevant , k = 3 )
assert score == 0.5

This example checks document-level recall. For chunk-based RAG, evaluation may need to operate at the chunk or evidence-passage level. A single document can contain both relevant and irrelevant chunks, so document-level recall may overstate the quality of evidence retrieval.

What to investigate when recall is low

Check whether the expected document is indexed and whether its latest version is active. Inspect query rewriting, embedding compatibility, chunk boundaries, search filters, and candidate depth. If relevant documents are consistently missing, modifying the generation prompt is unlikely to address the underlying problem.

Tracing Reranking Failures

Reranking failures are more subtle because the correct evidence may already exist in the retrieved candidates. Consider this example:

Initial Retrieval:

• Rank 1: database_manual
• Rank 2: security_policy_v3
• Rank 3: access_guidelines
• Rank 4: credential_faq
• Rank 5: deployment_notes

After reranking:

• Rank 1: database_manual
• Rank 2: deployment_notes
• Rank 3: credential_faq
• Rank 4: access_guidelines
• Rank 5: security_policy_v3

If the application sends only the top three passages to the LLM, the correct policy is excluded. Retrieval succeeded in finding the relevant document. Reranking failed to preserve it within the selected context.

Compare candidate rankings before and after reranking

Every reranking trace should retain candidate identities so engineers can compare positions across stages. A simple diagnostic function can identify which documents were dropped.

def dropped_candidates ( retrieved_ids , reranked_ids , top_k ) :
 selected = set ( reranked_ids [ : top_k ] )
 return [ document_id for document_id in retrieved_ids if document_id not in selected ]

Example:

retrieved = [ "database_manual" , "security_policy_v3" , "access_guidelines" , "credential_faq" ]
reranked = [ "database_manual" , "credential_faq" , "access_guidelines" , "security_policy_v3" ]
dropped = dropped_candidates ( retrieved , reranked , top_k = 3 )
assert "security_policy_v3" in dropped

This function does not determine whether the reranker is correct. It identifies candidates excluded by the cutoff, allowing an evaluation system to compare those candidates with known relevant evidence.

Common reranking problems

• Query mismatch: The reranker receives a rewritten query that no longer represents the user's original information need.
• Truncation: The reranker processes only part of a long document chunk, potentially losing the answer-bearing passage.
• Domain mismatch: A general-purpose reranker may not rank specialized technical or legal documents as expected.
• Aggressive filtering: A small final top-k value can remove useful evidence even when the initial candidate set has good recall.

Reranker scores should not be interpreted as calibrated probabilities unless the model documentation and evaluation support that interpretation.

Tracing Generation Failures

Generation debugging begins after the relevant evidence has successfully reached the model. Suppose the final context contains:

Production database credentials must be rotated through the approved secrets-management service. Howe