The Unit of Return: a 4-Arm Retrieval Experiment on a Real Codebase

Where It Started

The problem traces back to a fundamental limitation of RAG search across unfamiliar codebases: models can locate code by keywords or symbols, but cannot answer questions like "where is the real business logic here, and what can I safely throw away?" - because the answer often doesn't exist in static code representation.

To address this, a bootstrap pipeline was developed for indexing new projects. The pipeline centers on four pillars:

• Entities - types and data-classes as the domain backbone
• Entry points - decorators (@mcp_app.tool) as the system boundary
• Tests - treated as ground truth, providing the only deterministic signal for live execution logic
• Git → ADR - decision history extracted from commit logs

Step 3: Name-Based Linking Failed

The initial approach attempted linking tests by name ("A test has a name, a function has a name, let's link by name"). This failed catastrophically: out of 109 candidate name-matches in the evaluation sample, zero links were found.

A custom sys.settrace plugin traced all 1,727 tests, revealing:

• 1,551 (89.8%) execute ≥1 source function
• 1,212 unique functions
• Mean execution time across tests: 10.1 (median 6, range 1-118)

This disproved the "1 test = 1 function" myth. Edges required no ranker - TESTS edges represent the raw traced call set under Exp-7 filters.

Evaluation Metrics

Aspect	Result
Traced tests	Up to 3 covering tests appended per function
Dynamic-trace gold set	70% union recall
Portability (foreign Python repos)	97.3% (gemma_agent, 2805/2882) to 100% (commit-, 27/27)
Coverage.py vs lightweight plugin	+19.96% vs +13.6% (same session, median of A/B runs)

Static signals served as companions rather than drivers. Portability held consistently across repositories. E17 integrated the TESTS signal as secondary context for the LLM, but did not improve search performance - MRR remained stable at 1.0.

The Question

A public benchmarking discussion produced a clean comparison table using identical retrievers, stores, queries, and rankings - varying only the returned unit:

Variation	Correct-or-Partial Count
Ten chunks	9/14
Whole document	5/14
No retrieval	6/14

The critical observation: the one count that survived was a chunk ranker's top hit landing on the answer document 0 times out of 14.

Claim: The unit of return is a measurement variable, not a retrieval setting. The protocol used four arms, froze everything beforehand, plus two controls (oracle, closed book) - and this is what was actually evaluated.

Experiment Design: F5 Pilot

The authors ran a controlled pilot (F5) with the following setup:

• n = 16 frozen queries
• 160 evaluations per arm (16 queries × 4 arms × 10 trials)
• Each trial involves two LLM calls: reader + judge

Query Set

• 16 frozen queries: 8 code + 8 prose
• SHA: e048aa12d36d95fb6d252de015d0ae807cbbce718e9c78effcd334f7c3ce83cb
• Overlap check passed (frozen_overlap_check.py v2 → OVERLAP: PASS)
• Index snapshot taken on 2026-09-26: 10,106 chunks / 716 files / 14,099 symbols

Retriever Configuration

Three ranking modes were compared:

• Arm A: Top-10 chunks
• Arm B: Top-1 document (full text)
• Arm C: Gold file chunked to the answer span
• Arm D: Empty context (prompt minus context block)

Readers used: opencode-go/longcat-2.0
Judges used: opencode-go/qwen3.7-plus

Both were pinned in the harness (Sept 2026) and treated as blind evaluators.

Controls and Validity

• One control run was declared invalid and rerun instead of being silently kept
• One symptom list was lost and recovered from the session database
• One 0/3 result was identified as an undercooked reasoning budget issue (rerun fixed, logged as a pitfall)

Results: Halves

Half 1 - The Unit Did Not Move

Metric	Value
A (top-10 chunks) hit@1	5/16 (0.312)
A hit@3 = hit@10	6/16
B (top-1 doc) top-1-is-gold	5/16 (0.312)
C (oracle)	16/16
D (closed book)	0/16

Code performed worse than prose: A hit@1 4/8, prose 1/8. The two-population asymmetry was visible before any reader was involved.

Prose showed even more disparity: A 2/8, B 1/8. While B had slightly higher prose success (15/80 vs 21/80 overall), the difference was direction-only - no effect claim could be made.

Half 2 - The Unit Moved

Metric	Value
A (top-k chunks)	26/160 (16.3%, CI 0.11-0.23)
B (whole document)	55/160 (34.4%, CI 0.27-0.42)
C (oracle)	156/160 (97.5%, CI 0.94-0.99)
D (closed book)	0/160 (0.0%, 95% upper bound 1.9% by rule of three)

Majority per query-arm (strict >50%):

• A: 2/16
• B: 5/16
• C: 16/16
• D: 0/16

Population split (n=80):

• Code: 5/80 (6.3%, CI 2.7-13.9)
• Prose: 40/80 (50.0%, CI 39.3-60.7)
• Oracle: 76/80 (95.0%)
• Closed book: 0/80

Notably, code achieved 8.0× on point estimates (5/80 → 40/80) with CIs [2.7-13.9] vs [39.3-60.7] - no overlap between the two arms.

NodeRAG: Structure vs Chunks

A second rig (NodeRAG) tested whether structure outperforms chunks when the entry point is exact. On 10 rule-queries:

• Arm A (chunked TF-IDF): 8/10 = 80.0% hit rate (301,981 tokens)
• Arm B (graph BFS): 7/10 = 70.0% hit rate (170,140 tokens)

Conclusion: No evidence that graph-based traversal beats chunked TF-IDF. The graph approach saved tokens (-43.6%) but missed all 3 positive controls and 3 NONE controls. The entry-point fragility dominated over traversal quality.

Related Work

Two prior approaches were briefly compared:

• Anthropic's Contextual Retrieval (2024) - reported −49% retrieval failures (−67% with rerank) on codebases via per-chunk LLM context prepended before embedding. Compatible but not identical to the current setup.
• Jina's Late Chunking (2024) - conditions chunk embeddings on full-document context; shows larger gains on longer documents in BEIR eval. Also does not split code vs prose, unlike the current pilot.

What Could Go Wrong

Several limitations were identified:

1. Pilot scale (n=16) - percentages reflect directions, not absolute effects, especially for prose where two queries dominate the outcome.
2. Index snapshotting, not freezing - drift between runs was documented but not eliminated.
3. Drift between runs - acknowledged as present but not fully mitigated.
4. Judged noise band not re-run - some variance remains unexplored.
5. Token cost claims absent - only context character counts were recorded; no cost attribution was made.
6. Unverified constants - graph_score = 0.4 was shipped as an operating point without ablation against BM25/reranker.
7. No cap experiment - a follow-up on the author's own logs was not run.

Acknowledgments

The authors thank everyone who engages with the community. Feedback, real-world observations, counter-examples, and benchmark numbers directly shaped the experiments. Open technical critique is essential for keeping engineering honest.

Disclaimer & Status

This is a draft (source material for the article). It is not a feature advertisement but an honest engineering story: figures are reproducible, weaknesses are named, and unaddressed risks are listed in the "What Could Go Wrong" section. No AI detectors were consulted in creating this disclosure.