I Took the Wheel Off My RAG: Managed KB vs. Manual Chunking

In April I published a benchmark comparing 5 chunking strategies in Amazon Bedrock Knowledge Bases. The practical conclusion was simple: FIXED_SIZE with S3 Vectors as the backend, and that's it, unless your data justifies something more complex. Three months later, AWS took the question away from the problem entirely. Amazon Bedrock Managed Knowledge Base doesn't ask you to pick a chunking strategy. Smart Parsing decides parsing by document type, AWS manages the vector store, and if your questions are complex, AgenticRetrieveStream plans a retrieval strategy instead of running a single similarity search. Sounds like April's discussion is obsolete. So I asked the question that felt honest to me: how much does it cost in quality to have the wheel taken away from you, and under what conditions do you get that loss back? I didn't have a thesis before running this. I have one now, and it's not the one I expected. 🎯 Spoiler: Simple managed retrieval practically ties my manual April configuration. What didn't work is the agentic planner -the piece that justifies the "agentic" in the name- with the model AWS gives you by default: it didn't decompose a single query across 80 evaluated questions. This article covers the full methodology: five retrieval configurations over the same corpus, two question sets -the original 25 single-hop questions from April untouched, and 15 new multi-hop questions written against the actual document text-, three different planners, and the 6 infrastructure gotchas I had to resolve before I could measure anything. Because the planner finding has a twist: it depends entirely on which model does the planning, and I didn't know that until I stopped using the one that ships by default. 📌 TL;DR - Key data before you keep reading - Managed KB with simple Retrieve ≈ my manual April configuration in Correctness (0.88 vs. 0.84, a difference within the expected range).- Smart Parsing ingests without failing the two PDFs that broke SEMANTIC andNONE in April - and does it faster than my manual pipeline (183s vs. 407s). It even ingests and correctly retrieves a PDF with no extractable text layer.AgenticRetrieveStream with the default planner (MANAGED ) never generated a single sub-query across 80/80 evaluated questions, including ones a simple retrieval demonstrably couldn't solve.- On multi-hop questions, that made the "agentic" retriever land below simple retrieval (0.50 vs. 0.567 Correctness). - With a small CUSTOM planner (Claude Haiku 4.5), real decomposition does show up - but on questions that no longer needed it, not on the 5 that did.- With a large CUSTOM planner (Claude Sonnet 4.6, the same model as my generator), multi-hop Correctness jumped from 0.50 to 1.00, sustained across two independent runs - but the planner and generator share the same model in that test, and I'm stating that explicitly as a caveat, not as fine print.- The corpus isn't byte-identical to April's: my original bucket was destroyed and I had to re-download the documents. I'm saying that explicitly, not hiding it. Why This Matters to Me This isn't academic curiosity. If you're evaluating whether to migrate a production RAG to Managed Knowledge Base, the real question isn't "is it better?" - it's "what do I lose by giving up control of chunking, and what do I gain if I also let the service decide when and how to search more than once?" That second part is what almost nobody measures. AWS presents AgenticRetrieveStream with numbers from an academic benchmark (MuSiQue) showing gains of up to +37 recall points on 4-hop questions. Those are real numbers, published on their official agentic retrieval blog post, and I break them down further down. But an academic benchmark doesn't tell you what happens when the planner you get by default, without touching a single parameter, meets your own corpus. That's what I ran. The Design: Five Configurations, Two Question Sets I kept the generator (Claude Sonnet 4.6) and the judge (Nova Pro, cross-family) constant across four of the five configurations -the only exception is D, where the service itself generates- to isolate the retrieval layer as the sole variable: | Config | Retrieval | Generation | Question it answers | |---|---|---|---| | A | S3 Vectors + FIXED_SIZE (April) | Sonnet 4.6 | Baseline | | B | Managed KB, Retrieve | Sonnet 4.6 | Did I lose quality by handing over the wheel? | | C | Managed KB, AgenticRetrieveStream (generateResponse=False , planner MANAGED ) | Sonnet 4.6 | Does the default planner work? | | D | Managed KB, AgenticRetrieveStream (generateResponse=True , planner MANAGED ) | The service itself | What if I let AWS generate too? | | E | Managed KB, AgenticRetrieveStream (generateResponse=False , planner CUSTOM = Sonnet 4.6) | Sonnet 4.6 | Does a large planner help, not just a different one from the managed default? | The first four ran together. I added E a day later, as a direct response to what I found with C - I cover it in detail further down, in its own section, so as not to mix an exploratory run with the original pre-registered design. And two question sets over the same corpus of 3 technical documents (the Well-Architected Framework, the AgentCore developer guide, and a RAG evaluation blog post): - 1-hop set: the 25 original single-hop questions from April. Untouched, for comparability. - Multi-hop set: 15 new multi-hop and comparative questions, written against the actual document text, not against summaries. 🔍 ProTip #1: If you're going to compare an "agentic" retriever against a simple one, you need two question sets, not one. AWS itself reports that its gain on single-hop questions is under 5 recall points. Running only single-hop questions against a retriever that plans is measuring something you already know in advance won't show a difference - that's not a benchmark, it's a confirmation. A fairness control that cost me time but was non-negotiable: I re-ran configuration A in full, I didn't reuse the scores published in April. Those came from the native retrieveAndGenerate path; B, C, and D can only be evaluated through the bring your own inference responses (BYOI) path. Comparing scores produced by two different evaluation mechanisms would have been exactly the kind of methodological trap I criticize other benchmarks for. First Gotcha, Before Writing a Single Line of Retrieval Code My original plan was to reuse April's Knowledge Base as-is. I couldn't. April's KB and its corpus bucket no longer existed. That wasn't an oversight - I'd intentionally destroyed them after publishing the article, as I do with almost all my benchmark infrastructure. Verified against the real account before assuming anything: list-knowledge-bases returned zero results across five regions, the S3 Vectors bucket had no buckets, and the corpus bucket returned NoSuchBucket . I had to recreate just the FIXED_SIZE module from April's repo (Titan v2, 1024 dimensions, 512-token chunks, 20% overlap) - without the other 4 chunking modules, which weren't needed for this benchmark. And here's the uncomfortable part I do need to say: the corpus couldn't be byte-identical to April's. The data folder was in the .gitignore of the original repo - I never versioned the actual content, so there was no hash or manifest to compare against. Verified by HTTP HEAD: | Document | April (per README) | Today | Conclusion | |---|---|---|---| bedrock-agentcore-dg.pdf | ~17 MB | 30,420,374 bytes | Nearly double. Confirmed NOT identical. | wellarchitected-framework.pdf | ~14 MB | 14,189,927 bytes | Similar size, but without a hash I can't claim exact identity. | blog-rag-evaluation.html | - | Modified August 18 | Touched after April. | The AgentCore developer guide practically doubled in size over four months, which makes sense - it's the service that's evolved the most in that period. I'm reporting this as an explicit limitation of the re-run. I'm not hiding it, not minimizing it, and I also don't think it invalidates the comparison: the documents are still the same class of content (dense AWS technical documentation), which is what April's benchmark needed to be representative. ⚠️ ProTip #2: If you're going to publish a benchmark you plan to revisit months later, version the corpus with a public hash (even if you don't upload the full files). I didn't do that in April, and it cost me the ability to claim "byte-identical" with evidence, not just intent. Second Batch of Gotchas: Creating and Querying a Managed KB Isn't Like a Regular One Six real infrastructure problems, none of them documented together anywhere I could find. 1. Retrieve against a Managed KB rejects vectorSearchConfiguration . My April code used that parameter with no issue against the S3 Vectors KB. Against the Managed KB, the service responded: ValidationException: Incompatible configuration: vectorSearchConfiguration is not supported for managed knowledge bases. Use managedSearchConfiguration instead. Same internal shape, different container key. Config A uses one, config B needs the other. 2. Creating an S3 data source with type=S3 fails on a Managed KB. The error: ValidationException: Unsupported data source type for MANAGED knowledge base type. The correct form -which I only found against a real AWS example, not against the shape's reference documentation- is type=MANAGED_KNOWLEDGE_BASE_CONNECTOR , with the connector's configuration nested one level deeper than feels intuitive. 3. Creation is asynchronous in a way I didn't expect. With Terraform, config A accepts CreateKnowledgeBase and CreateDataSource back-to-back with no waiting. Against a Managed KB, calling CreateDataSource while the KB is still CREATING fails with ConflictException . You have to poll until AVAILABLE before continuing - typically 2-5 minutes. 4. ragSourceIdentifier in the eval job isn't a free-form label. I tried giving it a descriptive name ("D-setA" ) and the service rejected it: it has to match exactly the knowledgeBaseIdentifier carried by each line of t