The LLM Is Not Your Ranker. Four Layers, and It Only Touches Two

We need to reformat the article body of article into markdown, with paragraphs, headings where topic shifts, lists for enumerations, code fences for code/commands/identifiers. Must preserve facts, not add commentary. Must not repeat title as heading. Must not wrap whole answer in code fence. Output only reformatted article body.

We need to identify sections: The article repeats some paragraphs (duplicate). We need to keep all facts; duplication is part of original text, we must preserve it (not remove). So we keep duplicated content as is.

We need to break into short paragraphs, add headings where topic shifts.

Let's parse content.

Original article (the given raw) starts after "Article:" line. It includes repeated content: first paragraph: "The pitch usually goes: send the catalog and the user's history to a model, get back a ranked list, delete the recommender. It demos beautifully on 200 items and falls over the moment the catalog has 400,000 and the response has 80 milliseconds to come back. The 2026 research doesn't support the replacement story either. RecoChain (arxiv 2604.25787) proposes unified generative retrieval and ranking, then evaluates it against classic top-k ranking metrics, because the measurement requirement doesn't go away. RRCM (arxiv 2605.07129) frames LLM recommendation as retrieval-and-reasoning, where the model decides when to pull more evidence and optimization still runs against the final ranking reward, not against the quality of the model's prose. Which puts the LLM inside two layers of a four-layer stack, not on top of it. The four layers Data and features. User events, item metadata, collaborative signals. Both papers above assume this layer is solid before anything generative gets added. It is also where roughly 80% of the project effort actually goes, whether you build or buy. Candidate retrieval. Cut millions of items to a few hundred. Hybrid by default: a collaborative path plus a metadata and embedding path, usually pgvector or a managed equivalent. Ranking. A deterministic model that scores and orders the shortlist. This is the layer with offline and online evaluation attached, and it's the one you're actually shipping. Feedback and experimentation. Clicks, conversions, saves, return visits, flowing back into layer 1. The LLM spans layers 2 and 3. It does semantic feature extraction, and it decides what evidence to retrieve. It does not emit the final ordering. Why the latency budget forces this Work backwards from a 100ms p99 for the whole call and the architecture stops being a matter of taste: request ├─ 5ms feature fetch(user vector, recent events, cached) ├─ 20msretrieval, parallel │ ├─ collaborative: ANN over user-item embeddings→ 200 candidates │ └─ metadata:pgvector + hard filters→ 200 candidates │union, dedupe → ~300 ├─ 35msranking: one batched scoring pass over ~300 rows ├─ 10msbusiness rules: diversity, in-stock, dedupe by brand └─ 5ms assemble + log the impression There is no room in that budget for a generative call in the hot path. So the LLM runs where it can be precomputed or cached: item embeddings generated at ingestion, query understanding cached per distinct query, an evidence-selection decision made asynchronously and reused. When it truly must run per-request, it runs against a shortlist of tens, not the catalog, and behind a timeout with a deterministic fallback. That fallback matters more than the feature. A recommendation strip that degrades to a popularity-ranked list under load is fine. One that returns nothing because a model call hung is a broken page. Log impressions, not just clicks The single most common data-layer mistake: recording what users clicked without recording what they were shown. impression: { request_id , user_id , item_ids [], positions [], ranker_version , retrieval_source [], ts } interaction: { request_id , item_id , type , ts } Without the impression row you cannot compute a click-through rate, cannot correct for position bias, and cannot run a counterfactual evaluation of a new ranker against logged traffic. You are left with online A/B tests as the only way to learn anything, which is slow and expensive. retrieval_source per item matters too: when quality drops you need to know whether the collaborative path or the embedding path produced the bad candidate. Offline metrics worth wiring from the start: recall@k for retrieval (did the item the user eventually chose make the shortlist at all?) and NDCG@k for ranking. These answer different questions and a lot of teams conflate them. A ranking model cannot fix a candidate set that never contained the right item. Recommendation versus matching A recommendation engine ranks items for one user. A matching engine pairs two sides of a marketplace: buyer and seller, designer and client, shift and worker. Same four layers. Two differences worth planning for. Matching has mutual constraints, so a candidate has to clear both sides' filters, which means the retrieval layer runs twice and intersects. And matching usually has supply exhaustion: recommending the same top-rated contractor to 400 buyers produces 399 disappointments, so the ranking layer needs a fairness or throttling term that pure item recommendation doesn't. Build order, from zero Event pipeline and profiles first. Nothing downstream works without it and it's the longest pole. Hybrid candidate generator. Collaborative plus metadata and embeddings. Ranking with real evaluation, offline and online, before any tuning. Controlled LLM usage last, for context construction and retrieval decisions only. Teams that invert this ship a demo in week two and spend month four discovering they have no way to tell whether it works. Build or buy The real axis is who holds the behavioral data. If those signals are the product's differentiation, renting layers 1 and 2 means renting the moat. If recommendations are a convenience feature on someone else's core product, SaaS is the correct answer and the architecture above still tells you what you're renting. Either way the 80% data-preparation share doesn't move. Buying changes who runs the layers, not whether they exist. Checklist Impressions logged with positions, ranker version, and retrieval source per item recall@k for retrieval and NDCG@k for ranking, measured separately Latency budget written down per stage, with p99 targets, before the first model choice Deterministic fallback on every model call, behind a timeout LLM work precomputed or cached at ingestion and query level, never per-item in the hot path Cold start path for new users and new items, since embeddings alone won't cover it Diversity and dedupe rules as an explicit post-ranking stage, not baked into the model Supply fairness term if it's a matching engine, not a recommender Holdout traffic reserved permanently, so you can always answer "versus nothing" Ranker version pinned in every log row, so an A/B result is attributable One note on claims The vendor marketing in this category is unusually bad. "3% to 45% conversion lift" with no baseline, no methodology, and no named company is not a benchmark. If you haven't measured a number on your own traffic, saying so is a trust signal rather than a weakness, and it's a reasonable thing to expect from a partner too. If you're scoping one The order matters more than the model choice: data pipeline, then retrieval, then a ranking layer you can evaluate, then generative components under control. That's the conversation worth having before anyone picks a vector database. brocoders.com"

Then after that, there is a duplicate of the same content again starting from "The pitch usually goes:" again, then ends with "brocoders.com Top comments (0)". So we need to preserve duplication.

Thus final markdown will contain the same content twice, separated perhaps by a blank line.

We need to add headings where topic shifts. Let's identify logical sections:

• Intro / pitch and research (first paragraph up to "Which puts the LLM inside two layers of a four-layer stack, not on top of it.")
• The four layers description (starting "The four layers Data and features." up to "Feedback and experimentation...")
• The LLM spans layers 2 and 3 (starting "The LLM spans layers 2 and 3." up to "It does not emit the final ordering.")
• Latency budget forces this (starting "Why the latency budget forces this" up to the diagram and following paragraphs)
• LLM runs where it can be precomputed or cached (starting "So the LLM runs where it can be precomputed or cached:" up to "That fallback matters more than the feature.")
• Log impressions, not just clicks (starting "Log impressions, not just clicks" up to end of that section)
• Recommendation versus matching (starting "Recommendation versus matching" up to end of that section)
• Build order, from zero (starting "Build order, from zero" up to end)
• Build or buy (starting "Build or buy" up to end)
• Checklist (starting "Checklist" up to end)
• One note on claims (starting "One note on claims" up to end)
• If you're scoping one (starting "If you're scoping one" up to end)
• brocoders.com (maybe as footer)

We need to preserve the duplicated content exactly; thus after first block we will repeat same headings and content again.

We need to use markdown headings ## for sections. Use bullet lists for enumerations. Use inline code or fenced code blocks for code, commands, identifiers. For example, the impression and interaction JSON-like structures should be code fences. Also the latency diagram with unicode characters could be kept as is, maybe as a code block to preserve formatting.

We must not add any extra info.

Let's craft.

We'll produce:

Pitch and research

... paragraph(s)

The four layers

... paragraphs

LLM spans layers 2 and 3

... paragraphs

Latency budget forces this

... include the diagram as a code block maybe.

LLM runs where it can be precomputed or cached

... paragraphs

Log impressions, not just clicks

... paragraphs

Recommendation versus matching

... paragraphs

Build order, from zero

... paragraphs (with bullet list)

Build or buy

... paragraphs

Checklist

... bullet list

One note on claims

... paragraphs

If you're scoping one

... paragraph

brocoders.com

maybe just line.

We need to ensure we preserve the exact text, including punctuation and spacing. We'll need to copy paragraphs exactly as they appear, but we can break them into shorter paragraphs for readability. Must not add or remove information. So we can split long sentences into separate paragraphs, but must not change wording. That's okay as long as we keep the same sentences. We'll need to ensure we don't inadvertently omit any sentences.

Let's extract the first block text exactly as given (up to before the second repetition). I'll copy the first block from the article:

"The pitch usually goes: send the catalog and the user's history to a model, get back a ranked list, delete the recommender. It demos beautifully on 200 items and falls over the moment the catalog has 400,000 and the response has 80 milliseconds to come back. The 2026 research doesn't support the replacement story either. RecoChain (arxiv 2604.25787) proposes unified generative retrieval and ranking, then evaluates it against classic top-k ranking metrics, because the measurement requirement doesn't go away. RRCM (arxiv 2605.07129) frames LLM recommendation as retrieval-and-reasoning, where the model decides when to pull more evidence and optimization still runs against the final ranking reward, not against the quality of the model's prose. Which puts the LLM inside two layers of a four-layer stack, not on top of it. The four layers Data and features. User events, item metadata, collaborative signals. Both papers above assume this layer is solid before anything generative gets added. It is also where roughly 80% of the project effort actually goes, whether you build or buy. Candidate retrieval. Cut millions of items to a few hundred. Hybrid by default: a collaborative path plus a metadata and embedding path, usually pgvector or a managed equivalent. Ranking. A deterministic model that scores and orders the shortlist. This is the layer with offline and online evaluation attached, and it's the one you're actually shipping. Feedback and experimentation. Clicks, conversions, saves, return visits, flowing back into layer 1. The LLM spans layers 2 and 3. It does semantic feature extraction, and it decides what evidence to retrieve. It does not emit the final ordering. Why the latency budget forces this Work backwards from a 100ms p99 for the whole call and the architecture stops being a matter of taste: request ├─ 5ms feature fetch(user vector, recent events, cached) ├─ 20msretrieval, parallel │ ├─ collaborative: ANN over user-item embeddings→ 200 candidates │ └─ metadata:pgvector + hard filters→ 200 candidates │union, dedupe → ~300 ├─ 35msranking: one batched scoring pass over ~300 rows ├─ 10msbusiness rules: diversity, in-stock, dedupe by brand └─ 5ms assemble + log the impression There is no room in that budget for a generative call in the hot path. So the LLM runs where it can be precomputed or cached: item embeddings generated at ingestion, query understanding cached per distinct query, an evidence-selection decision made asynchronously and reused. When it truly must run per-request, it runs against a shortlist of tens, not the catalog, and behind a timeout with a deterministic fallback. That fallback matters more than the feature. A recommendation strip that degrades to a popularity-ranked list under load is fine. One that returns nothing because a model call hung is a broken page. Log impressions, not just clicks The single most common data-layer mistake: recording what users clicked without recording what they were shown. impression: { request_id , user_id , item_ids [], positions [], ranker_version , retrieval_source [], ts } interaction: { request_id , item_id , type , ts } Without the impression row you cannot compute a click-through rate, cannot correct for position bias, and cannot run a counterfactual evaluation of a new ranker against logged traffic. You are left with online A/B tests as the only way to learn anything, which is slow and expensive. retrieval_source per item matters too: when quality drops you need to know whether the collaborative path or the embedding path produced the bad candidate. Offline metrics worth wiring from the start: recall@k for retrieval (did the item the user eventually chose make the shortlist at all?) and NDCG@k for ranking. These answer different questions and a lot of teams conflate them. A ranking model cannot fix a candidate set that never contained the right item. Recommendation versus matching A recommendation engine ranks items for one user. A matching engine pairs two sides of a marketplace: buyer and seller, designer and client, shift and worker. Same four layers. Two differences worth planning for. Matching has mutual constraints, so a candidate has to clear both sides' filters, which means the retrieval layer runs twice and intersects. And matching usually has supply exhaustion: recommending the same top-rated contractor to 400 buyers produces 399 disappointments, so the ranking layer needs a fairness or throttling term that pure item recommendation doesn't. Build order, from zero Event pipeline and profiles first. Nothing downstream works without it and it's the longest pole. Hybrid candidate generator. Collaborative plus metadata and embeddings. Ranking with real evaluation, offline and online, before any tuning. Controlled LLM usage last, for context construction and retrieval decisions only. Teams that invert this ship a demo in week two and spend month four discovering they have no way to tell whether it works. Build or buy The real axis is who holds the behavioral data. If those signals are the product's differentiation, renting layers 1 and 2 means renting the moat. If recommendations are a convenience feature on someone else's core product, SaaS is the correct answer and the architecture above still tells you what you're renting. Either way the 80% data-preparation share doesn't move. Buying changes who runs the layers, not whether they exist. Checklist Impressions logged with positions, ranker version, and retrieval source per item recall@k for retrieval and NDCG@k for ranking, measured separately Latency budget written down per stage, with p99 targets, before the first model choice Deterministic fallback on every model call, behind a timeout LLM work precomputed or cached at ingestion and query level, never per-item in the hot path Cold start path for new users and new items, since embeddings alone won't cover it Diversity and dedupe rules as an explicit post-ranking stage, not baked into the model Supply fairness term if it's a matching engine, not a recommender Holdout traffic reserved permanently, so you can always answer "versus nothing" Ranker version pinned in every log row, so an A/B result is attributable One note on claims The vendor marketing in this category is unusually bad. "3% to 45% conversion lift" with no baseline, no methodology, and no named company is not a benchmark. If you haven't measured a number on your own traffic, saying so is a trust signal rather than a weakness, and it's a reasonable thing to expect from a partner too. If you're scoping one The order matters more than the model choice: data pipeline, then retrieval, then a ranking layer you can evaluate, then generative components under control. That's the conversation worth having before anyone picks a vector database. brocoders.com"

Now second block is identical except ends with "brocoders.com Top comments (0)". Let's copy second block exactly:

"The pitch usually goes: send the catalog and the user's history to a model, get back a ranked list, delete the recommender. It demos beautifully on 200 items and falls over the moment the catalog has 400,000 and the response has 80 milliseconds to come back. The 2026 research doesn't support the replacement story either. RecoChain (arxiv 2604.25787) proposes unified generative retrieval and ranking, then evaluates it against classic top-k ranking metrics, because the measurement requirement doesn't go away. RRCM (arxiv 2605.07129) frames LLM recommendation as retrieval-and-reasoning, where the model decides when to pull more evidence and optimization still runs against the final ranking reward, not against the quality of the model's prose. Which puts the LLM inside two layers of a four-layer stack, not on top of it. The four layers Data and features. User events, item metadata, collaborative signals. Both papers above assume this layer is solid before anything generative gets added. It is also where roughly 80% of the project effort actually goes, whether you build or buy. Candidate retrieval. Cut millions of items to a few hundred. Hybrid by default: a collaborative path plus a metadata and embedding path, usually pgvector or a managed equivalent. Ranking. A deterministic model that scores and orders the shortlist. This is the layer with offline and online evaluation attached, and it's the one you're actually shipping. Feedback and experimentation. Clicks, conversions, saves, return visits, flowing back into layer 1. The LLM spans layers 2 and 3. It does semantic feature extraction, and it decides what evidence to retrieve. It does not emit the final ordering. Why the latency budget forces this Work backwards from a 100ms p99 for the whole call and the architecture stops being a matter of taste: request ├─ 5ms feature fetch(user vector, recent events, cached) ├─ 20msretrieval, parallel │ ├─ collaborative: ANN over user-item embeddings→ 200 candidates │ └─ metadata:pgvector + hard filters→ 200 candidates │union, dedupe → ~300 ├─ 35msranking: one batched scoring pass over ~300 rows ├─ 10msbusiness rules: diversity, in-stock, dedupe by brand └─ 5ms assemble + log the impression There is no room in that budget for a generative call in the hot path. So the LLM runs where it can be precomputed or cached: item embeddings generated at ingestion, query understanding cached per distinct query, an evidence-selection decision made asynchronously and reused. When it truly must run per-request, it runs against a shortlist of tens, not the catalog, and behind a timeout with a deterministic fallback. That fallback matters more than the feature. A recommendation strip that degrades to a popularity-ranked list under load is fine. One that returns nothing because a model call hung is a broken page. Log impressions, not just clicks The single most common data-layer mistake: recording what users clicked without recording what they were shown. impression: { request_id , user_id , item_ids [], positions [], ranker_version , retrieval_source [], ts } interaction: { request_id , item_id , type , ts } Without the impression row you cannot compute a click-through rate, cannot correct for position bias, and cannot run a counterfactual evaluation of a new ranker against logged traffic. You are left with online A/B tests as the only way to learn anything, which is slow and expensive. retrieval_source per item matters too: when quality drops you need to know whether the collaborative path or the embedding path produced the bad candidate. Offline metrics worth wiring from the start: recall@k for retrieval (did the item the user eventually chose make the shortlist at all?) and NDCG@k for ranking. These answer different questions and a lot of teams conflate them. A ranking model cannot fix a candidate set that never contained the right item. Recommendation versus matching A recommendation engine ranks items for one user. A matching engine pairs two sides of a marketplace: buyer and seller, designer and client, shift and worker. Same four layers. Two differences worth planning for. Matching has mutual constraints, so a candidate has to clear both sides' filters, which means the retrieval layer runs twice and intersects. And matching usually has supply exhaustion: recommending the same top-rated contractor to 400 buyers produces 399 disappointments, so the ranking layer needs a fairness or throttling term that pure item recommendation doesn't. Build order, from zero Event pipeline and profiles first. Nothing downstream works without it and it's the longest pole. Hybrid candidate generator. Collaborative plus metadata and embeddings. Ranking with real evaluation, offline and online, before any tuning. Controlled LLM usage last, for context construction and retrieval decisions only. Teams that invert this ship a demo in week two and spend month four discovering they have no way to tell whether it works. Build or buy The real axis is who holds the behavioral data. If those signals are the product's differentiation, renting layers 1 and 2 means renting the moat. If recommendations are a convenience feature on someone else's core product, SaaS is the correct answer and the architecture above still tells you what you're renting. Either way the 80% data-preparation share doesn't move. Buying changes who runs the layers, not whether they exist. Checklist Impressions logged with positions, ranker version, and retrieval source per item recall@k for retrieval and NDCG@k for ranking, measured separately Latency budget written down per stage, with p99 targets, before the first model choice Deterministic fallback on every model call, behind a timeout LLM work precomputed or cached at ingestion and query level, never per-item in the hot path Cold start path for new users and new items, since embeddings alone won't cover it Diversity and dedupe rules as an explicit post-ranking stage, not baked into the model Supply fairness term if it's a matching engine, not a recommender Holdout traffic reserved permanently, so you can always answer "versus nothing" Ranker version pinned in every log row, so an A/B result is attributable One note on claims The vendor marketing in this category is unusually bad. "3% to 45% conversion lift" with no baseline, no methodology, and no named company is not a benchmark. If you haven't measured a number on your own traffic, saying so is a trust signal rather than a weakness, and it's a reasonable thing to expect from a partner too. If you're scoping one The order matters more than the model choice: data pipeline, then retrieval, then a ranking layer you can evaluate, then generative components under control. That's the conversation worth having before anyone picks a vector database. brocoders.com Top comments (0)"

Now we need to format.

We'll produce markdown with headings.

We need to ensure we keep code fences for the impression and interaction structures, and maybe for the latency diagram.

Let's decide headings:

Pitch and research

(cover first sentences up to "Which puts the LLM inside two layers of a four-layer stack, not on top of it.")

The four layers

(cover from "The four layers Data and features." up to "Feedback and experimentation. Clicks, conversions, saves, return visits, flowing back into layer 1.")

LLM spans layers 2 and 3

(cover "The LLM spans layers 2 and 3. It does semantic feature extraction, and it decides what evidence to retrieve. It does not emit the final ordering.")

Latency budget forces this

(cover from "Why the latency budget forces this" through the diagram and following sentences up to "That fallback matters more than the feature." maybe include the sentence about recommendation strip etc.)

Actually after diagram there is: "There is no room in that budget for a generative call in the hot path. So the LLM runs where it can be precomputed or cached: item embeddings generated at ingestion, query understanding cached per distinct query, an evidence-selection decision made asynchronously and reused. When it truly must run per-request, it runs against a shortlist of tens, not the catalog, and behind a timeout with a deterministic fallback. That fallback matters more than the feature. A recommendation strip that degrades to a popularity-ranked list under load is fine. One that returns nothing because a model call hung is a broken page."

We could keep that as part of Latency budget forces this section, or split into next section "LLM runs where it can be precomputed or cached". Let's follow natural split: after diagram, there is sentence "There is no room in that budget for a generative call in the hot path." then explanation about precomputed etc. So we can have:

Latency budget forces this

• include diagram and maybe the sentence "There is no room in that budget for a generative call in the hot path."

LLM runs where it can be precomputed or cached

• rest of that paragraph.

But we need to keep the exact text; we can split paragraphs arbitrarily as long as we don't change wording. So we can put a blank line after "There is no room in that budget for a generative call in the hot path." and then start new paragraph with "So the LLM runs where it can be precomputed or cached:".

Thus we need to ensure we keep the exact sentences.

Let's extract the relevant sentences from the first block:

After diagram: "There is no room in that budget for a generative call in the hot path. So the LLM runs where it can be precomputed or cached: item embeddings generated at ingestion, query understanding cached per distinct query, an evidence-selection decision made asynchronously and reused. When it truly must run per-request, it runs against a shortlist of tens, not the catalog, and behind a timeout with a deterministic fallback. That fallback matters more than the feature. A recommendation strip