E-commerce Product Search vs Media KBs: 2026 Retrieval Architecture and Crawl Boundaries

Short answer: use an allowlisted, incremental crawl for e-commerce product search, and a collection-scoped crawl for a media knowledge base; the boundary should be owned by the index, not guessed by the language model.

That sounds obvious until a catalog has faceted URLs, tracking parameters, expired offers, and supplier feeds that disagree about the same SKU. A media team has a different mess: drafts, transcripts, rights notes, and a CMS that can expose the same story at five URLs. Retrieval architecture has to make those distinctions explicit because index cost at scale is mostly a boundary problem.

What should retrieval architecture include for e-commerce product search?

Start with a crawl contract. It should name the allowed hosts, path prefixes, query parameters, document types, freshness target, and deletion behavior. “Crawl the site” is not a contract; it is an incident waiting to happen.

For product search, I use a canonical product identity as the join key. Every fetched page is normalized into a record with product_id, locale, availability, price timestamp, category, and source URL. Variant URLs can remain useful for display, but they should not create independent semantic documents unless a shopper can buy the variant separately.

A color filter is usually a field. A separate regional product with different compliance text may deserve its own record.

The media knowledge-base case needs a different scope. A collection might be “published explainers in the US newsroom,” while drafts and legal notes stay in separate indexes with separate permissions. The same chunking code can serve both systems; the crawl boundary cannot.

The invariant is simple: every indexed chunk must have a stable source, an owner, an access decision, and an expiry or refresh rule. If one of those is missing, retrieval can return an answer that is fluent and wrong.

Two boundary strategies and their cost profile

There are two defensible defaults. A bounded crawl follows an explicit allowlist and a queue of discovered URLs. A collection crawl starts from a known set of records, such as a product feed or CMS API export, and only fetches linked content needed to enrich those records.

For a large shop, I normally combine them: the feed defines product membership, then a bounded crawl enriches only canonical product pages and a small set of policy documents. For a newsroom bot, the CMS export defines membership and a link resolver fetches referenced transcripts or PDFs. This hybrid is less glamorous than “crawl everything.” It is also easier to budget.

The rejected option is an unrestricted recursive crawl. It is valid for a one-off discovery audit or a short-lived migration inventory, where you cap depth, bytes, and total requests and throw the result away. It is a poor permanent retrieval architecture because faceted navigation and calendar-like URLs turn a finite catalog into an unbounded queue.

Scope first. The expensive failure is usually delayed. A crawler may appear healthy while it admits every sort order, filter combination, and session-marked URL; the embedding queue then spends hours processing near-identical pages, and the index quietly gives duplicate chunks more voting power. In a media corpus, the equivalent failure is a nightly export that keeps old drafts because “published” was treated as a one-time event.

I keep an admission log with the source revision, canonical ID, and rejection reason, then compare its counts with the upstream feed before writing embeddings. If the feed says 240,000 products and the run proposes 410,000 accepted IDs, the run stops for inspection. That threshold is a team policy, not a property of any crawler library, but making it explicit turns a vague cost spike into a testable invariant.

Crawl boundaries are an operational control, not a scraper setting

The fetcher needs three ledgers: seen URLs, accepted canonical IDs, and index versions. Keep them separate. A URL can be seen and rejected; a product can be accepted from a feed before its page is fetched; an index version can remain queryable while a new embedding model is being evaluated.

Here is the critical path in Python. It is deliberately boring: canonicalize first, enforce scope second, then schedule work with a bounded queue.

from dataclasses import dataclass
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit

TRACKING_KEYS = {"utm_source", "utm_medium", "utm_campaign", "gclid", "fbclid"}

@dataclass(frozen=True)
class CrawlPolicy:
    hosts: frozenset[str]
    prefixes: tuple[str, ...]
    max_bytes: int = 2_000_000

def canonical_url(raw: str) -> str:
    parts = urlsplit(raw)
    pairs = [(k, v) for k, v in parse_qsl(parts.query, keep_blank_values=True) if k not in TRACKING_KEYS]
    path = parts.path.rstrip("/") or "/"
    return urlunsplit((parts.scheme.lower(), parts.netloc.lower(), path, urlencode(sorted(pairs)), ""))

def in_scope(url: str, policy: CrawlPolicy) -> bool:
    parts = urlsplit(url)
    return (parts.netloc in policy.hosts and any(parts.path == p or parts.path.startswith(p.rstrip("/") + "/") for p in policy.prefixes))

def admit(raw: str, policy: CrawlPolicy, seen: set[str]) -> str | None:
    url = canonical_url(raw)
    if url in seen or not in_scope(url, policy):
        return None
    seen.add(url)
    return url

The important detail is what this code does not do. It does not infer that ?color=red is harmless, and it does not treat a 200 response as indexable. A parser still has to check content type, robots policy, publication state, language, and a deletion marker. The fetch log should retain the rejection reason, because a “missing” product is otherwise indistinguishable from a bug in the parser.

I once assumed URL normalization would solve most cost problems. It removed tracking noise, then the bill kept climbing because every size and material combination was still being embedded as a separate document. The fix was a product-level admission rule plus variant fields. Small change. Large effect.

How do you measure retrieval quality without hiding index cost?

Measure recall and spend together. A useful offline set has real queries, expected product or article IDs, and a reason code for misses. For e-commerce, include zero-result queries, out-of-stock items, and queries where a facet should filter rather than create a new semantic document. For the media bot, include a permission-denied document and two versions of the same story.

Track these counters per crawl run:

• fetched bytes
• accepted canonical IDs
• rejected URLs by reason
• chunks created
• embeddings written
• deleted IDs
• query-time miss rate

Add p50 and p95 freshness lag. A lower embedding count is not a win if stale price or rights metadata makes the result unusable.

Use a fixed replay set before changing chunk size or embedding model. Compare top-k recall, duplicate rate, citation coverage, and index delta. Keep the old index until the new one passes, then switch an alias or version pointer. This makes rollback a pointer update instead of a second emergency crawl.

The catch is that a bounded crawl is not suitable when the source has no stable identity or deletion signal. In that case, stick with a carefully capped recursive audit, or build a feed/export first. Paying to embed uncertainty does not make the uncertainty disappear. Measure the boundary.

Rejected option: one universal corpus

Putting product pages, newsroom drafts, customer-service macros, and legal documents into one vector space is convenient for a demo. It is a bad access boundary. Metadata filters can fail open when a new document type arrives without the expected field, and a semantically similar draft can outrank a published article.

Separate collections by permission and lifecycle, then merge results with an explicit policy. The retrieval layer should be able to say “no eligible document” rather than silently return the closest public chunk. That behavior matters more than a small gain in semantic similarity.

For teams building both systems, the practical decision rule is this: choose feed-led collection crawling when membership, identity, and deletion are authoritative; choose bounded URL crawling when page attributes are the source of truth; use a hybrid when the feed is authoritative for identity but pages contain the searchable detail. Record the boundary as code and as an architecture decision record so a future engineer can change it intentionally.

References

• https://arxiv.org/abs/2005.11401
• https://www.rfc-editor.org/rfc/rfc9110
• https://www.sitemaps.org/protocol.html
• https://schema.org/Product
• https://www.w3.org/TR/robots/