Why our search engine returned T. Rowe Price for "supabase pricing"
DEV Community

Why our search engine returned T. Rowe Price for "supabase pricing"

We run our own web search engine, with our own crawler and index. Last week someone told us they were about to benchmark it by asking every search API the same simple questions, like "notion pricing" or "figma pricing", and checking whether the right page comes back first. So we tried it ourselves first.

"linear pricing" returned linear.app/pricing. "notion pricing" was fine. Then we typed "supabase pricing" and got this as the top result: troweprice.com, a page about investment insights. T. Rowe Price. A fund manager. Nothing to do with Supabase.

That one query led us to three separate bugs. None of them was exotic, and all three were the kind that hide behind a ranking pipeline that "mostly works". Here is what they were and how we fixed them.

How a query flows through our ranker

To understand the bugs you need the rough shape of the pipeline:

  • Tokenize and stem the query. "pricing" becomes "price", "supabase" becomes "supabas".
  • Collect candidates from the posting lists of each term.
  • If there are too many candidates, sample them down to a fixed budget.
  • Stage 1, a cheap pre-score: term presence weighted by IDF, then term frequency for the top few thousand. No documents are loaded.
  • Stage 2, full scoring for the best candidates: BM25, title and URL matches, phrase and proximity, domain signals.

Each stage hands the next one a smaller list. A page that falls out early never gets a chance later, no matter how good it is.

Bug 1: a small, perfect set of candidates got thrown away

For a two-word query, step 2 takes the intersection: pages that contain both words. The code had a fallback for when the intersection is tiny:

candidates = docs[a] & docs[b]
if len(candidates) < 20:
    candidates = docs[a] | docs[b] # widen with a union

That looks reasonable. It is not, when one word is rare and the other is very common. "figma" appears in relatively few pages. "price" appears in about a hundred times more. Pages with both words are few, and they are exactly the right ones (figma.com/pricing is among them).

But because there were fewer than 20, the code replaced that small, precise set with the union of both lists. Now almost every candidate was a page that only mentions "price". Step 3 then sampled that huge set down to the budget, and the handful of real matches were very likely to be dropped before anything scored them.

That is how a fund manager wins "supabase pricing": among thousands of pages that only say "price", a big, authoritative finance site scores well.

The fix: keep every full match, and only widen with the specific word.

both = docs[a] & docs[b]
if len(both) < 20:
    specific = [t for t in terms if df(t) <= 50_000] or terms
    rest = union(docs[t] for t in specific) - both
    rest = even_sample(rest, budget - len(both)) # never drop full matches
    candidates = both | rest

A page without the brand name is rarely what someone typing "figma pricing" wants, so widening with "price" alone was never useful. And the widened set stays under the sampling threshold, so the full matches survive into scoring.

Bug 2: we stemmed every query twice

After fixing Bug 1, our offline evaluation looked great, but the live site still returned T. Rowe Price. The evaluation script called the ranker directly. The website goes through one more layer, which normalizes the query for its cache key:

tokens = tokenize(query) # ["supabas", "price"]
normalized = " ".join(tokens) # "supabas price"
results = ranker.search(normalized) # the ranker tokenizes again

The ranker tokenizes and stems its input. So the already-stemmed text was stemmed a second time, and the Porter stemmer is not idempotent:

word stemmed once stemmed twice
supabase supabas supaba
pricing price price
figma figma figma

"supaba" is not a word in any document. With one of its two terms matching nothing, the query effectively became "price". Same symptom as Bug 1, completely different cause.

This was not specific to Supabase. Any word whose stem changes when stemmed again was broken on the website, while every offline test passed.

The fix was one line: pass the original query to the ranker and use the normalized form only as the cache key. Lesson: evaluate through the same code path your users hit. Our evaluation skipped one layer, and the bug lived in that layer.

Bug 3: homepages never reached the final round

With those two fixed, we ran our general evaluation set and found a different pattern: one-word navigational queries like "python" or "petronas" returned deep pages instead of the homepage.

Stage 1 only knows term presence and term frequency. For a one-word query, every candidate contains the word, so they all tie, and the tie was broken by document id. Homepages tend to be short (low term frequency) and tend to have been crawled early (low ids). So they lost the tie, again and again, and were never loaded for stage 2, where the domain match would have put them first.

When we scored python.org by hand, it beat the page we were returning by a factor of almost four. It just never got scored.

The fix has two parts:

  • For one- and two-word queries, scan the candidates' URLs (one batch read of the stored documents) and always send homepages whose host names the query to full scoring.
  • Give those homepages a fixed bonus in stage 2.

Our first version of "names the query" was too loose: any query word appearing in the hostname. That immediately broke "stripe pricing", where stripe.com's homepage jumped above its pricing page, and "windows 11 download", where a homepage on a download. subdomain jumped to the top.

The rule we kept: every query word must be a label of the hostname, or the words joined together must be one. "petronas" matches petronas.com; "stripe pricing" does not match stripe.com.

Bonus bug: the cache we thought we cleared

After deploying, we cleared the search cache with the usual one-liner:

redis-cli --scan --pattern "search_cache:*" | xargs redis-cli del

Our cache keys contain the normalized query and a JSON blob of filters, so they have spaces and quotes. xargs split every key into pieces, del was called with keys that didn't exist, and nothing was deleted. No error. The site kept serving the old answers for hours while we wondered why the fix "didn't work".

Deleting the keys from a few lines of Python with scan_iter fixed it.

The numbers

We now keep a small evaluation set for this exact case: 20 "X pricing" queries for well-known products, judged by whether the product's pricing page comes first.

before after
"X pricing": right page at #1 8 of 20 15 of 20
"X pricing": right page in top 10 11 of 20 17 of 20
General set (40 queries): right answer at #1 22 of 40 28 of 40
General set: right answer in top 10 32 of 40 38 of 40

Slowest query in the pricing set also went from about 1.0 s to about 0.5 s, because the candidate sets are smaller now.

What is still wrong

"stripe pricing" returns a localized Stripe pricing page (stripe.com/en-ro/pricing) instead of the main one. Queries with version numbers ("windows 11", "python 3.12") still lose information in tokenization. Our set of developer documentation queries ("pandas groupby", "kubernetes ingress") is still weak, mostly because those official docs pages need better coverage.

What we took away

One concrete query beats an abstract metric. "supabase pricing returns a fund manager" was more useful than any average score. Every stage that drops candidates is a place where good pages silently die. Log what each stage throws away. Test the path users take, including caches and wrappers. Fixes need regression checks. The loose homepage rule fixed five queries and broke two; only the evaluation set caught that.

If you run a search stack and have seen similar failure modes, we would like to hear about them.

Alex, NestDaddy

Originally published on the NestDaddy blog. NestDaddy gives your AI today's internet: live web search and real-time breaking news from its own index. nestdaddy.com/mcp

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.