Vector Databases, Deep Indexing & Token Economics: The Complete Story (phase 3)

Part 1: Where Embeddings Actually Get Saved

Before we talk about indexing, let's settle the basics. A vector by itself is useless. What you actually store is a row - the vector, plus everything needed to use it later, find it again, and avoid paying for it twice.

Table: document_chunks

Nephew: Why is metadata a separate jsonb column instead of just... more columns?

Uncle: Because metadata changes shape depending on the document. An HR policy chunk might have { department: "HR", doc_type: "policy" }. A support ticket chunk might have { customer_id: 4521, priority: "high" }. If you tried to make a rigid column for every possible field across every document type, you'd redesign your table every week. jsonb gives you that flexibility - and, this is the part people miss - you can still index inside a jsonb column, which we'll do properly in Part 7.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_chunks (
    id BIGSERIAL PRIMARY KEY,
    chunk_text TEXT NOT NULL,
    embedding VECTOR(1536) NOT NULL,
    embedding_model VARCHAR(50) NOT NULL,
    content_hash VARCHAR(64) NOT NULL,
    metadata JSONB DEFAULT '{}',
    created_at TIMESTAMP DEFAULT now()
);

Nephew: content_hash again - that's the same SHA-256 idea from Phase 1's file dedup, right?

Uncle: Same idea, one level deeper. In Phase 1 we hashed the whole file to avoid storing the same file twice. Here we hash each individual chunk, to avoid embedding the same chunk twice - because two completely different files can contain the exact same paragraph: a shared boilerplate legal clause, a repeated disclaimer, a standard company intro paragraph copy-pasted into fifty documents. Hold this thought. It becomes real money in Part 10.

Part 2: Why You Can't Just "Check Everything"

Now, the search speed question. Suppose you have 5 million chunk vectors, each with 1536 dimensions. A user asks a question, you embed it into a query vector. What's the dumbest possible way to find the most similar chunks?

Nephew: Compare the query vector against every single one of the 5 million vectors, calculate similarity for each, sort, take the top 5.

Uncle: Exactly - that's called a flat index, or exhaustive search. Let's see what it actually costs, in real numbers, not vague hand-waving.

• 5,000,000 vectors × 1536 dimensions each
• For EACH query: Compare against 5,000,000 vectors
• Each comparison: 1536 multiplications + additions (cosine similarity)
• Total operations per query: 5,000,000 × 1536 ≈ 7.68 BILLION operations
• On a typical machine: ~20-30 seconds per query

Nephew: 30 seconds?! Nobody waits 30 seconds for a chatbot answer.

Uncle: Exactly why brute force only works at small scale - a few thousand vectors, maybe. Past that, you need a fundamentally different strategy: don't check everything, only check the likely candidates. This trade-off has a name: Approximate Nearest Neighbor search, or ANN.

Nephew: "Approximate"? So it might give the wrong answer?

Uncle: It might give you the 4th-closest match instead of the actual 1st-closest, occasionally. That trade is almost always worth it, because the alternative is checking all 5 million vectors exactly, on every single query, forever. You're trading a tiny bit of accuracy for a massive amount of speed.

Nephew: Is this like the B-tree index we used on a normal database column, back when we built that email lookup?

Uncle: Same spirit - avoid scanning everything - but a completely different mechanism, and this distinction trips people up constantly. A B-tree index works because values have a natural sort order: "find email = x" can binary-search through sorted order, because "a" comes before "b" comes before "c". But vectors can't be sorted meaningfully in 1536-dimensional space. There's no "less than" between two directions in space. So vector databases use an entirely different family of tricks.

Every real ANN technique, no matter how fancy the name sounds, does one of two things: groups similar vectors together so you only search a small group, or builds a shortcut map that lets you jump toward the right neighborhood instead of walking through everyone. Let's go through both, properly.

Part 3: IVF (Inverted File Index) - The Neighborhood Approach

Remember Google Maps - it doesn't search the whole world to find Bangalore, it narrows down: Country → State → City. IVF works the same way for vectors.

Step 1 - Training: group vectors into "neighborhoods" (clusters)

Before any search happens, IVF looks at all your vectors and groups them into K clusters, using an algorithm like k-means. Imagine 5 million vectors get grouped into 1,000 clusters:

Each cluster has a centroid - the "average" vector representing everything in that group.

Step 2 - Query time: only search the closest clusters

Query: "What is our notice period?"
↓
Compare query ONLY against the 1,000 centroids (fast - just 1,000 comparisons)
↓
Find the 5 closest centroids (say, clusters 3, 47, 112, 289, 501)
↓
Only search vectors INSIDE those 5 clusters (~25,000 vectors, not 5 million!)
↓
Return top-5 most similar from that much smaller set

// Conceptual illustration of IVF search (real implementations are C++/Rust,
// but this shows the logic clearly)
function ivfSearch(queryVector, centroids, clusters, nProbe = 5) {
  // Step 1: Find nProbe closest centroids (cheap - only 1000 comparisons)
  const centroidDistances = centroids.map((centroid, i) => ({
    clusterId: i,
    distance: cosineSimilarity(queryVector, centroid)
  }));
  const closestClusters = centroidDistances
    .sort((a, b) => b.distance - a.distance)
    .slice(0, nProbe);

  // Step 2: Only search vectors WITHIN those clusters
  let candidates = [];
  for (const cluster of closestClusters) {
    candidates.push(...clusters[cluster.clusterId]);
  }

  // Step 3: Full precise search, but only on the much smaller candidate set
  return candidates
    .map(v => ({ vector: v, score: cosineSimilarity(queryVector, v.embedding) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 5);
}

Nephew: So instead of 5 million comparisons, we do 1,000 (to find clusters) + ~25,000 (inside the clusters) ≈ 26,000. That's roughly 200x fewer operations!

Uncle: Exactly the math. And notice the tuning knob - nProbe, how many clusters you search:

This is the recall-vs-speed tradeoff, and it's the central tension in every vector index that exists.

One more practical detail: how many clusters (lists, in pgvector's terminology) should you even create in the first place? A common rule of thumb:

lists ≈ sqrt(number of rows)

For 5 million rows: sqrt(5,000,000) ≈ 2,236, so roughly 2,000-2,500 lists.

• Too few lists → clusters are huge, barely faster than brute force.
• Too many lists → clusters are tiny, and centroids get unreliable without enough data per cluster.

CREATE INDEX ON document_chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 2000);

Part 4: HNSW (Hierarchical Navigable Small World) - The Highway Approach

IVF groups things into neighborhoods. HNSW takes a completely different approach - it builds a multi-level shortcut map, like a highway system.

Nephew: Highway system?

Uncle: Think about how you'd actually drive from Bangalore to a small village 800 km away. You don't take village roads the whole way.

Layer 2 (Highways):     Bangalore ─────────────────→ Big City X
                                    │
Layer 1 (State roads):  Bangalore ──→ Town A ──→ Town B ──┤
                                    │                     │
Layer 0 (Local roads):  Bangalore ──→ ... ──→ ... ──→ Village

You take the highway to get close fast, then drop to smaller roads only for the final stretch. HNSW builds exactly this structure over your vectors - multiple layers, where the top layer has very few "long-distance" connections, and each layer below has progressively more, finer-grained, local connections.

How HNSW search actually works:

Layer 2 (few nodes, long connections):
  Start at entry point → jump to nearest node in this sparse layer
  ↓ (drop down one layer, using that node as new starting point)
Layer 1 (more nodes, medium connections):
  From current position → jump to nearest node in this layer
  ↓ (drop down again)
Layer 0 (ALL nodes, short/local connections):
  From current position → carefully walk to the actual nearest neighbors

Total "hops" needed: roughly log(N) instead of N. For 5 million vectors: ~22 hops instead of 5 million comparisons!

// Conceptual illustration of HNSW search
function hnswSearch(queryVector, graph, entryPoint, topK = 5) {
  let currentNode = entryPoint;
  let currentLayer = graph.topLayer;

  // Phase 1: Greedy descent through upper layers (the "highway" phase)
  while (currentLayer > 0) {
    currentNode = greedySearchInLayer(queryVector, currentNode, graph, currentLayer);
    currentLayer--;
  }

  // Phase 2: Careful search in the bottom layer (the "local roads" phase)
  const candidates = beamSearchInLayer(queryVector, currentNode, graph, 0, topK * 4);
  return candidates
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

Nephew: So HNSW is like... a skip list, but for vectors in high-dimensional space?

Uncle: That's a genuinely excellent way to think about it. If you've seen skip lists in data structures, HNSW is that exact idea, generalized from a sorted 1-D list to a graph in 1536-dimensional space - sparse at the top for big jumps, dense at the bottom for precision.

IVF vs HNSW - When to Use Which

Nephew: So if HNSW is usually faster and more accurate, why would anyone use IVF?

Uncle: Two real reasons. First, memory - at massive scale (hundreds of millions of vectors), HNSW's graph connections take noticeably more RAM than IVF's simpler cluster-list structure. Second, update patterns - if your dataset is constantly getting new vectors (a live chat system logging every message, for instance), IVF handles that more gracefully than rebuilding parts of an HNSW graph.

For most RAG systems - a few hundred thousand to a few million chunks, updated periodically rather than constantly - HNSW is the common default today, and it's what most managed vector databases use out of the box.

Part 5: Product Quantization - Compressing Vectors When Even HNSW Gets Too Big

Nephew: Is there anything for when even HNSW's memory usage becomes a problem?

Uncle: Yes - Product Quantization, or PQ. This doesn't replace IVF or HNSW; it works alongside them, by compressing the vectors themselves.

Original vector: 1536 dimensions × 4 bytes each = 6,144 bytes per vector

Product Quantization:
  Split the 1536 dimensions into, say, 96 sub-vectors of 16 dimensions each
  For each sub-vector, find its closest match from a small pre-learned "codebook"
  Store just the CODE (an index into the codebook), not the raw numbers

Compressed vector: 96 codes × 1 byte each = 96 bytes per vector
Compression: 6,144 bytes → 96 bytes = 64x smaller!

The tradeoff, predictably, is a small loss in precision - you're now comparing compressed approximations, not exact vectors. But for 100 million+ vectors, that memory savings can be the difference between fitting in RAM and not fitting at all. Real large-scale systems often combine all three techniques: HNSW for navigation, PQ for compression, and careful metadata filtering to reduce the search space before the vector comparison even begins.