Why LioranDB Uses LSM-Based Secondary Indexes

Secondary Indexes in Document Databases

A document database usually stores a primary record by document ID. But applications also ask questions such as:

• Find users where status = "active"
• Find invoices where customerId = 42
• Find posts where category = "rust"

Those queries need secondary indexes. LioranDB implements secondary indexing through an LSM-style index store.

The Mutation Path

When a document changes, the DBMS creates index mutations. These mutations are written to an index mutation log and then applied asynchronously.

Document write
     ↓
Index mutation log
     ↓
Pending mutation queue
     ↓
Index memtable
     ↓
Immutable table
     ↓
Segment file
     ↓
Compaction

This separates the primary document write path from heavier index maintenance.

Why LSM?

An LSM design converts many random updates into sequential batches. Instead of repeatedly rewriting one large index structure, the engine creates immutable sorted segments. Reads search across those segments and newer in-memory data. Background compaction later merges segments.

Different Workers for Different Workloads

LioranDB defines separate worker groups for:

• Secondary index application
• Text index application
• Index flushing
• Secondary compaction
• Text compaction

This matters because secondary and text indexes have very different costs. A secondary mutation may be a compact key-to-document mapping. A text mutation may generate many terms and postings. Treating them as identical would make scheduling less predictable.

Lag and Safety Limits

Asynchronous indexing introduces index lag. LioranDB tracks conditions such as:

• Pending mutations
• Mutation log bytes
• Lag in LSNs
• Lag in milliseconds
• Segment counts
• Compaction debt

Soft limits begin rescue work. Hard limits trigger stronger backpressure. This prevents the primary writer from outrunning the indexing pipeline forever.

The Trade-Off

LSM indexes are not free speed. They exchange immediate random writes for background complexity, read amplification, compaction, and lag management. Used carefully, that trade is extremely powerful. Used carelessly, it creates a segment confetti cannon.

LioranDB is built by Swaraj Puppalwar at Lioran Group. Links: LioranDB · Lioran Developer Solutions · Lioran Group

Top comments (0)