Inside LioranDB's Full-Text Search Segments

A normal secondary index can answer: status = "active". It cannot efficiently answer: documents containing "distributed database". LioranDB therefore has a dedicated text-segment architecture.

Tokenization

Text is split on non-alphanumeric characters. Depending on index options, tokens can be normalized to lowercase and filtered through stopwords. "Building Distributed Databases" becomes ["building", "distributed", "databases"].

Segment contents

A LioranDB text segment can contain several files and structures:

• Term dictionary
• Posting lists
• Document map
• Document-length norms
• Optional term positions
• Bloom filter
• Segment metadata

A posting connects a term to the local documents containing it. "database" → [doc 2, doc 8, doc 19]. Positions can record where the term appears inside each document. That enables more advanced query behaviour and phrase-aware features.

Global and local document IDs

Each segment assigns compact local IDs to its documents. A separate document map translates them back to global document IDs. This keeps postings smaller while preserving the external identity of the record.

Bloom filters

Each segment also maintains a Bloom filter for terms. Before reading a segment's postings, the query path can test whether the term might exist there. A negative answer is definitive. A positive answer means the segment may contain the term and should be checked.

Query modes

The text query layer supports modes such as:

• AND
• OR

It also emits scored documents and metrics including:

• Query time
• Postings read
• Candidate documents
• Segments searched

Full-text search is essentially a specialized database living beside the document database. Its data structures, compaction behaviour, scoring, and caching needs are different enough that treating it as a plain secondary index would be a mistake.

Built by Swaraj Puppalwar under Lioran Group . Learn more: LioranDB Lioran Developer Solutions Lioran Group