The Database That Tells You What It Knows

“Store the data” is only the beginning of the problem. The difficult questions usually come afterward: - What structure does this data actually have? - Which fields are missing or inconsistent? - Which values are invalid? - Which changes are safe to apply automatically? - What exactly changed after a repair? - Can the system prove that its storage and indexes are still consistent? I built Atlas to answer those questions inside the database engine itself. Atlas is a zero-dependency embedded database for semi-structured data. It stores records, builds a full-text search index, infers schema, analyzes data quality, proposes safe repairs, preserves uncertain records, and records an audit trail of applied changes. It does not use SQLite or SQL. It is not intended to replace SQLite for relational workloads. Instead, Atlas focuses on a gap that is usually handled by external scripts and tools: Data inspection, diagnosis, and safe repair as first-class database capabilities. That is the problem Atlas was built to solve. Why data quality belongs inside the database engine Most databases are very good at storing and retrieving data. That is necessary, but real-world data work rarely stops there.** Operational records, imported JSON, CSV files, event payloads, and semi-structured documents often arrive with problems: {"id": "T-1", "title": " Connection timeout ", "priority": "HIGH"} {"id": "T-1", "title": "connection timeout", "priority": "high"} {"id": "T-2", "title": "Unicode café search", "priority": null} These records contain several potential issues: - Duplicate logical identifiers - Leading or trailing whitespace - Inconsistent capitalization - Null-like values - Missing fields - Mixed data types - Malformed email addresses - Different date formats - Inconsistent structures across records A storage engine can preserve these values perfectly while still leaving the data difficult to understand and use. The usual response is to add external tools: - A schema profiler - A data-quality script - A search engine - A cleanup job - A validation pipeline - An audit system - A quarantine workflow This creates a fragmented system. The external tools may use different assumptions from the database. Repairs may not update the search index. Scripts may mutate records without a durable audit trail. Diagnosis may happen outside the storage boundary, where the system cannot guarantee that the result still matches the data. The fundamental problem I chose was this: Most embedded databases provide storage and query primitives, but inspection, diagnosis, and safe repair are usually handled by external tools. Atlas brings those capabilities into the database engine itself. The Atlas data-quality contract Atlas is built around a simple operational contract: Data should be stored durably, made searchable, inspected structurally, repaired explicitly, and verified afterward. That contract creates a complete workflow: INGEST | v STORE | v INDEX | v UNDERSTAND | v REPAIR | v VERIFY Each stage has a clear responsibility. Ingest Atlas accepts JSON arrays, JSON Lines, and CSV files. CSV values receive conservative type coercion, and JSONL and CSV inputs can be processed incrementally. Store Records are written to an append-only storage file with checksums and write-ahead journaling. Index Atlas maintains a persisted inverted index for full-text search and updates affected documents incrementally. Understand The engine infers field types, profiles field statistics, measures coverage and null values, and calculates an explainable health score. Repair Atlas detects and classifies issues. It does not modify records during diagnosis or dry-run repair. Verify The system checks storage blocks, journal state, index consistency, and audit-chain integrity. The goal is not simply to add more commands to a database CLI. The goal is to keep the entire data lifecycle connected. Why Atlas does not use SQLite SQLite is an excellent database. It is mature, reliable, portable, and extremely useful for embedded relational applications. SQLite is optimized for: - Tables - SQL queries - Transactions - Joins - Relational application state - A mature embedded database ecosystem Atlas targets a different optimization point. Atlas is designed for: - Semi-structured JSON-like records - Built-in schema inference - Full-text BM25 search - Explainable data-quality analysis - Safe repair proposals - Quarantine of uncertain records - Repair audit trails - Direct inspection of the storage lifecycle So the comparison is not that Atlas is universally better than SQLite. The more accurate claim is: Atlas is better suited when the question is not only “Can this record be stored?” but also “What is this data, what is wrong with it, which changes are safe, and how can those changes be proven?” Atlas does not use SQLite or SQL because the project is intentionally built around a different data model and workflow. SQLite remains an excellent choice for relational application data. Atlas is designed for a different class of problem: semi-structured data that must be stored, searched, understood, repaired, and verified in one place. The write path: durable storage with a visible trail Atlas uses append-only binary storage with a write-ahead journal. The write path is: Acquire lock | v Write transaction to journal | v Synchronize journal | v Apply committed blocks to data | v Synchronize data | v Truncate journal | v Release lock More precisely: - Atlas acquires the database lock. - It writes a transaction-begin marker. - It appends record or tombstone blocks. - It writes a transaction-commit marker. - It synchronizes the journal. - It replays committed entries into the main data file. - It synchronizes the data file. - It truncates the journal. - It releases the lock. The journal is not treated as temporary noise. It is the recovery boundary. If the process stops during a write, Atlas can distinguish committed transaction work from incomplete work. On the next open, committed transactions can be replayed while unfinished transactions are discarded. Each binary block contains a checksum: [MAGIC] [VERSION] [BLOCK TYPE] [SEQUENCE] [PAYLOAD LENGTH] [PAYLOAD] [SHA-256 CHECKSUM] The checksum covers the block header and payload. A damaged block is detected rather than silently accepted. This design also makes the storage system inspectable. The database is not just a black box that returns records. It has a visible trail of how those records reached disk. The database directory is part of the design An Atlas database directory contains separate files for separate responsibilities: mydb/ data.atf primary append-only record log journal.atf write-ahead journal index.atj persisted inverted index audit.atf chained repair audit trail quarantine.json uncertain records meta.json human-readable metadata lock platform-native lock file Each file answers a different question. - data.atf : What records are currently stored? - journal.atf : What committed work is recoverable? - index.atj : How are records searchable? - audit.atf : What repairs have been applied? - quarantine.json : Which records were preserved for review? - meta.json : What does the database know about itself? The directory is not just an implementation detail. It is part of Atlas’s inspectability model. Search that explains its ranking Atlas includes a full-text search engine based on a persisted inverted index and Okapi BM25 ranking. The index stores information such as: { "doc_lengths": {}, "doc_fields": {}, "postings": {} } The index tracks: - Term frequency - Document frequency - Document length - Token positions - Field-specific tokens - Incremental document updates Atlas supports: - Term searches - Phrase searches - Boolean expressions - Field filters - Numeric ranges - Date-like ranges - Explainable scoring For example, the query language can express: "connection timeout" (database OR storage) AND recovery priority:[2 TO 5] Search is not implemented as a full scan for ordinary term, phrase, and field queries. Atlas uses its persisted index to find candidate records, then applies BM25 scoring to rank them. The result is not just a list of matches. With explanation enabled, Atlas can show how search terms contributed to the ranking. That is important for a database whose goal is understanding. A search result should be relevant, but it should also be explainable. Understanding data without a predefined schema Semi-structured data does not always arrive with a reliable schema. Atlas infers field structure from the records it receives. It recognizes categories such as: - Null - Boolean - Integer - Number - String - Date-like value - Array - Object The profiler also reports: - Field coverage - Null counts - Distinct values - Common values - Numeric minimums and maximums - Averages - Nested field paths This allows Atlas to answer questions that a basic storage layer cannot answer: - Which fields are present in most records? - Which fields contain mixed types? - Which values are mostly null? - Which fields have unexpectedly high cardinality? - Which records contain nested structures? - Which fields might need a repair rule? Atlas then combines multiple signals into a health score covering: Storage integrity Schema consistency Data quality Search-index health The score is intentionally explainable. Atlas does not treat health as a mysterious number. It exposes the components behind the result. Safe repair is a database operation The most important difference between Atlas and a typical cleanup script is that repair is part of the engine’s controlled workflow. Atlas does not mutate records simply because it found something suspicious. The repair lifecycle is: DIAGNOSE | v DRY RUN | v APPLY SAFE CHANGES | v AUDIT AND VERIFY The repair engine classifies proposals by confidence and action. The important boundary is explicit mutation. Diagnosis does not modify records. Dry-run repair does not modify records. Only an explicit apply operation ch