Storage & Op-Log

OMGDB has one canonical source of truth on disk: an append-only operation log named oplog.ndjson. Every mutation is recorded as a framed, CRC-protected line of canonical JSON, and the entire logical state — documents, collection validation rules, and secondary indexes — can be rebuilt by replaying that log. Next to the log live five derived cache sidecars that give the CLI and MCP server direct read and write paths without a full replay — but they hold zero authoritative bits; the log is the database.

A live store is a directory bundle, not a single file. The directory contains oplog.ndjson, the cache sidecars, and (when used) change-audit sidecars and an advisory lock file. The single-file .omgdb form you may have seen is a transport/archive format produced by omgdb pack — it is not the live storage engine. This page documents the on-disk record format, the durability model, the derived cache sidecars, and the pack/unpack archive. For how operations are grouped atomically, see transactions; for how documents and _id are modeled, see the data model.

The store directory

Each store directory holds exactly one log file plus derived, rebuildable sidecars:

app.omgdb/
  oplog.ndjson            # the canonical, append-only operation log — the only source of truth
  primary-id.cache        # derived: (collection, _id) -> log offset, for point reads
  live-docs.cache         # derived: live-document checkpoint, for direct scans
  secondary-index.cache   # derived: secondary/compound/unique/partial index buckets
  metadata.cache          # derived: validation specs + index definitions
  open-checkpoint.cache   # derived: proof the sidecars match a specific log boundary
  LOCK                    # transient: the single-process advisory lock

Every .cache file is a cache in the strict sense: you can delete any of them, at any time, and the next reader rebuilds or tail-replays it with no change in query-visible behavior. Only oplog.ndjson is authoritative.

The log is plain newline-delimited text. You can read it with any tool — including cat — and every line is self-describing:

cat app.omgdb/oplog.ndjson

Because the log fully determines state, derived structures (secondary indexes, vector records, the cache sidecars) hold zero authoritative bits. This is invariant I1 (text completeness): replaying oplog.ndjson reconstructs the entire logical state.

Framed record format

Each physical line is framed as:

<canonical-json>\t<crc32-hex>\n

That is, a single canonical-JSON object, a literal ASCII tab, the CRC-32 of the JSON bytes as 8 lowercase hexadecimal digits (zero-padded, {crc:08x}), then a newline. Canonical JSON escapes tabs inside strings, so the framing tab can never collide with record content.

An example line (an insert of a one-field document keyed by integer 1):

{"lsn":0,"ts":{"$date":0},"op":"insert","ns":"c","id":1,"doc":{"_id":1,"v":"row-1"}}	a1b2c3d4

Note: The 8-hex-digit suffix after the tab is the CRC-32 of everything before the tab. The value shown above is illustrative; the real CRC is computed from the exact canonical-JSON bytes of that line.

Record fields

Every record serializes a common header plus op-specific fields:

Op kinds

The op string takes one of eight tokens:

Note: An edit is recorded as an append, never an in-place rewrite. A delete appends a tombstone and a replace appends a new full document; the superseded records remain in the file until compaction rewrites the log to its minimal form.

Transaction markers on disk

The store itself never writes an abort record. It aborts a transaction by writing nothing: a dangling begin with no matching commit (for example, after a crash mid-transaction) is dropped entirely on replay. The explicit abort token is only produced by external log producers, but replay honours it for completeness. Atomicity, grouping, and isolation are covered in transactions.

Durability model

The write path for every mutation is strictly ordered:

append framed record to oplog.ndjson  ->  flush + fsync  ->  apply in memory

The fsync happens before the in-memory state changes, so a crash can never leave committed memory ahead of the durable log. A transaction appends begin, its ops, and commit, fsyncs once, then applies all of them in memory together.

Durable writes are therefore fsync-bound — the same physical wall every embedded database hits. The batch paths (insert-many, import-jsonl, transactions) amortize a single fsync across the whole batch, which is why bulk loads should always go through them.

Per-record CRC-32

When the log is read, the CRC is recomputed over the JSON bytes and compared to the stored value. A complete (newline-terminated) line whose CRC fails to match is treated as corruption and the default open path stops (fail-stop):

// oplog.rs test crc_detects_corruption
let corrupted = fs::read_to_string(&path).unwrap().replacen('{', "[", 1);
fs::write(&path, corrupted).unwrap();
assert!(matches!(read_log(&path), Err(LogError::Corruption(_))));

LSNs are also integrity-checked: a record’s lsn must equal its expected dense position, so a gap or a repeat (for example, two writers each starting at lsn 0) is reported as corruption.

Torn-tail crash recovery

A record is durable only once its terminating newline reaches disk. On open, the bytes are split at the last newline: everything up to and including it is the durable region; any trailing fragment after it is an incomplete crash-time write. The fragment is never silently discarded — Store::open preserves its bytes to the oplog.torn.bak forensic sidecar (appended, with a #-prefixed separator line) and then trims the log to its durable prefix before opening the writer, so a later append can never splice onto torn bytes:

crash mid-append  ->  open: preserve fragment to oplog.torn.bak  ->  trim log to durable prefix  ->  replay

This works even when the torn fragment is not valid UTF-8 — for example, a multi-byte character cut mid-encoding. Such a fragment in the unterminated tail is a recoverable torn tail. By contrast, invalid UTF-8 inside a complete (newline-terminated) record is genuine corruption and is fail-stop.

The writer also guards against splicing: if any write fails, it is poisoned and refuses all further appends and syncs, so the next record’s bytes can never be concatenated onto torn bytes to form a complete-but-invalid line. A failed fsync additionally rolls back the unacknowledged bytes (a best-effort truncation to the last durably synced length), so a write that was reported as failed cannot resurrect on a later replay. Recovery from any of this is simply to reopen: open preserves and trims the torn tail and replays the durable prefix.

Limitation: The strict open path is fail-stop on any complete corrupt record. Recovering the intact prefix of a damaged log is an opt-in operation, exposed via the repair workflow rather than silently performed on open.

Two read paths: full replay and the direct paths

Full replay — folding the whole op-log from scratch — is the write-capable in-process open path: Store::open streams every record into an in-memory state, which is why the logical dataset must fit in RAM for the in-process engine. Reopening reproduces the exact same logical state — invariant I2 (rebuild equivalence).

The CLI and MCP server avoid full replay: every command works through direct paths that read the log and the derived cache sidecars described below, loading only what the operation needs.

Limitation: There is no paged or memory-mapped binary store yet, so a write-capable in-process open does not scale beyond available memory. The direct paths avoid full replay, and point reads decode only the entries they probe — but full scans still decode a whole sidecar per process.

Integrity check

The integrity check (which backs the verify workflow) re-reads the on-disk log verifying every CRC, re-folds it, and asserts that the rebuilt data, catalog, and indexes reproduce the state. It also checks every cache sidecar — proving checkpoint + tail replay == full replay at runtime — and reports each checkpoint. It is the runtime proof of invariants I1 and I2, and it re-proves the entire database — every record, every derived cache — in about a second at 50,000 documents:

omgdb verify app.omgdb

Derived cache sidecars

Five derived binary sidecars live next to oplog.ndjson. All five are rebuildable from the log and never authoritative — deleting any of them, at any time, does not change query-visible behavior; the next reader rebuilds or tail-replays it.

Deferred persistence: checkpoints may lag the log

Each sidecar is a checkpoint at some durable log boundary plus canonical-log tail replay from that boundary. The contract is:

checkpoint + tail replay == full replay

verify proves that contract at runtime. Persistence is threshold-based and deliberately lazy:

• A checkpoint may lag the durable log by up to 256 KiB before any path persists a refresh. Mutations never rewrite sidecars per operation — that would reintroduce write amplification on the hot path.
• Readers replay the small canonical-log tail across the gap, and heal lazily: a read that needs a sidecar advances it.
• A clean in-process close persists the final primary snapshot; verify, compaction, and DDL (create_index/define) persist eagerly.
• A lagging checkpoint is always valid. Persistence is an optimization, never a correctness event.

Because the tail being replayed is the canonical log itself, none of this weakens the invariants: delete any sidecar — or all five — and the store answers identically.

Limitation: Point reads decode only what they probe (v2 per-entry offset tables), but a full scan still decodes a whole cache file before streaming, and the secondary-index cache is still read whole-file; those remaining whole-file reads dominate direct scan latency today.

Compaction

Over time the log accumulates superseded records (overwritten documents, tombstones, aborted transactions). Compaction rewrites the log to its minimal canonical form: one define per collection spec, one create_index per index (keys sorted), and one insert per surviving document in _id order.

omgdb compact app.omgdb

Compaction is crash-safe. The minimal log is written to a temporary oplog.ndjson.compacting file and fsynced, then read back with full CRC verification and a record-count check (rejecting any truncated tail). Only then is it atomically renamed over oplog.ndjson. The original file stays intact until the rename, so a failed rename is recoverable by reopening the original. An orphaned .compacting temp left by a crash is removed on the next open. The deterministic minimal form is invariant I3 (export stability): a replay of a compacted log yields identical state.

Compaction preserves each data record’s original timestamp — the rewritten insert records carry the ts the document was actually written with, so the log’s history stays honest. Only the metadata records (collection specs and index definitions) carry the compaction time.

The single-file .omgdb archive

A live store is a directory, but you often want to move, copy, or attach it as one file. omgdb pack bundles the store directory into a single .omgdb archive, and omgdb unpack restores it into a fresh directory.

# Bundle a live store directory into one file
omgdb pack app.omgdb app.omgdb.pack

# Restore it into a new, empty store directory
omgdb unpack app.omgdb.pack restored.omgdb

Note: The .omgdb archive is the transport/archive form, not the live storage engine. The engine always runs against a store directory; you unpack an archive back into a directory before opening it.

Archive format

The archive is a tiny, dependency-free, legible format you can also cat. It begins with a magic header line (OMGDB-PACK v1) followed by one or more entries:

OMGDB-PACK v1
FILE <relative-path> <byte-len>\n<raw bytes>\n

pack bundles the canonical oplog.ndjson plus the change-audit sidecars (pending/, changes/) in deterministic, sorted order. The transient advisory LOCK file, the derived .cache sidecars, and any .bak/scratch files are intentionally left out — the caches are rebuilt on demand at the destination. Because the op-log fully determines state (I1), an unpacked store replays to exactly the packed one.

unpack refuses to write into a directory that already contains a store (it checks for an existing oplog.ndjson), so it never clobbers live data. It also rejects unsafe entry paths — absolute paths and any ../root components — so a crafted archive cannot escape the destination directory.

Single-process exclusive lock

Opening a store acquires an exclusive advisory lock via a LOCK file in the store directory. While the first process holds the store open, a second open of the same directory is cleanly refused:

// store.rs test second_open_is_locked_out
let first = Store::open(&dir).unwrap();
assert!(matches!(Store::open(&dir), Err(StoreError::Locked { .. })));
drop(first);
let _reopened = Store::open(&dir).unwrap(); // available again after the first is dropped

The lock is held for the lifetime of the open store and released when it is dropped.

Limitation: Concurrency is single-process and single-writer only — the advisory lock plus serialized mutation. There is no multi-reader/multi-writer model; use one process at a time. On POSIX filesystems the store directory is also fsynced after open and after a compaction rename so new directory entries survive a crash; directory fsync is a deliberate no-op on Windows.