---
title: Storage & Op-Log
description: How OMGDB persists data — the human-readable append-only NDJSON operation log, the five rebuildable cache sidecars, and the single-file .omgdb transport archive.
---

OMGDB has one canonical source of truth on disk: an append-only operation log named `oplog.ndjson`. Every mutation is recorded as a framed, CRC-protected line of canonical JSON, and the entire logical state — documents, collection validation rules, and secondary indexes — can be rebuilt by replaying that log. Next to the log live **five derived cache sidecars** that give the CLI and MCP server direct read and write paths without a full replay — but they hold zero authoritative bits; the log *is* the database.

A live store is a **directory bundle**, not a single file. The directory contains `oplog.ndjson`, the cache sidecars, and (when used) change-audit sidecars and an advisory lock file. The single-file `.omgdb` form you may have seen is a transport/archive format produced by `omgdb pack` — it is not the live storage engine. This page documents the on-disk record format, the durability model, the derived cache sidecars, and the pack/unpack archive. For how operations are grouped atomically, see [transactions](/docs/transactions/); for how documents and `_id` are modeled, see the [data model](/docs/data-model/).

## The store directory

Each store directory holds exactly one log file plus derived, rebuildable sidecars:

```text
app.omgdb/
  oplog.ndjson            # the canonical, append-only operation log — the only source of truth
  primary-id.cache        # derived: (collection, _id) -> log offset, for point reads
  live-docs.cache         # derived: live-document checkpoint, for direct scans
  secondary-index.cache   # derived: secondary/compound/unique/partial index buckets
  metadata.cache          # derived: validation specs + index definitions
  open-checkpoint.cache   # derived: proof the sidecars match a specific log boundary
  LOCK                    # transient: the single-process advisory lock
```

Every `.cache` file is a cache in the strict sense: you can delete any of them, at any time, and the next reader rebuilds or tail-replays it with no change in query-visible behavior. Only `oplog.ndjson` is authoritative.

The log is plain newline-delimited text. You can read it with any tool — including `cat` — and every line is self-describing:

```sh
cat app.omgdb/oplog.ndjson
```

Because the log fully determines state, derived structures (secondary indexes, vector records, the cache sidecars) hold **zero authoritative bits**. This is invariant **I1 (text completeness)**: replaying `oplog.ndjson` reconstructs the entire logical state.

### Framed record format

Each physical line is framed as:

```text
<canonical-json>\t<crc32-hex>\n
```

That is, a single canonical-JSON object, a literal ASCII tab, the CRC-32 of the JSON bytes as 8 lowercase hexadecimal digits (zero-padded, `{crc:08x}`), then a newline. Canonical JSON escapes tabs inside strings, so the framing tab can never collide with record content.

An example line (an `insert` of a one-field document keyed by integer `1`):

```text
{"lsn":0,"ts":{"$date":0},"op":"insert","ns":"c","id":1,"doc":{"_id":1,"v":"row-1"}}	a1b2c3d4
```

> **Note:** The 8-hex-digit suffix after the tab is the CRC-32 of everything before the tab. The value shown above is illustrative; the real CRC is computed from the exact canonical-JSON bytes of that line.

### Record fields

Every record serializes a common header plus op-specific fields:

| Field | Type | Description |
| --- | --- | --- |
| `lsn` | integer (>= 0) | Log sequence number. Dense and contiguous — each record's `lsn` must equal its position in the file. |
| `ts` | datetime (ms since Unix epoch) | Wall-clock time of the append. Serialized as a `$date` value; also accepted as a plain integer on read. |
| `txn` | integer (>= 0), optional | The owning transaction id. Omitted for auto-committed (single) operations. See [transactions](/docs/transactions/). |
| `op` | string | The operation discriminant (see below). |
| `ns` | string | The collection (namespace). Present on data, `define`, and `create_index` ops. |
| `id` | any | The document `_id`. Present on `insert`, `replace`, and `delete`. |
| `doc` | object | The full document including its `_id`. Present on `insert` and `replace`. |
| `spec` | object | A collection validation spec. Present on `define`. |
| `field` / `fields` | string / array | The indexed field name(s). A single-field index writes `field`; a compound index writes a `fields` array. Present on `create_index`. |
| `unique` | bool, optional | Present (as `true`) on a `create_index` record for a unique index. |
| `partialFilter` | object, optional | The partial-index predicate. Present on a `create_index` record for a partial index. |

### Op kinds

The `op` string takes one of eight tokens:

| `op` | Fields | Meaning |
| --- | --- | --- |
| `insert` | `ns`, `id`, `doc` | Add a new document. |
| `replace` | `ns`, `id`, `doc` | Replace an existing document by `_id` (an edit). |
| `delete` | `ns`, `id` | Remove a document — written as a tombstone record, not by erasing earlier bytes. |
| `define` | `ns`, `spec` | Define or redefine a collection's validation rules. See [schema validation](/docs/schema-validation/). |
| `create_index` | `ns`, `field`/`fields`, `unique`?, `partialFilter`? | Create a secondary index (single-field, compound, unique, or partial). See [indexes](/docs/indexes/). |
| `begin` | — | Start of a multi-operation transaction. |
| `commit` | — | Commit a transaction; its buffered ops become visible on replay. |
| `abort` | — | Discard a transaction's ops on replay. |

> **Note:** An edit is recorded as an append, never an in-place rewrite. A `delete` appends a tombstone and a `replace` appends a new full document; the superseded records remain in the file until [compaction](#compaction) rewrites the log to its minimal form.

### Transaction markers on disk

The store itself never writes an `abort` record. It aborts a transaction by writing *nothing*: a dangling `begin` with no matching `commit` (for example, after a crash mid-transaction) is dropped entirely on replay. The explicit `abort` token is only produced by external log producers, but replay honours it for completeness. Atomicity, grouping, and isolation are covered in [transactions](/docs/transactions/).

## Durability model

The write path for every mutation is strictly ordered:

```text
append framed record to oplog.ndjson  ->  flush + fsync  ->  apply in memory
```

The fsync happens *before* the in-memory state changes, so a crash can never leave committed memory ahead of the durable log. A transaction appends `begin`, its ops, and `commit`, fsyncs once, then applies all of them in memory together.

Durable writes are therefore **fsync-bound** — the same physical wall every embedded database hits. The batch paths (`insert-many`, `import-jsonl`, transactions) amortize a single fsync across the whole batch, which is why bulk loads should always go through them.

### Per-record CRC-32

When the log is read, the CRC is recomputed over the JSON bytes and compared to the stored value. A complete (newline-terminated) line whose CRC fails to match is treated as corruption and the default open path stops (fail-stop):

```rust
// oplog.rs test crc_detects_corruption
let corrupted = fs::read_to_string(&path).unwrap().replacen('{', "[", 1);
fs::write(&path, corrupted).unwrap();
assert!(matches!(read_log(&path), Err(LogError::Corruption(_))));
```

LSNs are also integrity-checked: a record's `lsn` must equal its expected dense position, so a gap or a repeat (for example, two writers each starting at `lsn` 0) is reported as corruption.

### Torn-tail crash recovery

A record is durable only once its terminating newline reaches disk. On open, the bytes are split at the last newline: everything up to and including it is the durable region; any trailing fragment after it is an incomplete crash-time write. The fragment is **never silently discarded** — `Store::open` preserves its bytes to the `oplog.torn.bak` forensic sidecar (appended, with a `#`-prefixed separator line) and then trims the log to its durable prefix before opening the writer, so a later append can never splice onto torn bytes:

```text
crash mid-append  ->  open: preserve fragment to oplog.torn.bak  ->  trim log to durable prefix  ->  replay
```

This works even when the torn fragment is not valid UTF-8 — for example, a multi-byte character cut mid-encoding. Such a fragment in the *unterminated tail* is a recoverable torn tail. By contrast, invalid UTF-8 inside a *complete* (newline-terminated) record is genuine corruption and is fail-stop.

The writer also guards against splicing: if any write fails, it is poisoned and refuses all further appends and syncs, so the next record's bytes can never be concatenated onto torn bytes to form a complete-but-invalid line. A failed **fsync** additionally rolls back the unacknowledged bytes (a best-effort truncation to the last durably synced length), so a write that was reported as failed cannot resurrect on a later replay. Recovery from any of this is simply to reopen: open preserves and trims the torn tail and replays the durable prefix.

> **Limitation:** The strict open path is fail-stop on any complete corrupt record. Recovering the intact prefix of a damaged log is an opt-in operation, exposed via the `repair` workflow rather than silently performed on open.

### Two read paths: full replay and the direct paths

Full replay — folding the whole op-log from scratch — is the **write-capable in-process open path**: `Store::open` streams every record into an in-memory state, which is why the logical dataset must fit in RAM for the in-process engine. Reopening reproduces the exact same logical state — invariant **I2 (rebuild equivalence)**.

The CLI and MCP server avoid full replay: every command works through **direct paths** that read the log and the derived cache sidecars described below, loading only what the operation needs.

> **Limitation:** There is no paged or memory-mapped binary store yet, so a write-capable in-process open does not scale beyond available memory. The direct paths avoid full replay, and point reads decode only the entries they probe — but full scans still decode a whole sidecar per process.

### Integrity check

The integrity check (which backs the `verify` workflow) re-reads the on-disk log verifying every CRC, re-folds it, and asserts that the rebuilt data, catalog, and indexes reproduce the state. It also checks every cache sidecar — proving `checkpoint + tail replay == full replay` at runtime — and reports each checkpoint. It is the runtime proof of invariants I1 and I2, and it re-proves the entire database — every record, every derived cache — in about a second at 50,000 documents:

```sh
omgdb verify app.omgdb
```

## Derived cache sidecars

Five derived binary sidecars live next to `oplog.ndjson`. All five are rebuildable from the log and never authoritative — deleting any of them, at any time, does not change query-visible behavior; the next reader rebuilds or tail-replays it.

| File | Contents |
| --- | --- |
| `primary-id.cache` | `(collection, _id) -> {lsn, log byte offset}` for point reads and offset scans. |
| `live-docs.cache` | Live-document checkpoint for direct document reads. |
| `secondary-index.cache` | Secondary/compound/unique/partial index buckets for direct candidate reads. |
| `metadata.cache` | Collection validation specs and index definitions. |
| `open-checkpoint.cache` | Proof that the sidecars match a specific log boundary, enabling checkpointed opens. |

### Deferred persistence: checkpoints may lag the log

Each sidecar is a **checkpoint at some durable log boundary** plus canonical-log **tail replay** from that boundary. The contract is:

```text
checkpoint + tail replay == full replay
```

`verify` proves that contract at runtime. Persistence is threshold-based and deliberately lazy:

- A checkpoint may lag the durable log by up to **256 KiB** before any path persists a refresh. Mutations never rewrite sidecars per operation — that would reintroduce write amplification on the hot path.
- Readers replay the small canonical-log tail across the gap, and heal lazily: a read that needs a sidecar advances it.
- A clean in-process close persists the final primary snapshot; `verify`, compaction, and DDL (`create_index`/`define`) persist eagerly.
- A lagging checkpoint is always valid. Persistence is an optimization, never a correctness event.

Because the tail being replayed is the canonical log itself, none of this weakens the invariants: delete any sidecar — or all five — and the store answers identically.

> **Limitation:** Point reads decode only what they probe (v2 per-entry offset tables), but a full scan still decodes a whole cache file before streaming, and the secondary-index cache is still read whole-file; those remaining whole-file reads dominate direct scan latency today.

## Compaction

Over time the log accumulates superseded records (overwritten documents, tombstones, aborted transactions). Compaction rewrites the log to its minimal canonical form: one `define` per collection spec, one `create_index` per index (keys sorted), and one `insert` per surviving document in `_id` order.

```sh
omgdb compact app.omgdb
```

Compaction is crash-safe. The minimal log is written to a temporary `oplog.ndjson.compacting` file and fsynced, then read back with full CRC verification and a record-count check (rejecting any truncated tail). Only then is it atomically renamed over `oplog.ndjson`. The original file stays intact until the rename, so a failed rename is recoverable by reopening the original. An orphaned `.compacting` temp left by a crash is removed on the next open. The deterministic minimal form is invariant **I3 (export stability)**: a replay of a compacted log yields identical state.

Compaction preserves each data record's **original timestamp** — the rewritten `insert` records carry the `ts` the document was actually written with, so the log's history stays honest. Only the metadata records (collection specs and index definitions) carry the compaction time.

## The single-file `.omgdb` archive

A live store is a directory, but you often want to move, copy, or attach it as one file. `omgdb pack` bundles the store directory into a single `.omgdb` archive, and `omgdb unpack` restores it into a fresh directory.

```sh
# Bundle a live store directory into one file
omgdb pack app.omgdb app.omgdb.pack

# Restore it into a new, empty store directory
omgdb unpack app.omgdb.pack restored.omgdb
```

> **Note:** The `.omgdb` archive is the transport/archive form, not the live storage engine. The engine always runs against a store *directory*; you `unpack` an archive back into a directory before opening it.

### Archive format

The archive is a tiny, dependency-free, legible format you can also `cat`. It begins with a magic header line (`OMGDB-PACK v1`) followed by one or more entries:

```text
OMGDB-PACK v1
FILE <relative-path> <byte-len>\n<raw bytes>\n
```

`pack` bundles the canonical `oplog.ndjson` plus the change-audit sidecars (`pending/`, `changes/`) in deterministic, sorted order. The transient advisory `LOCK` file, the derived `.cache` sidecars, and any `.bak`/scratch files are intentionally left out — the caches are rebuilt on demand at the destination. Because the op-log fully determines state (I1), an unpacked store replays to exactly the packed one.

`unpack` refuses to write into a directory that already contains a store (it checks for an existing `oplog.ndjson`), so it never clobbers live data. It also rejects unsafe entry paths — absolute paths and any `..`/root components — so a crafted archive cannot escape the destination directory.

## Single-process exclusive lock

Opening a store acquires an exclusive advisory lock via a `LOCK` file in the store directory. While the first process holds the store open, a second open of the same directory is cleanly refused:

```rust
// store.rs test second_open_is_locked_out
let first = Store::open(&dir).unwrap();
assert!(matches!(Store::open(&dir), Err(StoreError::Locked { .. })));
drop(first);
let _reopened = Store::open(&dir).unwrap(); // available again after the first is dropped
```

The lock is held for the lifetime of the open store and released when it is dropped.

> **Limitation:** Concurrency is single-process and single-writer only — the advisory lock plus serialized mutation. There is no multi-reader/multi-writer model; use one process at a time. On POSIX filesystems the store directory is also fsynced after open and after a compaction rename so new directory entries survive a crash; directory fsync is a deliberate no-op on Windows.