OMGDB’s performance story is deliberately honest. This page lists what is fast today, explains why the text-canonical design does not cost you durability throughput, documents the regression gate that keeps the hot paths from backsliding, and names the current frontier. All measurements come from the production benchmark harness plus end-to-end CLI timing on one development machine — your hardware will differ, so rerun them yourself with just bench.
What’s fast today
| Task | Result | How to reproduce |
|---|---|---|
| Bulk load | import-jsonl loads 50,000 documents in about 2 seconds (one fsync per batch, default batch size 1,000). | omgdb import-jsonl app.omgdb users data.jsonl |
| Durable single-document writes | Fsync-bound: sustained more than 2,500 durable inserts per second in-process, flat across store sizes. | just bench — the insert-with-fsync rows |
| Batched writes | One fsync per batch instead of per document: batches exceed 100,000 documents per second in-process. | just bench — the direct batch-insert rows |
| Full self-verification | verify re-proves the entire database — every record, every derived cache — in about a second at 50,000 documents. | omgdb verify app.omgdb |
| Indexed lookups | An indexed in-process find answers in well under a millisecond (0.22 ms measured). | just bench — the indexed-find rows |
The pattern behind the table: durability costs one fsync, so the fast paths are the ones that amortize it (batches, imports) or skip the disk entirely (index lookups against in-memory state). verify is fast because replaying a 50,000-record NDJSON log is a linear streaming pass.
Why a text log isn’t the bottleneck
It is tempting to assume a human-readable log must be slow. It is not, for two reasons.
Durable writes are fsync-bound — the same physical wall every embedded database hits. A durable write means the bytes must reach the disk before the call returns, and the flush dominates the cost no matter how the bytes are encoded. Serializing a canonical-JSON line is cheap next to that flush. SQLite in its durable journal modes pays the same wall; so does every engine that keeps its promises. Choosing a binary record format would not buy those fsyncs back.
Reads never parse the whole log. Five derived binary cache sidecars sit next to oplog.ndjson — a primary _id-to-offset map, a live-document checkpoint, secondary index buckets, metadata, and a checkpoint proof. The CLI and MCP server answer reads and writes through these caches directly, seeking the log only for the bytes they need. The text stays the single source of truth: every sidecar is rebuildable, deletable at any time, and verify proves at runtime that checkpoint-plus-tail-replay equals a full replay. See storage and architecture for the full model.
In other words: the text format is on the durability path, where fsync dominates; the binary formats are on the read path, where layout matters. Each format sits where its cost profile belongs.
The regression gate
The benchmark harness is part of the test discipline, not a marketing artifact. It measures the hot paths — insert-with-fsync, direct batch insert, single direct insert, direct point reads, full and warm store open, in-memory scan/indexed/sorted find, direct cache scans, index creation, and direct aggregation:
just bench
# or directly:
cargo bench -p omgdb-cli --bench productionWith OMGDB_BENCH_ASSERT=1 the same run becomes a coarse regression gate: every row is checked against a generous per-10,000-documents ceiling, so a reintroduced O(N) cost on a hot path fails loudly while ordinary machine-to-machine noise does not.
OMGDB_BENCH_ASSERT=1 just benchStore sizes are configurable, so scaling behavior is measured rather than assumed:
OMGDB_BENCH_SIZES=10000,100000,1000000 just benchCurrent frontier
Two costs dominate small-operation latency at larger stores today, and neither is the text log.
First, every CLI invocation is a fresh process, so each command pays process startup before any engine work. Second, each cache sidecar is currently one monolithic file: a fresh process decodes the whole cache before its first direct read, and that decode grows with the dataset. Together they set the latency floor for one-off commands against big stores — a coding agent looping single CLI calls feels this, while in-process library callers and batched workloads largely do not.
The roadmap answer is paged cache formats: page, shard, or memory-map the sidecars so a point read stops paying a dataset-sized decode, then move general query-visible state off RAM the same way. And one broader honesty note: at raw scan scale, mature engines are still faster today — decades of tuning are real. What OMGDB offers in exchange is a canonical store you can read, verify, and hand to an agent. Progress on the paged formats is tracked on the roadmap.
Related
- Storage & op-log — the append-only log, durability, and crash recovery.
- Architecture — the derived cache sidecars and the direct read/write paths.
- Indexes — what the planner accelerates and how to check with
explain. - OMGDB vs the alternatives — the qualitative comparison.