Architecture

OMGDB is a single Cargo workspace (resolver = "2", members = ["crates/*"]) split into nine focused crates. The shape of the codebase mirrors the central design idea: there is one storage spine that owns the durable, human-readable source of truth, and every other capability — querying, aggregation, introspection, vector search, agent-safe mutations — is a layer built on top of it that holds no authoritative state of its own.

This page describes the crate layout, how the crates depend on one another bottom-to-top, the “text is canonical, binary is a rebuildable cache” spine that the whole design rests on, and the engineering practices that keep the workspace honest. For the on-disk details see storage; for the transaction model see transactions.

The nine crates

The workspace is dual-licensed MIT OR Apache-2.0. Each crate has a single, narrow responsibility.

Note: There is no omgdb-index, omgdb-api, or omgdb-mcp crate. Indexes live inside omgdb-core, and the MCP server lives inside omgdb-cli (src/mcp.rs). Older design sketches that list those crates as separate members are stale.

Layering, bottom to top

The crates form a clean acyclic dependency graph. omgdb-core sits at the bottom and depends on no other workspace crate — it is the storage spine and defines the Value/Document model, the codec, and the Store. Everything else builds upward from there:

                         omgdb-cli
                    (binary + MCP server)
                            │  depends on all eight below
   ┌──────────┬──────────┬──┴────────┬──────────┬──────────┬──────────┐
omgdb-agg  omgdb-change  omgdb-     omgdb-     omgdb-     omgdb-md   omgdb-query
                         introspect vector     search        │           │
   │           │            │          │          │          │           │
   └───────────┴─── omgdb-query ───────┴──────────┘          │           │
                            │      (search also uses vector) │           │
                            └──── omgdb-core ────────────────┴───────────┘

The actual [dependencies] confirm the layers:

• omgdb-core — no internal dependencies. It is the foundation.
• omgdb-query and omgdb-md — depend on omgdb-core only.
• omgdb-agg, omgdb-change, omgdb-introspect, and omgdb-vector — each depend on omgdb-core + omgdb-query (they all need to read and match documents through the same total-order semantics).
• omgdb-search — depends on omgdb-core + omgdb-query + omgdb-vector: it filters through the shared query plan and reuses verified vectors for semantic mode.
• omgdb-cli — depends on all eight library crates and ties them together into the omgdb binary, and additionally hosts the MCP server.

Because querying, aggregation, change planning, introspection, Markdown parsing, and full-text/vector search all sit above core and never reach around it, they cannot smuggle authoritative state outside the op-log — even the search-index definition reaches disk only as an op-log operation. That structural constraint is what makes the storage invariants enforceable rather than aspirational.

The text-canonical spine

The defining idea of OMGDB is that the on-disk source of truth is an append-only, human-readable NDJSON operation log — <store>/oplog.ndjson, with a per-record CRC32 — that you can cat. Every index, vector record, and cache is a derived, rebuildable artifact that holds zero authoritative bits. This legibility is what lets the database explain (describe), verify (verify), and repair (repair) itself.

A live store is a directory bundle (<store>/oplog.ndjson). The single-file .omgdb form is produced by pack/unpack as a transport/archive format — it is not the live storage engine. The physical log line format is:

<canonical-json>\t<crc32-hex>\n

with supported log ops insert, replace, delete, begin, commit, abort, define, and create_index.

The derived cache sidecars

Next to the log sit five derived binary sidecars — primary-id.cache, live-docs.cache, secondary-index.cache, metadata.cache, and open-checkpoint.cache — that give the CLI and MCP server direct read and write paths without replaying the whole log. Each sidecar is a checkpoint at some durable log boundary plus canonical-log tail replay from that boundary; a checkpoint may lag the durable log by up to 256 KiB before any path persists a refresh, and readers replay the small tail across the gap. The contract — checkpoint + tail replay == full replay — is proven at runtime by verify. All five are rebuildable and deletable at any time; none is ever authoritative. See storage for the full policy.

Full log replay remains the write-capable in-process open path (Store::open rebuilds the entire state in RAM); the direct CLI/MCP paths avoid it by reading through the sidecars.

The three invariants

The whole architecture is held to three invariants. They are stated briefly here; see storage and transactions for the full treatment.

• I1 — Text completeness. oplog.ndjson fully determines the logical state; caches and indexes hold zero authoritative bits.
• I2 — Rebuild equivalence. Deleting and rebuilding derived (binary cache) state from the log yields identical query-visible behavior after rebuild — same results, same order under the engine’s defined total order. Note that byte-identical binary rebuild is an explicit non-goal; I2 is behavioral equivalence only.
• I3 — Export stability. Canonical export is deterministic: dump -> load -> dump is byte-identical for the canonical representation. This is the headline, property-tested claim.

Engineering practices

The workspace enforces a consistent set of practices across all crates.

Error handling

Library crates that define their own error types use thiserror (omgdb-core, -query, -agg, -change); omgdb-introspect and omgdb-vector reuse the lower-layer error types and pull in no extra error crate. Only the omgdb-cli binary pulls in anyhow. Library code paths avoid unwrap/expect — fallible operations return typed errors rather than panicking. This keeps the libraries embeddable and lets the binary be the single place that flattens errors for human-facing reporting.

Tests and quality gates

The three storage invariants (I1/I2/I3) are backed by property tests rather than examples alone. The full quality-gate set runs on every commit (and is what CI runs, in order):

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all
cargo doc --no-deps --all-features
cargo test --doc --all

Shared lints are applied to every crate via the workspace (unsafe_code, missing_docs, and rust_2018_idioms set to warn; clippy all = warn). The release profile uses lto = "thin" and codegen-units = 1.

Toolchain and CI

The toolchain is Rust stable, edition 2021, with an MSRV (rust-version) of 1.89 — required for File::try_lock, the advisory store lock that enforces single-process access. CI runs the latest stable toolchain on Linux, Windows, and macOS, plus a dedicated MSRV check.

Limitation: OMGDB is an early project (v0.0). The durability and codec layers are hardened — crash recovery, the repair tool, a bit-exact canonical codec proven by a property test, a crash-truncation matrix, and the cross-platform + MSRV CI — but it is not yet production-ready at scale. A write-capable in-process open replays the whole log into RAM, so the logical dataset must fit in memory; the direct CLI/MCP paths avoid full replay through the cache sidecars, and point reads decode only the entries they probe (per-entry offset tables) — but full scans still decode a whole sidecar per process. There is no paged/mmap binary store yet. A store is single-process (it takes an exclusive advisory lock on open), with no multi-reader/multi-writer concurrency model.

Where each capability lives

If you are looking for the code (or docs) behind a feature, the crate boundary is the map: