Vector Search

OMGDB ships a local semantic search layer that turns a string field into a vector and ranks documents by cosine similarity to a query — with no external service, network call, or API key. Text is embedded by a pluggable Embedder; the bundled HashingEmbedder is a deterministic, offline bag-of-words embedder. Ranking is a flat (exact) cosine kNN over the collection.

This makes semantic search reproducible and self-contained: the same input always yields the same vector, so results are stable across runs and machines. It is the foundation that context packs build on for token-budgeted, cited retrieval.

Scope: This page documents the compatibility vector surface — vsync/vstale/vsearch with the bundled HashingEmbedder, a deterministic baseline (the hashing trick + L2 normalization) that is explicitly not a neural model and works on every build. Real neural embeddings from pinned, checksum-verified local models live behind the unified search --mode semantic|hybrid surface on supported native builds. On every surface, ranking is flat/exact — there is no approximate-nearest-neighbor (ANN) index, HNSW, or vector quantization.

How it works

Text becomes a fixed-length f32 vector through three pieces:

• Embedder — a trait abstracting text-to-vector: dim() (vector length), embed(text) (the vector), and model_id() (a stable name+version+shape identifier).
• HashingEmbedder — the bundled implementation. It tokenizes, hashes each token into a bucket, and L2-normalizes.
• cosine(a, b) — cosine similarity in [-1, 1], returning 0.0 if the lengths differ or either vector is all-zero.

The HashingEmbedder

HashingEmbedder is deterministic and dependency-free. For each input it:

1. Splits text on any non-alphanumeric character and drops empty tokens.
2. Lowercases each token.
3. Hashes it with 64-bit FNV-1a, takes the result modulo dim to pick a bucket, and increments that bucket’s count.
4. L2-normalizes the resulting vector.

The default dimensionality is 256 (HashingEmbedder::default()); HashingEmbedder::new(dim) clamps dim to at least 1. The model_id is the string hashing-v1/dim={dim}, so two embedders with different dimensions are treated as different models for provenance purposes. All CLI commands construct HashingEmbedder::default() (dim 256).

Because it is a pure bag-of-words hash, the embedder captures lexical overlap, not deep semantics. A document that shares more tokens with the query ranks higher. The version suffix in model_id (v1) is bumped whenever the tokenization or hashing changes the produced vectors.

Searching

omgdb vsearch ranks documents in a collection by cosine similarity of a string field to a query and prints up to --k hits, best-first. Each hit is one line: the score formatted to four decimal places, a tab, then the matching document as canonical JSON.

Synopsis

omgdb vsearch <path> <collection> <field> <query> [--k N] [--filter JSON]

Example

omgdb create app.omgdb
omgdb insert app.omgdb docs '{"text":"embedded vector database search engine"}'
omgdb insert app.omgdb docs '{"text":"a recipe for chocolate cake"}'

# Rank docs in `docs` by similarity of their `text` field to the query (top 5).
omgdb vsearch app.omgdb docs text "database search" --k 5

Each output line is <score>\t<document>, best-first. The database-related document scores higher than the recipe because it shares more tokens with the query.

Note: Only string fields are embedded. Documents that lack a string value at the given field are silently skipped by search. A document missing _id reports its id as null.

Persisted vectors are reused

Search and context-pack scoring consult the persisted <collection>.__vectors records (see vsync below) before embedding anything. A stored vector whose provenance is fresh for the candidate document — same model, same dimensionality, same source field, and a content hash matching the document’s current text — is reused as-is; only stale or missing entries fall back to embedding the text on the fly. Results are identical either way (the provenance check guarantees the stored vector is exactly what re-embedding would produce); only the work changes. Running vsync first therefore makes repeated searches over a stable collection cheaper — the scan still visits every document, but unchanged documents skip the embedding step.

Hybrid search

Supplying --filter performs hybrid search: a structured MongoDB-style pre-filter is applied first, then the surviving documents are ranked semantically — both in a single scan pass. Only documents that match the filter are eligible to rank.

# Pre-filter to published docs, then rank those by relevance.
omgdb vsearch app.omgdb docs text "database search" --filter '{"status":"published"}'

If a draft document is equally relevant to a published one, the filter excludes it entirely. The filter uses the same syntax as the rest of OMGDB — see query operators.

Persisting embeddings: vsync

omgdb vsync embeds the string field of every document in a collection and persists each vector — together with a provenance envelope — into the sibling collection <collection>.__vectors. That target is an ordinary, op-log-backed, inspectable collection: it shows up in inspect, survives a store reopen, and passes the integrity check. Once synced, those vectors are the ones vsearch and context packs reuse.

Synopsis

omgdb vsync <path> <collection> <field>

vsync is incremental and idempotent. Documents whose persisted vector is still fresh (same model, dimensionality, and content hash) are skipped outright — not re-embedded, not re-written — so an immediate re-run appends nothing to the log. Stale entries are replaced by _id and new documents inserted, batched into one transaction per kind rather than one write cycle per record. Documents lacking a string field are skipped. The printed count is the number of vectors actually written:

omgdb vsync app.omgdb docs text
# stdout: synced 1 embedding(s) into `docs.__vectors` (fresh ones skipped)
omgdb vsync app.omgdb docs text
# stdout: synced 0 embedding(s) into `docs.__vectors` (fresh ones skipped)

Each persisted record has the shape {_id, provenance, vector}, where provenance is a document with keys model, dim, contentHash, and sourceField, and vector is an array of doubles.

The provenance envelope

Every persisted embedding stores enough to trace it to its producer and detect staleness:

This makes AI-derived state auditable rather than opaque: a stored vector can always be traced back to the model and the exact text it came from — and the same provenance check is what lets search trust a stored vector enough to reuse it.

Detecting stale embeddings: vstale

omgdb vstale reports the _ids in a collection whose persisted embedding is stale relative to its source. An embedding is stale when:

• No persisted vector exists for the document, or
• The embedder’s model_id differs from the recorded model (different model or configuration), or
• The dimensionality differs, or
• The source text has changed (the content hash no longer matches).

Synopsis

omgdb vstale <path> <collection> <field>

vstale prints the stale _ids (canonical JSON) to stdout and a summary count (N stale embedding(s)) to stderr.

Example

omgdb create app.omgdb
omgdb insert app.omgdb docs '{"text":"alpha beta"}'

omgdb vstale app.omgdb docs text   # stderr: "1 stale embedding(s)" (no persisted vector yet)
omgdb vsync  app.omgdb docs text   # stdout: "synced 1 embedding(s) into `docs.__vectors` (fresh ones skipped)"
omgdb vstale app.omgdb docs text   # stderr: "0 stale embedding(s)"

Editing a document’s source text marks only that document stale; re-running vsync re-embeds only it. Because the vectors live in an ordinary op-log-backed collection, they survive a reopen and remain consistent — and a stale vector is never silently served: search’s freshness check falls back to on-the-fly embedding for exactly those documents until the next vsync.

Library API

The omgdb-vector crate exposes these as plain Rust functions over a Store:

let e = HashingEmbedder::default();

// Flat cosine kNN: Vec<(Value /* _id */, f32 /* score */)>, best-first.
// Fresh persisted vectors from `docs.__vectors` are reused automatically.
let results = search(&store, "docs", "text", "database search", 5, &e);

// Hybrid search with a compiled pre-filter.
let filter = omgdb_query::Filter::compile(
    &Value::from_json_str(r#"{"status":"published"}"#).unwrap(),
).unwrap();
let hits = search_where(&store, "docs", "text", "database search", 5, &e, &filter);

MCP tools

The vector surface is also exposed over MCP for agents. The read tools are idempotent and gated at the read scope, so they are available even on a read-only MCP server; vsync writes to the store and requires the read-write scope.

See MCP for connecting an agent.

Limitations and caveats

Limitation: Search is flat/exact cosine kNN with a full collection scan — there is no ANN index. Cost is O(N) per query: every document is visited and scored, even though documents with fresh synced vectors skip the re-embedding step.

• The bundled HashingEmbedder is a lexical bag-of-words baseline, not a semantic neural model. For real neural embeddings, install a pinned local model (omgdb models pull) and use search --mode semantic — available on supported native builds.
• Only string fields are embedded and searched; documents lacking a string value at the field are silently skipped by sync_vectors, search, and context packs.
• Both content hashing (staleness detection) and the embedder’s bucketing use a 64-bit FNV-1a hash, chosen for reproducibility, not collision resistance. The contentHash is a 16-hex-digit string.
• An unreadable or absent __vectors collection never fails a search — the lookaside simply comes up empty and every document is embedded on the fly.

Related

• Full-text search — the unified typed search surface: BM25, semantic mode with pinned local models, and true RRF hybrid fusion.
• Context packs — token-budgeted, cited retrieval built on the same ranking.
• Query operators — the filter syntax used by hybrid search.
• MCP — running semantic search and context packs from an agent.