---
title: Vector Search
description: Local, dependency-free semantic search over a text field — a deterministic offline embedder plus exact flat cosine kNN, hybrid filters, and persisted embeddings that searches reuse.
---

OMGDB ships a local semantic search layer that turns a string field into a vector and ranks documents by cosine similarity to a query — with no external service, network call, or API key. Text is embedded by a pluggable `Embedder`; the bundled `HashingEmbedder` is a deterministic, offline bag-of-words embedder. Ranking is a flat (exact) cosine kNN over the collection.

This makes semantic search reproducible and self-contained: the same input always yields the same vector, so results are stable across runs and machines. It is the foundation that [context packs](/docs/context-packs/) build on for token-budgeted, cited retrieval.

> **Scope:** This page documents the compatibility vector surface — `vsync`/`vstale`/`vsearch` with the bundled `HashingEmbedder`, a deterministic baseline (the hashing trick + L2 normalization) that is explicitly **not** a neural model and works on every build. Real neural embeddings from pinned, checksum-verified local models live behind the unified [`search --mode semantic|hybrid`](/docs/search/) surface on supported native builds. On every surface, ranking is flat/exact — there is no approximate-nearest-neighbor (ANN) index, HNSW, or vector quantization.

## How it works

Text becomes a fixed-length `f32` vector through three pieces:

- **`Embedder`** — a trait abstracting text-to-vector: `dim()` (vector length), `embed(text)` (the vector), and `model_id()` (a stable name+version+shape identifier).
- **`HashingEmbedder`** — the bundled implementation. It tokenizes, hashes each token into a bucket, and L2-normalizes.
- **`cosine(a, b)`** — cosine similarity in `[-1, 1]`, returning `0.0` if the lengths differ or either vector is all-zero.

### The HashingEmbedder

`HashingEmbedder` is deterministic and dependency-free. For each input it:

1. Splits text on any non-alphanumeric character and drops empty tokens.
2. Lowercases each token.
3. Hashes it with 64-bit FNV-1a, takes the result modulo `dim` to pick a bucket, and increments that bucket's count.
4. L2-normalizes the resulting vector.

The default dimensionality is **256** (`HashingEmbedder::default()`); `HashingEmbedder::new(dim)` clamps `dim` to at least 1. The `model_id` is the string `hashing-v1/dim={dim}`, so two embedders with different dimensions are treated as different models for provenance purposes. All CLI commands construct `HashingEmbedder::default()` (dim 256).

Because it is a pure bag-of-words hash, the embedder captures lexical overlap, not deep semantics. A document that shares more tokens with the query ranks higher. The version suffix in `model_id` (`v1`) is bumped whenever the tokenization or hashing changes the produced vectors.

## Searching

`omgdb vsearch` ranks documents in a collection by cosine similarity of a string field to a query and prints up to `--k` hits, best-first. Each hit is one line: the score formatted to four decimal places, a tab, then the matching document as canonical JSON.

### Synopsis

```text
omgdb vsearch <path> <collection> <field> <query> [--k N] [--filter JSON]
```

| Flag | Description | Default |
| --- | --- | --- |
| `--k` | Number of results to return. | `5` |
| `--filter` | MongoDB-style filter (JSON) to pre-filter candidates — enables hybrid search. | none |

### Example

```sh
omgdb create app.omgdb
omgdb insert app.omgdb docs '{"text":"embedded vector database search engine"}'
omgdb insert app.omgdb docs '{"text":"a recipe for chocolate cake"}'

# Rank docs in `docs` by similarity of their `text` field to the query (top 5).
omgdb vsearch app.omgdb docs text "database search" --k 5
```

Each output line is `<score>\t<document>`, best-first. The database-related document scores higher than the recipe because it shares more tokens with the query.

> **Note:** Only string fields are embedded. Documents that lack a string value at the given field are silently skipped by search. A document missing `_id` reports its id as `null`.

### Persisted vectors are reused

Search and context-pack scoring consult the persisted `<collection>.__vectors` records (see [`vsync`](#persisting-embeddings-vsync) below) before embedding anything. A stored vector whose provenance is **fresh** for the candidate document — same model, same dimensionality, same source field, and a content hash matching the document's current text — is reused as-is; only stale or missing entries fall back to embedding the text on the fly. Results are identical either way (the provenance check guarantees the stored vector is exactly what re-embedding would produce); only the work changes. Running `vsync` first therefore makes repeated searches over a stable collection cheaper — the scan still visits every document, but unchanged documents skip the embedding step.

### Hybrid search

Supplying `--filter` performs **hybrid search**: a structured MongoDB-style pre-filter is applied first, then the surviving documents are ranked semantically — both in a single scan pass. Only documents that match the filter are eligible to rank.

```sh
# Pre-filter to published docs, then rank those by relevance.
omgdb vsearch app.omgdb docs text "database search" --filter '{"status":"published"}'
```

If a draft document is equally relevant to a published one, the filter excludes it entirely. The filter uses the same syntax as the rest of OMGDB — see [query operators](/docs/query-operators/).

## Persisting embeddings: `vsync`

`omgdb vsync` embeds the string `field` of every document in a collection and persists each vector — together with a provenance envelope — into the sibling collection `<collection>.__vectors`. That target is an ordinary, op-log-backed, inspectable collection: it shows up in `inspect`, survives a store reopen, and passes the integrity check. Once synced, those vectors are the ones `vsearch` and [context packs](/docs/context-packs/) reuse.

### Synopsis

```text
omgdb vsync <path> <collection> <field>
```

`vsync` is **incremental and idempotent**. Documents whose persisted vector is still fresh (same model, dimensionality, and content hash) are skipped outright — not re-embedded, not re-written — so an immediate re-run appends nothing to the log. Stale entries are replaced by `_id` and new documents inserted, batched into one transaction per kind rather than one write cycle per record. Documents lacking a string `field` are skipped. The printed count is the number of vectors actually **written**:

```sh
omgdb vsync app.omgdb docs text
# stdout: synced 1 embedding(s) into `docs.__vectors` (fresh ones skipped)
omgdb vsync app.omgdb docs text
# stdout: synced 0 embedding(s) into `docs.__vectors` (fresh ones skipped)
```

Each persisted record has the shape `{_id, provenance, vector}`, where `provenance` is a document with keys `model`, `dim`, `contentHash`, and `sourceField`, and `vector` is an array of doubles.

### The provenance envelope

Every persisted embedding stores enough to trace it to its producer and detect staleness:

| Field | Description |
| --- | --- |
| `model` | The `model_id` of the embedder that produced the vector (e.g. `hashing-v1/dim=256`). |
| `dim` | The embedding dimensionality. |
| `contentHash` | A 16-hex-digit FNV-1a hash of the exact embedded text. |
| `sourceField` | The document field the text was taken from. |

This makes AI-derived state auditable rather than opaque: a stored vector can always be traced back to the model and the exact text it came from — and the same provenance check is what lets search trust a stored vector enough to reuse it.

## Detecting stale embeddings: `vstale`

`omgdb vstale` reports the `_id`s in a collection whose persisted embedding is stale relative to its source. An embedding is stale when:

- No persisted vector exists for the document, **or**
- The embedder's `model_id` differs from the recorded `model` (different model or configuration), **or**
- The dimensionality differs, **or**
- The source text has changed (the content hash no longer matches).

### Synopsis

```text
omgdb vstale <path> <collection> <field>
```

`vstale` prints the stale `_id`s (canonical JSON) to stdout and a summary count (`N stale embedding(s)`) to stderr.

### Example

```sh
omgdb create app.omgdb
omgdb insert app.omgdb docs '{"text":"alpha beta"}'

omgdb vstale app.omgdb docs text   # stderr: "1 stale embedding(s)" (no persisted vector yet)
omgdb vsync  app.omgdb docs text   # stdout: "synced 1 embedding(s) into `docs.__vectors` (fresh ones skipped)"
omgdb vstale app.omgdb docs text   # stderr: "0 stale embedding(s)"
```

Editing a document's source text marks only that document stale; re-running `vsync` re-embeds only it. Because the vectors live in an ordinary op-log-backed collection, they survive a reopen and remain consistent — and a stale vector is never silently served: search's freshness check falls back to on-the-fly embedding for exactly those documents until the next `vsync`.

## Library API

The `omgdb-vector` crate exposes these as plain Rust functions over a `Store`:

```rust
let e = HashingEmbedder::default();

// Flat cosine kNN: Vec<(Value /* _id */, f32 /* score */)>, best-first.
// Fresh persisted vectors from `docs.__vectors` are reused automatically.
let results = search(&store, "docs", "text", "database search", 5, &e);

// Hybrid search with a compiled pre-filter.
let filter = omgdb_query::Filter::compile(
    &Value::from_json_str(r#"{"status":"published"}"#).unwrap(),
).unwrap();
let hits = search_where(&store, "docs", "text", "database search", 5, &e, &filter);
```

| Function | Purpose |
| --- | --- |
| `search` / `search_where` | Flat cosine kNN, optionally with a structured pre-filter (hybrid). Reuses fresh persisted vectors via a `VectorLookaside`. |
| `sync_vectors` | Incrementally persist embeddings + provenance into `<ns>.__vectors`; returns the count written (fresh entries are skipped). |
| `list_stale_vectors` | The `_id`s whose persisted embedding is stale. |
| `VectorLookaside` | The loaded `<ns>.__vectors` records; hands search a stored vector only when its provenance is fresh. |
| `cosine` | Cosine similarity of two slices. |
| `vector_ns` | The sibling collection name `{ns}.__vectors`. |
| `context_pack` / `context_pack_where` | Token-budgeted, cited retrieval bundle (see [context packs](/docs/context-packs/)). |

## MCP tools

The vector surface is also exposed over MCP for agents. The read tools are idempotent and gated at the read scope, so they are available even on a read-only MCP server; `vsync` writes to the store and requires the read-write scope.

| Tool | Scope | Args | Description |
| --- | --- | --- | --- |
| `vsearch` | read | `path`, `collection`, `field`, `query`, optional `k` (default 5), optional `filter` | Semantic search over a text field; `filter` enables hybrid search. |
| `vstale` | read | `path`, `collection`, `field` | List the `_id`s whose persisted embedding is missing or stale. |
| `context_pack` | read | `path`, `collection`, `field`, `query`, optional `budget` (default 1000), optional `filter` | Token-budgeted, cited [context pack](/docs/context-packs/). |
| `vsync` | read-write | `path`, `collection`, `field` | Persist or refresh embeddings into `<collection>.__vectors`. |

See [MCP](/docs/mcp/) for connecting an agent.

## Limitations and caveats

> **Limitation:** Search is flat/exact cosine kNN with a full collection scan — there is no ANN index. Cost is O(N) per query: every document is visited and scored, even though documents with fresh synced vectors skip the re-embedding step.

- The bundled `HashingEmbedder` is a lexical bag-of-words baseline, not a semantic neural model. For real neural embeddings, install a pinned local model (`omgdb models pull`) and use [`search --mode semantic`](/docs/search/) — available on supported native builds.
- Only string fields are embedded and searched; documents lacking a string value at the field are silently skipped by `sync_vectors`, `search`, and context packs.
- Both content hashing (staleness detection) and the embedder's bucketing use a 64-bit FNV-1a hash, chosen for reproducibility, not collision resistance. The `contentHash` is a 16-hex-digit string.
- An unreadable or absent `__vectors` collection never fails a search — the lookaside simply comes up empty and every document is embedded on the fly.

## Related

- [Full-text search](/docs/search/) — the unified typed search surface: BM25, semantic mode with pinned local models, and true RRF hybrid fusion.
- [Context packs](/docs/context-packs/) — token-budgeted, cited retrieval built on the same ranking.
- [Query operators](/docs/query-operators/) — the filter syntax used by hybrid search.
- [MCP](/docs/mcp/) — running semantic search and context packs from an agent.