Full-Text Search

OMGDB has native full-text search. You define a search index with a typed JSON document, and that definition is written to the op-log as a versioned operation — replayed, transacted, and compacted like any insert. Queries are a strict typed AST (term, phrase, prefix, fuzzy, regex, boolean), scored by a deterministic BM25 implementation, and every hit comes back with a highlighted snippet and provenance digests.

There are three retrieval modes:

• lexical (default) — typed BM25 full-text search. Works on every build, including the browser playground.
• semantic — exact cosine ranking over locally computed embeddings from a pinned, checksum-verified model.
• hybrid — lexical and semantic rankings fused with deterministic reciprocal-rank fusion (RRF), captured at one consistent log boundary.

Limitation: Search is correctness-first today. Running a search opens the full in-memory store, and a stale derived index triggers a synchronous whole-index rebuild before answering — there is no incremental tail indexing yet. Semantic mode exact-scores every filtered chunk per request; there is no ANN index. Semantic and hybrid modes need a locally installed model, which is available on supported native builds only (not in the browser, not on Windows GNU).

Defining a search index

define-search-index takes a complete version-1 definition as JSON. Nothing is defaulted silently: fields, weights, and the full analyzer configuration are explicit, so the definition in your op-log is the whole truth about how text is indexed.

omgdb define-search-index app.omgdb '{
  "version": 1,
  "name": "posts-text",
  "collection": "posts",
  "fields": [
    { "path": "title", "weight": 2.0, "analyzer": { "id": "omgdb.english", "version": 1,
      "tokenizer": { "kind": "unicode" }, "stemming": "english", "stopWords": [],
      "normalization": { "lowercase": true, "unicode": "nfc", "foldDiacritics": false },
      "phrasePositions": true, "prefix": { "minChars": 2, "maxChars": 48, "maxExpansions": 1024 } } },
    { "path": "body", "weight": 1.0, "analyzer": { "id": "omgdb.english", "version": 1,
      "tokenizer": { "kind": "unicode" }, "stemming": "english", "stopWords": [],
      "normalization": { "lowercase": true, "unicode": "nfc", "foldDiacritics": false },
      "phrasePositions": true, "prefix": { "minChars": 2, "maxChars": 48, "maxExpansions": 1024 } } }
  ],
  "bm25": { "k1": 1.2, "b": 0.75 }
}'

The command answers with the definition’s SHA-256 digest, and a define_search_index record appears in oplog.ndjson. drop-search-index <path> <collection> <index> removes it — also an op-log operation.

• Fields are dotted paths with per-field weights, so a title match can legitimately outrank a body match.
• Analyzers are versioned and explicit: omgdb.english and omgdb.greek (Unicode tokenization + stemming), omgdb.keyword (exact values), and omgdb.ngram (substring matching). Each field carries its own complete analyzer configuration — tokenizer, stemming, stop words, normalization, phrase positions, prefix expansion bounds.
• BM25 parameters (k1, b) are part of the definition, not a runtime flag, so scores are reproducible from the log.

Querying

Queries are a typed AST, not a query-string dialect — there is nothing to escape and no parser ambiguity. An agent (or you) builds JSON:

# One term. Empty `fields` means "all configured fields".
omgdb search app.omgdb posts posts-text '{"term":{"text":"deploying","fields":[]}}'

# Stemming is on for omgdb.english: "deploying" finds a document whose body says "deployment".

The leaf and composite node types:

Useful flags: --limit N (default 10), --min-score, --no-highlight, --highlight-bytes N, --explain, --timeout-ms (default 30000).

What a hit looks like

Every response records the exact log boundary it answered at, and every hit carries its score, the matched fields, a bounded UTF-8-safe snippet with match offsets, and the digests of the definition and analyzers that produced the score:

{
  "mode": "lexical",
  "index": "posts-text",
  "boundary": { "nextLsn": 6, "durableBytes": 1412, "logPrefixSha256": "3d0294dd…" },
  "hits": [{
    "document": { "_id": { "$oid": "…" }, "title": "deploys", "body": "the deployment pipeline ships one static binary" },
    "lexicalRank": 1,
    "bm25Score": 0.9808292530117263,
    "matchedFields": ["body"],
    "highlights": [{ "field": "body", "snippet": "the deployment pipeline ships one static binary",
                     "matchStart": 4, "matchEnd": 14 }],
    "provenance": { "index": { "definitionSha256": "…", "backend": "reference",
                               "backendFormatVersion": "omgdb-reference-bm25-v1" } }
  }]
}

--explain adds a bounded per-hit score breakdown with stable semantics versions, so an agent can see why a document ranked where it did.

Combining with structured filters

--filter takes a MongoDB-style filter that compiles through the same query planner as find. It is intersected with the text query before candidate ranking, and every surviving hit is re-checked against canonical live-document state:

omgdb search app.omgdb posts posts-text '{"term":{"text":"every","fields":[]}}' \
  --filter '{"title":"durability"}'

Structured and full-text search in one call, one store, one consistency boundary.

Semantic and hybrid modes

--mode semantic embeds the positive text of your query with a locally installed model and ranks by exact cosine similarity; --mode hybrid runs both rankings at one captured boundary and fuses them with versioned deterministic RRF, reporting the fused score alongside both ranks.

Semantic query text is derived from the AST deterministically (positive term/phrase/prefix/fuzzy leaves in traversal order); mustNot branches contribute nothing, and a query with no positive text leaf is rejected for semantic/hybrid mode rather than guessed at. Positive regex is rejected too — a pattern has no literal text to embed.

Local models

Semantic and hybrid modes use pinned, verified local models — never a network call at query time:

omgdb models pull bge-small-en-v1.5     # English
omgdb models pull multilingual-e5-small # Multilingual
omgdb models list
omgdb models inspect bge-small-en-v1.5

models pull is the only command that touches the network. It validates the artifact set — revision, size, SHA-256, tokenizer and config, license evidence — and installs atomically; every later open re-verifies the installed bytes. search itself never downloads anything, and a build without model support says so explicitly instead of silently substituting a weaker embedder.

Limitation: Real model inference compiles on supported native targets only. In the browser playground and on Windows GNU builds, semantic/hybrid modes report the capability as unavailable; lexical search works everywhere. The legacy vector surface (vsync/vsearch with the offline hashing embedder) remains available on every build.

The derived index is honest about itself

Like every binary artifact in OMGDB, the accelerated search index is derived, deletable, and rebuildable — the op-log stays the only source of truth. search-index-status reports exactly where a generation stands:

omgdb search-index-status app.omgdb posts posts-text

[{
  "collection": "posts", "index": "posts-text", "state": "ready",
  "currentBoundary":    { "nextLsn": 6, "durableBytes": 1412, "logPrefixSha256": "3d0294dd…" },
  "checkpointBoundary": { "nextLsn": 6, "durableBytes": 1412, "logPrefixSha256": "3d0294dd…" },
  "tailBytes": 0,
  "definitionSha256": "2a79f487…",
  "documentCount": 3, "tokenCount": 28, "termCount": 28,
  "rebuildReason": "reference-backend-on-demand"
}]

A generation records the definition and analyzer digests, its exact log boundary (next LSN, durable bytes, log-prefix SHA-256), file checksums, and why it was rebuilt. Search refuses to serve a missing, stale, corrupt, swapped, partial, or incompatible generation — it rebuilds cleanly instead, and never answers from an index that omits durable log tail. rebuild-search-index forces that rebuild explicitly.

On supported native builds, a Tantivy-backed generation accelerates candidate collection — but it supplies the candidate set only. The target-independent reference implementation owns tokenization, matching, and BM25 scores, so results are identical (and identically ordered: score-descending, then canonical _id) on every platform, browser included.

MCP tools

The full surface is exposed over MCP with the same scoping rules as the rest of the database: search and search_index_status are read-scope tools; define_search_index, drop_search_index, and rebuild_search_index require the read-write scope.

Limitations and caveats

• Search opens the full in-memory store. Definition, documents, derived generation, filter, and model execution share one immutable boundary — which also means search is subject to the store-fits-in-RAM limit.
• Rebuilds are synchronous and whole-index. Any durable log tail beyond a persisted generation’s boundary makes it stale, and the next search rebuilds before answering. Incremental tail indexing is not implemented.
• No ANN. Semantic mode exact-cosine scores every chunk that survives the filter, per request. Correct, not approximate — and priced accordingly.
• Deadlines are cooperative. --timeout-ms is checked between bounded phases; it cannot preempt a single filesystem or model-inference call.
• Model support is target-specific. Pinned-model inference and models pull require a supported native build; other targets report the capability as unavailable.

Related

• Vector search — the legacy offline embedding surface (vsync/vstale/vsearch), still available everywhere.
• Context packs — token-budgeted, cited retrieval bundles.
• Query operators — the filter syntax --filter accepts.
• Storage & op-log — why the definition living in the log matters.