Aggregation Pipeline

An aggregation pipeline transforms a stream of documents through an ordered list of stages. In OMGDB a pipeline is a JSON array of stage documents — each stage is an object with exactly one $-prefixed key — passed to the CLI as a single JSON-string argument:

omgdb aggregate app.omgdb <collection> '[<stage>, <stage>, ...]'

The pipeline runs against the documents scanned from <collection>, then each stage feeds its output documents into the next. Stages, group accumulators, and expression operators are each dispatched by a single flat match on the operator name in the omgdb-agg crate. The design is deliberately additive — supporting a new operator means adding one match arm — but it is not a runtime registry: there is no registration API, trait objects, or plugin mechanism, and an unrecognized operator returns an unknown-operator error (with a did-you-mean suggestion when a close match exists).

Expressions are evaluated by a recursive engine. A string beginning with $ (e.g. "$a.b") is a field path resolved against the current document (missing paths yield null); a string beginning with $$ is a system variable; any other string is a literal. An object whose first key starts with $ is a single-operator expression (and must have exactly one key); any other object is a literal sub-document evaluated field by field; arrays are evaluated element-wise.

This is a compatible subset of MongoDB’s aggregation framework, not a complete implementation. See query operators for the filter syntax used inside $match.

Stages

Stages run in array order. Each stage object must have exactly one $-key; otherwise the pipeline is malformed.

Stage notes

• $documents requires an array whose entries are all objects, and it is rejected anywhere except position 0.
• $project operates on top-level fields. An inclusion projection keeps _id first unless _id:0; in an inclusion projection, excluded fields other than _id are simply ignored rather than raising an error. A projection with only 0/false values is an exclusion projection.
• $unset takes a field-name string or an array of them. Names may be dotted paths (meta.debug); removing through an array removes the key from every element. Names must be nonempty and must not start with $.
• $group requires an _id. Every non-_id field must be a single-accumulator object. Groups are keyed by the canonical JSON of the _id value and stored in an ordered map, so output is ordered by that canonical key rather than by input order.
• $bucket requires groupBy and a boundaries array of at least two values that share one type (mixed integer/float is allowed) and are strictly ascending. A document whose groupBy value falls outside every boundary goes to the default bucket — the default id must itself sort outside the boundary range — or, with no default, is a type error. output declares accumulators per bucket (the default output is {"count":{"$sum":1}}); empty buckets are omitted.
• $sortByCount groups by any expression (a document literal is rejected), counts each group with checked arithmetic, and sorts by count descending with ties broken by _id under the engine’s value order.
• $sort keys support dotted paths. Missing field values sort before present ones. A direction other than 1 or -1 is a malformed error.
• $unwind accepts a "$field" string or {"path":"$field"}. It operates on a top-level field (the leading $ is stripped). A missing or null array value silently drops the document; a non-array value is passed through as a single document. There are no preserveNullAndEmptyArrays or includeArrayIndex options.
• $replaceRoot errors if newRoot does not evaluate to an object. $replaceWith: <expr> is sugar for $replaceRoot: { newRoot: <expr> }.
• $unionWith accepts a collection-name string, or {coll, pipeline} where the optional pipeline runs over the unioned collection first. coll may be omitted only when the pipeline starts with $documents (an inline union).
• $facet collapses the stream to a single document. Sub-pipelines inherit the store, so they may use $lookup and $unionWith.

$lookup in depth

$lookup supports all three MongoDB join forms:

{"$lookup":{"from":"items","localField":"item","foreignField":"_id","as":"itemDocs"}}
{"$lookup":{"from":"items","pipeline":[{"$match":{"active":true}}],"as":"activeItems"}}
{"$lookup":{"from":"orders","let":{"uid":"$_id"},
  "pipeline":[{"$match":{"$expr":{"$eq":["$userId","$$uid"]}}}],"as":"orders"}}

• The equality form builds a hash map over the foreign side (one pass, not a nested scan per input document) and is array-aware on both sides: a scalar on either side matches an element of an array on the other. Dotted localField/foreignField paths are supported.
• The uncorrelated pipeline form runs pipeline over the foreign collection once and attaches the result to every input document.
• The correlated form binds let variables per input document; the sub-pipeline reads them as $$name (typically inside $match + $expr). Variable names must be nonempty and must not start with $ or contain ..
• localField and foreignField must appear together, and at least one of localField/pipeline is required. Unknown options are rejected by name.

Note: $lookup and a collection-sourced $unionWith join other collections and therefore need a backing store. They are available when the pipeline runs against a collection (as the omgdb aggregate command does); running a pipeline over an in-memory document list without a store makes them malformed errors.

Accumulators

Accumulators appear inside $group (and $bucket’s output), one per output field, applied to the values produced by evaluating the accumulator’s argument expression for each document in the group.

Expression operators

Expression operators are used inside $project, $addFields/$set, $group accumulator arguments, $replaceRoot, $bucket’s groupBy, $sortByCount, and $match’s $expr. They take a single argument or an argument array, each element of which is itself an expression evaluated against the current document.

System variables

A string beginning with $$ reads a system variable. $$ROOT and $$CURRENT both resolve to the current document, and both accept dotted paths ("$$ROOT.a.b"). Inside a correlated $lookup sub-pipeline, the join’s let bindings are also available as $$name (with optional dotted paths into object values). Any other $$-name is a malformed error.

Literal and conditional

Note: No matching $switch branch and no default is a type error.

Arithmetic

All arithmetic operators require numeric arguments; a non-numeric argument is a type error.

Comparison

Each comparison takes exactly 2 arguments and returns a boolean, comparing under the engine’s ordering.

Boolean logic

A value is falsy if it is null, false, integer 0, or float 0.0; everything else (including empty string or array) is truthy.

Note: $and and $or do not short-circuit — all arguments are evaluated before the result is combined.

Math

Limitation: $round and $trunc take no precision/decimal-places argument (unlike MongoDB). $ceil/$floor/$round/$trunc applied to a float return a float with an integral value (e.g. 3.0), not an integer-typed value.

Strings

Arrays and objects

Limitation: Genuinely missing from the expression engine today: $map, $filter, $reduce, $arrayToObject, $reverseArray, $sortArray, the date operators, $regexMatch, $substr/$substrCP, $trim, and the type-conversion operators ($toString, $toInt, $convert, …). The $$NOW system variable is also unsupported. Each returns an unknown-operator or malformed error rather than a wrong answer.

Limitation: Missing stages include $bucketAuto, $out, $merge, $sample, and $graphLookup — an unrecognized stage returns an unknown-operator error (with a did-you-mean hint when a close match exists).

Worked example

Group by a field with $sum and $avg, then sort the groups. Given a salaries collection of {"dept": ..., "sal": ...} documents:

omgdb aggregate app.omgdb salaries '[
  {"$group":{"_id":"$dept","total":{"$sum":"$sal"},"avg":{"$avg":"$sal"},"n":{"$sum":1}}},
  {"$sort":{"total":-1}}
]'

For input documents:

[
  {"dept":"a","sal":100},
  {"dept":"a","sal":200},
  {"dept":"b","sal":50}
]

The $group stage produces one document per department, then $sort orders them by total descending:

[
  {"_id":"a","total":300,"avg":150.0,"n":2},
  {"_id":"b","total":50,"avg":50.0,"n":1}
]

Note that total is an integer (300) because every summed value was an integer, while avg is a float (150.0). The {"$sum":1} accumulator counts documents per group.

A second example combines computed fields with a projection — adding a concatenated full field, then keeping only it:

omgdb aggregate app.omgdb people '[
  {"$addFields":{"full":{"$concat":["$first"," ","$last"]}}},
  {"$project":{"_id":0,"full":1}}
]'

For {"first":"ada","last":"lovelace"} this yields {"full":"ada lovelace"}.

Error behavior

Pipeline errors are surfaced rather than silently swallowed:

• A pipeline that is not an array, or a stage that is not a single $-operator object, is a malformed error.
• An unrecognized stage, accumulator, or expression operator is an unknown operator error, and when a known operator is within edit distance 2 the message adds a did-you-mean suggestion — so a typo like $grup comes back as $grup (stage) (did you mean `$group`?) and an agent can repair the pipeline in one step.
• An operand of the wrong type (e.g. $concat on a number, $divide by zero, integer overflow in $add/$subtract/$multiply/$mod/$sum) is a type error. Integer arithmetic uses checked operations, so overflow returns an error instead of panicking.
• A $match filter that fails to compile surfaces the underlying query error.