[ragnorc]

Summary

A correlated block such as

not {
    $s0: Source { slug: $exclude }
    $t0: Touchpoint
    $t0 touchpointOnSource $s0
    $t0 touchpointByIdentity $x
}

ran its inner tree once, but from the outer rows: every outer row's $x
was expanded to all of that identity's touchpoints, each looked up in
Touchpoint and then Source in slices of 256 rows. The cost was the outer rows
times $x's degree, even though the excluded side is small.

The planner now builds a second inner tree for blocks that touch the outer row
only through the correlation binding's identity: the same pattern, rooted at the
block's most bounded binding (a key or unique equality or an in list on its
own scan), with each Expand toward that root reversed, ending on $x's id.
A new physical pass, anti_join_strategy, prices both trees by their summed row
estimates and lowers the cheaper one. The physical AntiJoin records
strategy: correlated | key_set. Under key_set, AntiJoinMaskExec streams
the inner rows into one SubqueryAggregate group per outer key and gives each
outer row its key's verdict. This is a set-based anti-join on the shared
variable; count and aggregate blocks (count { } >= 2, sum($t0.w) { } > 3)
use the same path.

The logical plan still holds the correlated tree as the block's meaning. A block
stays correlated when it reads an outer column (including properties of the
correlation binding), when no binding is bounded, when it nests a block or
starts a second component, or under a traversal work limit.

Measurements

All numbers below are from a read-only copy of a production CRM graph's storage
(synced 2026-10-12, the same table versions the live server plans with:
identical explain output and assumptions). Nothing was written to the live
deployment, and no maintenance ran on it. On the live 0.13 server, the read took
about 88 s and the same read without the block about 25 s.

Where the 60 s went

The cardinalities for one channel watching 143 sources and one excluded source:

	rows
outer rows reaching the block ($t, $x pairs)	3,747 (2,250 distinct $x)
correlated inner expansion $x → $t0	40,630
… then Touchpoint id lookups / Source id lookups	159 slices each
excluded source's touchpoints ($s0 → $t0)	1,601

The id_lookup scans each read the whole table, because the live tables have no
indexes (index_facts are empty, and there is no _indices/ under any table),
and the pinned table versions have 9 to 35 fragments each.

Object-store requests: correlated vs key set

These numbers come from the same storage served by a local S3 endpoint (moto),
counting the object GETs in its access log per query. Lance data/*.lance range
reads dominate them. Wall time is through that endpoint: the absolute times
include moto's per-request cost, but the 0.13 times track the live server
closely (21.8 s vs 25 s live for the base read, 74.6 s vs 88 s live with the
block).

binary / read	GETs	wall
0.13, no block	15,120	21.8 s
0.13, with not { } (correlated)	27,004	74.6 s
this branch, with not { } (key_set)	15,380	31.1 s ¹
this branch, block forced correlated ²	27,004	54.5 s ¹

¹ A debug build; its base read takes 30.8 s through the same endpoint, so the
block adds about 0.3 s instead of about 53 s.
² $x.slug = $x.slug inside the block reads the outer row, so the block stays
correlated. This reproduces 0.13's request count exactly.

The block's own cost drops from +11,884 to +260 GETs (46×). Results are
byte-identical across 0.13, key_set and forced correlated.

The checked-in instrument (anti_join_cost.rs, ignored, instrument:) runs the
three reads over a synthetic CRM-shaped graph (48 commits, no indexes) or over
any graph named by OMNIGRAPH_ANTI_JOIN_STORE, and reports rows through the
operators and wall time. On this copy:

read	rows through all operators	rows through the block's inner tree	local wall
no block	57,690	0	2.57 s
key_set	65,194	4,804	2.67 s
correlated	187,451	127,061	6.66 s

The 25 s base cost is storage fragmentation, not missing indexes

I measured the base read (no block) on three storage layouts of the same rows:

layout	fragments per query table	indexes	GETs	wall (moto)
live copy as-is	9–35	none	15,120	21.8 s
same 7 tables reloaded in 1–2 loads	1–2	none	1,118	3.4 s
live copy after omnigraph optimize (0.13)	1	built	1,777	4.2 s

Compaction alone removes 93% of the base read's requests. Index builds do not
help this read: every id_lookup slice adds BTREE probes, so the indexed
layout issues more requests than the compacted, unindexed one. The plan
multiplies the per-scan cost: Touchpoint is reached by id_lookup in 168
slices because the planner estimated a frontier of 1,890 rows where there were
42,912 (average fan-out over a skewed channel). A hash join would read the
table once, but the estimate kept it below HASH_JOIN_RATIO. Requests are
therefore roughly slices × fragments × columns.

Recommended for that deployment (not done here): run optimize on the graph,
with its writers paused. With this change
deployed, the lookalike read should take about as long as its base read: on
today's layout +260 GETs over the base, and after compaction about 1,800 GETs
in total.

Tests

• crates/omnigraph-gqt/cases/v2/planner/anti_join_key_set_from_a_bounded_binding.gqt:
  the CRM shape at small scale. It pins anti join $x: key_set and
  pass anti_join_strategy with expect same as v1, covers count, sum and
  bound-edge blocks, and has two counterexamples that stay correlated (an
  unbounded inner binding, and a block that reads $t). The new plan line
  anti join $var: <correlated|key_set> is added to the GQT grammar with
  parser and checker tests. The runner's saved-plan round trip now also
  compares the strategy.
• engine_v2.rs::bounded_correlated_block_runs_once_from_its_key_set: a
  400-outer-row graph where the correlated tree would expand 12,030 rows. It
  asserts the KeySet plan, no OuterReference, the anti-join input of 400
  rows, both inner expands at exactly 3 rows (the excluded touchpoints), and the
  kept identities.
• query_plan/cost.rs: the key-set tree's shape (a table scan of the bounded
  binding, then both hops reversed, ending on $a with no scan), the
  anti_join_rows($a) statistic both ways (4001 vs 121 chooses key_set,
  122 vs 1201 keeps correlated), and the ineligible shapes.
• report/tests.rs: a key-set block's operators all execute and the plan has no
  outer leaf.
• crates/omnigraph-gqt/cases/v2/planner/anti_join_key_set_reverses_hops_and_counts.gqt:
  the reversal's semantics, each step pinned to key_set. It covers
  knows{2,2}, knows{1,2} and a block written from the bound binding
  ($x knows $p), each with expect same as v1. Over
  parallel edges, an unbound count { } = 1 counts distinct neighbours, and a
  bound-edge count { } = 1 / = 2 counts edge rows.
• bound_plan.rs: BOUND_PLAN_VERSION is 9, every older version is refused,
  and strategy round-trips.
• I explained all 130 block queries in the GQT corpus with this build. Besides the
  two new cases, anti_join_negation_forms_agree's not_at_acme and
  not_at_acme_binding now plan key_set and keep their expected rows. Every
  other existing block stays correlated, because it has no key-bounded inner
  binding or reads the outer row.

Run locally: cargo test -p omnigraph-planner, cargo test -p omnigraph-gqt-core --lib,
the full GQT corpus (350 cases; 29 write/DST cases time out at 10 s under
concurrent machine load and pass when rerun serially),
cargo test -p omnigraph-engine --test integration and the engine --lib
report tests, cargo fmt --all --check, and clippy on the changed crates.

Notes and follow-ups

• Saved bound plans move to version 9 (main took 7 for count(distinct ...)
  in feat(gq)!: add count(distinct ...) #965 and 8 for typed ordering in feat(planner): accept every read plan against its checked query (RFC 0047) #839; all unreleased). An older plan is
  refused with the usual regeneration instruction; see
  correlated-block-key-set.breaking.md.
• A correlated block that stays correlated still carries the whole outer row
  width into its inner tree and expands every outer row, duplicates included.
  Deduplicating the correlation keys first would bound that path by distinct
  keys too. This PR does not do that.
• IoCounts.data_reads in the cost harness misses most table reads on some
  graphs (0 on the reloaded layout, about 2,000 where moto saw 15,120 on the
  as-is copy). That is why this PR's request counts come from the S3 endpoint
  log rather than the in-process probe.
• The base read's Touchpoint access path is chosen from a 1,890-row estimate
  against 42,912 actual rows. An id_lookup that observes a frontier past
  HASH_JOIN_RATIO could switch to a hash join at run time, as Expand
  switches modes; today the declared switch only goes the other way.
• order { repos desc } limit 500 over an aggregate declares no tie-break, so
  which tied rows make the cut depends on the storage layout.

[cursor]

Review: key-set anti-join for bounded correlated blocks

Recommendation: approve after the docs conflict with main is resolved. I found no correctness defect. The one required action is mechanical. The other findings are optional follow-ups. I posted this as comments because my tool cannot submit an approve event.

Reviewed head: d08410f260f938ffd890a3b1a40557ebe55e3c32, in a clean, detached worktree.

Why, what, how, impact

• Why: A correlated block ran its inner tree from the outer rows. Its work was the outer rows times the correlation binding's degree, even when the block's own pattern was small. In the measured CRM read, the block added about 53 s and about 11,900 object GETs.
• What: When a block reads the outer rows only through the correlation binding's identity, and one of its own bindings is bounded by a key, a unique equality or an in list, the planner builds a second inner tree. That tree starts at the most bounded binding, reverses each hop written toward it, and ends on $x's id. A new pass, anti_join_strategy, compares the summed row estimates of both trees and lowers the cheaper one. The physical AntiJoin records strategy, and BOUND_PLAN_VERSION goes from 8 to 9.
• How it runs: Under key_set, AntiJoinMaskExec drains the outer rows and makes one SubqueryAggregate group per distinct key. It streams the inner rows into those groups without holding them, and gives each outer row its key's verdict. A null key gets the verdict of no matches.
• Impact: Results do not change, and the logical plan keeps the correlated tree as the block's meaning. Plans, explain output and bound plans gain strategy. Saved bound plans at version 8 are refused, and the breaking changelog note says so.

Findings

Severity	Where	Finding
Required (mechanical)	docs/dev/execution.md:385	Merge conflict with #974's Projection row. Keep both rows. I tested the resolution.
Consider (follow-up)	physical.rs:690	Plan acceptance does not check the strategy against the inner tree's shape. A mismatched, edited plan fails closed, but only at execution and as an internal error.
Consider	key_set.rs:10	Add a differential property, key-set against forced correlated, to guard the "reversed hop gives the same pairs" rule against future Expand changes.
Nit	key_set.rs:216	Root choice among several bounded bindings, and the tie rule, are untested. Two mutations survive. This is cost-only.
Nit	optimizer.rs:2934	The arena swap depends on a key-set tree never holding an AntiJoin. Add a debug_assert!.
FYI	cost.rs:150	Rows-only pricing with average degree. A hub root under a tiny outer side can be under-priced. Relevant to #996.

Tradeoffs and liability

• Correctness and the contract: The rewrite is a planner choice between two physical forms of one logical meaning. That matches the invariant "physical acceleration is derived state". The eligibility rules fail toward correlated: any outer read (the correlation binding's own properties included), a nested block, a second component, fragments, a runtime filter, a work limit, or an unknown estimate. I checked the empty-group, null-key, duplicate-key and discarded-key paths in key_set_mask. The i128 sum and the u64 count cannot overflow from the shared discarded group.
• Memory: Under key_set the inner rows stream, and the operator holds the outer batch plus one accumulator per distinct key. The correlated path drains and holds all inner batches. This is an improvement, and every allocation goes through WorkMemory.
• Observability: The decision is visible. anti_join_rows($x) goes into the decisions, explain prints key_set, and GQT can assert anti join $x: <strategy>. The cost claim has a checked-in instrument: test, as the invariants require.
• Ongoing liability (five more changes of the same kind):
  • The design adds one more planner-only rewrite whose soundness rests on an Expand property: a reversed traversal emits the same pairs. That property is the main thing to guard. rfc: polymorphic types with interfaces, unions and polymorphic edges #884 (polymorphic EdgeSelection) will change reversed() and the endpoint types.
  • key_set_mask groups by a bare Utf8 id (key_array). That is correct today, because the correlation binding has one concrete type. Under rfc: polymorphic types with interfaces, unions and polymorphic edges #884's identity work, anti-join keys become the packed node reference, and this map must change with them.
  • Row-only pricing will need rfc: request-aware plan cost #996's request model to stay correct on fragmented, unindexed tables.
• Deprioritized by design: Cost accuracy on skewed graphs, in exchange for a simple, deterministic, inspectable rule. This is acceptable because the bad case is bounded by the root's real neighbourhood.

Validation

Toolchain 1.97.1 (pinned) and protoc 3.21.12, on a local filesystem. All commands ran with --locked.

Command	Result
cargo test -p omnigraph-planner	lib 64 passed, 1 ignored. Integration 92 passed.
cargo test -p omnigraph-engine --lib anti_join	2 passed, including key_set_anti_join_executes_its_inner_tree_without_an_outer_leaf
cargo test -p omnigraph-engine --test integration -- engine_v2::bounded_correlated_block_runs_once_from_its_key_set	passed
cargo test -p omnigraph-engine --test integration -- proptest_equivalence:: anti_join bound_plan	4 passed. The anti_join_cost instrument stayed ignored, as designed.
cargo test -p omnigraph-gqt	356 of 356 cases passed, plus the runner suites (47, 14, 26)
cargo fmt --all --check	clean for every PR file
Local differential property (review-only, not pushed)	64 random graphs, 9 block shapes, each also with duplicated outer rows. The written block ran as key_set in 283–398 runs per shape, and the forced-correlated twin never did. All row multisets were equal.
Mutations in key_set.rs	Removing the residual filter was caught by GQT. Removing the outer-read check was caught by GQT and a planner test. Changing the root comparison to > or <= was not caught.
Trial merge with main cea15256	Docs conflict resolved as described inline. Planner: 66 passed and 1 ignored (lib), 92 passed (integration). GQT 361 of 361 passed. Both key-set engine tests passed.
CI at this head	28 passed, 5 skipped

Limitations:

• I did not run clippy locally. CI passed it at this head.
• I did not run the S3, Azure, DST or server suites.
• I did not run the cost instrument on production-shaped storage, so the measurement tables in the description are not reproduced here.
• I did not execute the mismatched-strategy plan. That finding comes from reading the code.
• I read the Lance performance, read/write, object-store, observability, index-format, BTREE and DataFusion pages. The key-set root scan uses the existing scan path, so Lance index coverage changes cost but not results. I found no Lance 11 behaviour that this change depends on beyond the existing scan path.