Conversation
…raints Cardinality, range, and histogram queries funnel through cardinalityDocMatchesConstraints, a linear scan over a doc-id slice that is invoked once per scanned index entry. For histogram/range children the full field prefix is re-scanned per bucket and tested against that bucket's doc-id list, making the scan O(entries * constraints) -- i.e. quadratic in the document count. These were the only algebraic queries measured slower than a plain document scan in the algebraic benchmark. Introduce DocIdConstraintSet, a borrowed-key hash-set built once per scan, and replace the linear membership test at all eight cardinality scan/collect sites. Each membership test becomes O(1) and the scan is a single linear pass again. Results are unchanged (the benchmark cross-checks algebraic output against document-scan checksums). https://claude.ai/code/session_01UVKuQU8S6THBNkSBHoVnzz
Histogram, range, and terms cardinality queries computed each bucket's nested child cardinality by re-scanning the entire child field once per bucket (scanDistributedCardinalityPartialsForDocIds), making the cost O(buckets * field_entries). Build a ChildCardinalityIndex once per child field (a single cursor scan grouping distinct value-keys by document) and look up each bucket's documents against it, turning the work into O(field_entries + bucket_members). Output partials are byte-identical, so cross-engine correctness checksums are unchanged. Quick profile (1000 docs), algebraic engine avg latency: histogram_nested_cardinality 15.2ms -> 5.4ms range_nested_cardinality 14.3ms -> 10.9ms root_cardinality 5.6ms -> 3.5ms
mergeOneSlotValuesAlloc wrapped each fold/merge contribution in two throwaway single-slot Rows (two row allocations, two value dupes, a combine, and a final dupe) just to call law.combineAlloc on one slot. On the per-contribution fold and distributed-merge hot paths this is ~6 allocations for what is a single law combine. law.combineAlloc already returns a freshly owned buffer, so call it directly and fall back to the law identity when it yields null (preserving the previous "never null" contract). Output bytes are unchanged: every algebraic benchmark checksum is identical before and after.
DerivedJoinFoldAccumulator round-tripped each group's running total
through decimal text on every contribution: combineAlloc parsed the
accumulated text and the contribution, added, and re-formatted, once per
folded row.
Hold the running value natively instead - i64 for count, f64 for
sum/sumsquares, and {sum,count} for avg - and format to text only once
in entriesAlloc. encode/parse round-trips exactly for i64 and f64 ("{d}")
and contributions are applied in the same order, so the formatted result
is byte-identical (every algebraic benchmark checksum is unchanged). The
non-additive laws (min/max, booleans, set/tuple union, max-timestamp)
keep the text-merge path, where original-token text and order matter.
The linear doc-id membership helper has no callers since the cardinality scans switched to DocIdConstraintSet hash-set membership.
Contributor
Author
|
Superseded by #509, which consolidates the extracted cardinality and dynamic-template algebraic work into one PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Why
These PR #141 optimizations are result-preserving and independent of HLL, adaptive materialization, relational storage, and the larger #145 integration.
Impact
Exact algebraic cardinality paths allocate less and avoid repeated traversal while retaining canonical results.
Validation
git diff --check