fix(precompute): keep wall-clock fallback from force-closing panes still under active ingest - #536
Open
Conversation
…ill under active ingest The wall-clock fallback in flush_all() ages a pane using the wall-clock time it was first touched, never refreshed. A bulk load whose rows all share one event-time (so the event-time watermark never advances) that takes longer than window_size_ms + wall_clock_grace_period_ms to ingest therefore gets its window force-closed mid-ingest, and every sample arriving afterward is silently dropped by the (hardcoded) late-data path — a uniform undercount across every group key (#474). Refresh the pane's wall-clock bookkeeping on every touch instead of only the first, so the fallback measures idle time, not age since birth: a pane still receiving samples is never force-closed no matter how long it's been open, only a pane nothing has touched in a while is. Renamed pane_wall_clock_starts_ms -> pane_wall_clock_last_touch_ms (and its prune helper) to match, and updated the doc comments describing the old birth-time semantics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
…k-single-timestamp-ingest-silently-drops-rows-when-wall-clock-fallback-closes-the-window-mid-ingest-1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
flush_all()aged a pane by time-since-first-touch, never refreshed. A bulk load whose rows all share one event-time (so the watermark never advances) that takes longer thanwindow_size_ms + wall_clock_grace_period_msto ingest got its window force-closed mid-ingest, silently dropping every row that arrived afterward (uniform undercount across every group key). Fixes fix(precompute): bulk single-timestamp ingest silently drops rows when wall-clock fallback closes the window mid-ingest #474.pane_wall_clock_starts_ms→pane_wall_clock_last_touch_ms(+ its prune helper) to match the corrected semantics, and updated the doc comments that described the old birth-time behavior.First of a 3-PR stack; this one is the minimal, self-contained bug fix and can ship on its own. #2 and #3 add e2e test infrastructure on top.
Test plan
wall_clock_fallback_does_not_close_a_pane_still_receiving_samples, verified red against the pre-fixor_insertsemantics before restoring the fix.precompute_engine::unit suite (81 tests) passes.e2e_netflow_single_secondande2e_precompute_equivalencee2e tests pass.cargo clippy -p query_engine_rust --lib --testsclean.🤖 Generated with Claude Code