test(precompute): add real-pipeline e2e regression test for #474 - #538
Open
milindsrivastava1997 wants to merge 2 commits into
Open
Conversation
Stacked on the wall-clock override hook. Adds JsonFileIngestConfig:: batch_delay_ms (default 0, a no-op everywhere in production and the existing netflow_single_second e2e test) — a real, explicit delay after each batch send, so an integration test can force ingest to span real wall-clock time deterministically instead of hoping incidental scheduling overhead is enough. New e2e test drives the real JsonFileIngestSource -> PrecomputeEngine -> sink pipeline with a bulk single-timestamp load spread across 10 batches, using the wall-clock override (unscaled — an earlier scaled version amplified ordinary scheduling jitter into spurious failures) and generous, seconds-scale margins so the periodic flush timer gets many real chances to land mid-ingest without the test depending on tight timing. Verified red against the pre-fix birth-time semantics (temporarily reverted, ran, confirmed rows were dropped mid-ingest, restored the fix) before counting it as a real regression test. Also fixed allowed_lateness_ms in the new test's config: every flush_all() call unconditionally nudges the event-time watermark forward by 1ms (the "boundary advance" that lets an idle stream make progress), independent of the wall-clock fallback. With allowed_lateness_ms=0 and multiple batches spaced across several flush ticks, that 1ms drift alone was enough to mark the next same-timestamp batch "late" via a wholly different code path, unrelated to #474. A generous, production-realistic value avoids that. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…mpute-wall-clock-e2e-test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JsonFileIngestConfig::batch_delay_ms(default0, a no-op everywhere in production and in the existingnetflow_single_seconde2e test) — a real, explicit delay after each batch send, so an integration test can force ingest to span real wall-clock time deterministically instead of hoping incidental scheduling overhead is enough.JsonFileIngestSource → PrecomputeEngine → sinkpipeline with a bulk single-timestamp load spread across 10 batches, using test(precompute): add a wall-clock override hook to PrecomputeEngine #537's wall-clock override (unscaled — an earlier scaled version amplified ordinary scheduling jitter into spurious failures) and generous, seconds-scale margins so the periodic flush timer gets many real chances to land mid-ingest without depending on tight timing.allowed_lateness_msin the new test's config: everyflush_all()call unconditionally nudges the event-time watermark forward by 1ms (the "boundary advance" that lets an idle stream make progress), independent of the wall-clock fallback. Withallowed_lateness_ms=0and multiple batches spaced across several flush ticks, that 1ms drift alone was enough to mark the next same-timestamp batch "late" via a wholly different code path, unrelated to fix(precompute): bulk single-timestamp ingest silently drops rows when wall-clock fallback closes the window mid-ingest #474. A generous, production-realistic value avoids that.Test plan
e2e_netflow_single_secondande2e_precompute_equivalencestill pass.precompute_engine::unit suite (81 tests) passes.cargo clippy -p query_engine_rust --lib --testsclean.🤖 Generated with Claude Code