This repository was archived by the owner on Oct 8, 2026. It is now read-only.
Repository navigation
test(perf): add manual-only extended coverage and DuckDB baseline - #1241
Merged
Merged
Conversation
Test Impact PlanDeterministic summary of how this PR changes tests, CI runners, and coverage-risk signals. Summary
Signals
Coverage risk: neutral or increased No coverage-reduction warnings detected. |
bill-ph
marked this pull request as ready for review
September 25, 2026 17:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The frozen benchmarks leave common analytical query shapes unmeasured. Add 12 versioned workloads covering filtered uniques/trends, two person joins, event and actor retrieval, ordered funnels, sampled-week retention, two cohorts, and daily-person/session modeling SELECTs.
Manually select
scenario=posthog_frozen_perf_extendedto run all five configurations, or setcoverage_targetfor one. Shared image builds feed isolated, sequential target jobs with separate artifacts and a pinned Trino digest. Each target gets a four-hour scenario / 270-minute job budget; failures do not cancel remaining targets. Extended queries stay out of nightly runs.Each query runs one warmup and four measurements. Fixed populated windows, tenant-aware person deduplication, and fixture checks avoid misleadingly empty workloads. Scenario Job names respect Kubernetes limits.
Comparable dev-sized worker resources
Standard frozen perf and extended perf use three Trino workers at 7 CPU / 28 GiB each and one DuckDB worker at 21 CPU / 84 GiB. Worker requests equal limits. Trino uses a 20 GiB heap, 10 GiB per-worker query memory and 30 GiB cluster query memory. DuckDB sizing admission caps rise with its requested profile; the current headroom policy derives a 63 GiB engine memory limit. Aggregate container resources match; coordinator/support overhead and engine memory policies differ.
Extended Trino jobs deploy only the selected comparison cache mode. Bootstrap/general E2E resources retain their small defaults. Nightly keeps its original query set with the larger comparison resources.
Deployment prerequisite: charts #16434 allows node sizes that can accommodate the larger pods. Apply it before running this profile. No larger-profile benchmark has run yet.
Historical measurement
The uncached DuckDB baseline run, commit
9c9f5f305f87b7f0d41c0b54d64e5648aeb8ef83, completed 12 warmups and 48 measurements without errors: 87m06s benchmark phase, 1h42m45s workflow. Sanitized samples and methodology remain intests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25.That run used the old 3 CPU / 12 GiB aggregate worker budget. It is historical data, not a like-for-like comparison base for the new profile. Its runtime extrapolations are also historical; a fresh measured run is required to establish the larger-profile baseline and forecast. The subsequent all-target run hit Trino's old 2 GiB per-node query-memory limit on joins and session modeling; this sizing change addresses that constraint but does not claim those queries now pass.
Validation and limits
just lintstill reports six pre-existing SA4023 diagnostics at three unchanged sites.