From b7edd6a7a598061d3c385656cb6e40e76de3d0a3 Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 11:58:05 -0400 Subject: [PATCH 01/13] test(perf): add analytical workload coverage and focused baseline scenario --- .github/workflows/scenario-dev.yml | 4 +- docs/runbooks/perf-coverage.md | 83 +++++++++ tests/mw-dev/run.sh | 2 +- tests/mw-dev/run_sh_test.go | 36 ++-- tests/mw-dev/scenario/coverage_test.go | 76 ++++++++ tests/mw-dev/scenario/runner_test.go | 1 + .../scenarios/posthog_frozen_perf.yaml | 37 ++++ ...posthog_frozen_perf_coverage_uncached.yaml | 84 +++++++++ .../sql/validate_posthog_coverage.sql | 56 ++++++ tests/perf/README.md | 8 +- tests/perf/core/catalog_test.go | 2 +- tests/perf/core/coverage_test.go | 128 +++++++++++++ tests/perf/core/suite_test.go | 16 ++ tests/perf/core/types.go | 9 +- tests/perf/publisher/publisher_test.go | 83 +++++---- .../queries/ducklake_posthog_coverage.yaml | 175 ++++++++++++++++++ 16 files changed, 737 insertions(+), 63 deletions(-) create mode 100644 docs/runbooks/perf-coverage.md create mode 100644 tests/mw-dev/scenario/coverage_test.go create mode 100644 tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_uncached.yaml create mode 100644 tests/mw-dev/scenario/sql/validate_posthog_coverage.sql create mode 100644 tests/perf/core/coverage_test.go create mode 100644 tests/perf/core/suite_test.go create mode 100644 tests/perf/queries/ducklake_posthog_coverage.yaml diff --git a/.github/workflows/scenario-dev.yml b/.github/workflows/scenario-dev.yml index f1db8dcae..e23a0987a 100644 --- a/.github/workflows/scenario-dev.yml +++ b/.github/workflows/scenario-dev.yml @@ -149,7 +149,7 @@ jobs: run: aws eks update-kubeconfig --name "$CLUSTER_NAME" --region "$AWS_REGION" --alias "$KUBE_CONTEXT" - name: Load frozen perf identity and Athena configuration - if: env.SCENARIO_NAME == 'posthog_frozen_perf' + if: env.SCENARIO_NAME == 'posthog_frozen_perf' || env.SCENARIO_NAME == 'posthog_frozen_perf_coverage_uncached' run: bash scripts/scenario_athena_config.sh >> "$GITHUB_ENV" - name: Load properties fixture override @@ -212,7 +212,7 @@ jobs: run: | set -euo pipefail mapfile -d '' perf_summaries < <( - find artifacts/scenario-dev -type f \( -path '*/perf/summary.json' -o -path '*/perf-properties/summary.json' \) -print0 2>/dev/null | sort -z + find artifacts/scenario-dev -type f \( -path '*/perf/summary.json' -o -path '*/perf-properties/summary.json' -o -path '*/perf-coverage/summary.json' \) -print0 2>/dev/null | sort -z ) if [ "${#perf_summaries[@]}" -eq 0 ]; then echo "No scenario perf artifact was produced; skipping historical publish." diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md new file mode 100644 index 000000000..44f2b39d7 --- /dev/null +++ b/docs/runbooks/perf-coverage.md @@ -0,0 +1,83 @@ +# Workload coverage benchmark + +The workload coverage catalog supplements the original frozen table and +properties comparisons. It measures filtered unique users, grouped trends, +joins, event browsing, actor retrieval, a funnel, sampled-week retention, +behavioral cohorts, and read-side modeling queries. Modeling queries measure +the SELECT computation, not writes or a complete materialization job. + +## Run + +The regular `posthog_frozen_perf` scenario includes all five configurations. +For a focused DuckDB baseline, dispatch the same workflow on the PR branch: + +```sh +gh workflow run scenario-dev.yml --ref \ + -f scenario=posthog_frozen_perf_coverage_uncached +``` + +The focused scenario builds the revision being tested, deploys an isolated +warehouse, registers the same frozen files, and executes only +`pgwire_uncached`. It does not require a running local database. Both scenarios +use one warmup and four measured repetitions per query, sequential execution, +and the workflow's explicit worker resource settings. "Uncached" disables the +DuckDB external file cache; it does not guarantee cold OS or storage caches. +See [cache settings](../../tests/perf/README.md#duckdb-cache-comparison). + +Artifacts live in `perf-coverage/`, with a `-coverage` run ID suffix. The +original `perf/` and `perf-properties/` histories are preserved. Main-branch +runs publish coverage separately; PR runs retain downloadable artifacts. + +## Dataset limits and validation + +The frozen events are sampled days, not a continuous event history. Coverage +queries use populated fixture windows. In particular, retention measures a +return between sampled weeks; it is a performance workload, not a production +retention estimate. Do not interpret absent days as zero user activity. +Person enrichment uses tenant and person keys and handles duplicate person +records. Fixture checks must reject empty populations rather than silently +turning joins or funnels into cheap empty-result benchmarks. + +## Record a comparison baseline + +Keep the successful focused run URL, tested commit, catalog version/hash, +dataset version, worker resources, cache settings, and repetition counts +alongside sanitized per-query timing samples. Do not commit query result +values, fixture locations, credentials, tenant identifiers, or raw run logs. +GitHub artifacts expire, so retained timing samples must be sufficient to +recalculate each query's mean, median, and observed range. + +Compare future runs by matching intent ID, protocol, fixture, resources, and +methodology. A query or window change needs a new versioned intent ID and a +new baseline. Report per-query ratios as well as the sum; a summed ratio is +weighted toward slow queries. Four repetitions do not establish tail latency. +The uncached DuckDB baseline is measured evidence only for that configuration; +it is not a measured Trino, cached DuckDB, or Athena baseline. + +## Estimate the expanded full suite + +For each added query, use its measured uncached mean as the initial planning +estimate for the warmup and four measured executions. With twelve queries and +five configurations, the equal-speed estimate is: + +```text +additional query time = sum(uncached means) * 5 repetitions * 5 configurations +full workflow estimate = original full workflow time + additional query time +``` + +Report this explicitly as an equal-speed estimate, not a forecast of engine +performance. Include a sensitivity range for the other four configurations. +Account separately for fixture validation, setup, collection, and teardown; +the focused workflow's build/deployment time must not be multiplied by five. +Use the measured coverage phase wall time to check the sum-based estimate, +because warmups and harness overhead are not in `query_results.csv`. + +## Failure recovery + +Keep failed artifacts and query errors. An empty fixture population, SQL +compatibility failure, timeout, or platform error invalidates that query's +baseline; never record a failure as zero latency or change assertions to make +a failing platform look successful. Fix the cause and dispatch a fresh isolated +run. The workflow always tears down its stack; if cancellation interrupts +cleanup, follow the existing scenario runner cleanup runbook. Do not patch a +running benchmark or reuse a partially configured worker for the baseline. diff --git a/tests/mw-dev/run.sh b/tests/mw-dev/run.sh index 3051249ed..dec08940d 100755 --- a/tests/mw-dev/run.sh +++ b/tests/mw-dev/run.sh @@ -130,7 +130,7 @@ require_pr_identity() { } frozen_perf_scenario() { - [ "$SCENARIO_NAME" = posthog_frozen_perf ] + [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage_uncached ] } hoglake_perf_enabled() { diff --git a/tests/mw-dev/run_sh_test.go b/tests/mw-dev/run_sh_test.go index 001e4916e..5f61c4679 100644 --- a/tests/mw-dev/run_sh_test.go +++ b/tests/mw-dev/run_sh_test.go @@ -138,21 +138,27 @@ func TestTrinoDeployStartsWorkloadsWithoutScaleSubresource(t *testing.T) { } } -func TestDeployCreatesDedicatedScenarioPodIdentityForAthenaPerf(t *testing.T) { - fakes := newRunSHFakes(t) - cmd := runSHCommand(t, fakes.binDir, "deploy", - "SCENARIO_DEV_ALLOW_DUCKLING_DELETE=1", - "SCENARIO_NAME=posthog_frozen_perf", - "SCENARIO_POD_IDENTITY_ROLE=arn:aws:iam::123456789012:role/athena-perf", - ) - out, err := cmd.CombinedOutput() - if err != nil { - t.Fatalf("Athena perf deploy failed: %v\n%s", err, out) - } - calls := fakes.calls(t) - want := "aws eks create-pod-identity-association --region us-east-1 --cluster-name test-cluster --namespace duckgres-ci-pr-123 --service-account duckgres-scenario --role-arn arn:aws:iam::123456789012:role/athena-perf" - if !strings.Contains(calls, want) { - t.Fatalf("deploy did not create dedicated scenario Pod Identity; calls:\n%s", calls) +func TestDeployCreatesDedicatedScenarioPodIdentityForFrozenPerf(t *testing.T) { + for _, scenario := range []string{"posthog_frozen_perf", "posthog_frozen_perf_coverage_uncached"} { + t.Run(scenario, func(t *testing.T) { + fakes := newRunSHFakes(t) + cmd := runSHCommand(t, fakes.binDir, "deploy", + "SCENARIO_DEV_ALLOW_DUCKLING_DELETE=1", + "SCENARIO_NAME="+scenario, + "E2E_SUITE=neutral", + "SCENARIO_POD_IDENTITY_ROLE=arn:aws:iam::123456789012:role/athena-perf", + ) + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("Athena perf deploy failed: %v\n%s", err, out) + } + calls := fakes.calls(t) + want := "aws eks create-pod-identity-association --region us-east-1 --cluster-name test-cluster --namespace duckgres-ci-pr-123 --service-account duckgres-scenario --role-arn arn:aws:iam::123456789012:role/athena-perf" + if !strings.Contains(calls, want) { + t.Fatalf("deploy did not create dedicated scenario Pod Identity; calls:\n%s", calls) + } + + }) } } diff --git a/tests/mw-dev/scenario/coverage_test.go b/tests/mw-dev/scenario/coverage_test.go new file mode 100644 index 000000000..5cdf51967 --- /dev/null +++ b/tests/mw-dev/scenario/coverage_test.go @@ -0,0 +1,76 @@ +package scenario + +import ( + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/posthog/duckgres/tests/mw-dev/scenario/core" +) + +func TestCoverageScenariosPreserveBaselineAndFocusedIsolation(t *testing.T) { + for _, focused := range []bool{false, true} { + name := "posthog_frozen_perf" + if focused { + name += "_coverage_uncached" + } + t.Run(name, func(t *testing.T) { + scenario, err := core.LoadScenario(filepath.Join("scenarios", name+".yaml")) + if err != nil { + t.Fatal(err) + } + var coverage *core.Step + var validation *core.Step + for i := range scenario.Steps { + step := &scenario.Steps[i] + if step.ID == "validate_coverage" { + validation = step + } + if step.ID == "coverage_queries" { + coverage = step + } + if focused && (strings.Contains(step.Type, "trino") || step.Type == "setup_hoglake" || step.Type == "properties_comparison") { + t.Fatalf("focused run includes %s", step.Type) + } + } + if validation == nil || validation.With["file"] != "../sql/validate_posthog_coverage.sql" { + t.Fatal("missing coverage fixture validation") + } + if coverage == nil { + t.Fatal("missing coverage phase") + } + if !reflect.DeepEqual(coverage.DependsOn, []string{"validate_coverage"}) { + t.Fatal("coverage must depend on fixture validation") + } + if coverage.With["catalog_file"] != "../../../perf/queries/ducklake_posthog_coverage.yaml" { + t.Fatal("wrong coverage catalog") + } + if coverage.With["output_subdir"] != "perf-coverage" || coverage.With["run_id"] != "${run_id}-coverage" || coverage.With["nightly_run_id"] != "${run_id}" { + t.Fatal("coverage must have separate artifacts and shared nightly identity") + } + if coverage.With["suite"] != "coverage" { + t.Fatal("coverage must have its own comparison suite") + } + if coverage.With["fail_on_query_errors"] != true { + t.Fatal("coverage errors must fail") + } + if focused { + if !reflect.DeepEqual(coverage.With["targets"], []any{"pgwire_uncached"}) { + t.Fatalf("focused targets = %v", coverage.With["targets"]) + } + for _, env := range scenario.RequiredEnv { + if strings.Contains(env, "TRINO") || strings.Contains(env, "ATHENA") || strings.Contains(env, "HOGLAKE") { + t.Fatalf("unnecessary focused requirement %s", env) + } + } + request := scenario.Steps[0].With["request"].(map[string]any) + if _, ok := request["trino"]; ok { + t.Fatal("focused run provisions Trino") + } + } else { + assertPerfTargetsPGWireTrinoAndAthena(t, *coverage) + } + }) + } +} diff --git a/tests/mw-dev/scenario/runner_test.go b/tests/mw-dev/scenario/runner_test.go index 74a34a768..d4ff98916 100644 --- a/tests/mw-dev/scenario/runner_test.go +++ b/tests/mw-dev/scenario/runner_test.go @@ -182,6 +182,7 @@ func TestFrozenSuccessScenariosUseIsolatedStackWarehouseIdentity(t *testing.T) { for _, scenarioFile := range []string{ "posthog_frozen_metadata.yaml", "posthog_frozen_perf.yaml", + "posthog_frozen_perf_coverage_uncached.yaml", "posthog_frozen_dbt.yaml", "fast-suite.yaml", "full-suite.yaml", diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml index 3dcbce2f3..ac76c5860 100644 --- a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml @@ -146,6 +146,43 @@ steps: worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST} worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST} + - id: validate_coverage + type: sql + depends_on: [perf_queries] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/validate_posthog_coverage.sql + max_attempts: 1 + + - id: coverage_queries + type: perf_queries + depends_on: [validate_coverage] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + catalog_file: ../../../perf/queries/ducklake_posthog_coverage.yaml + targets: [pgwire_uncached, pgwire_cached, trino, trino_cached, athena] + trino_hoglake_catalog: ${env:DUCKGRES_SCENARIO_ORG_ID}-frozen + trino_ca_cert_file: ${env:DUCKGRES_SCENARIO_TRINO_CA_CERT} + trino_startup_timeout: 2m + trino_startup_poll_interval: 2s + athena_region: ${env:DUCKGRES_SCENARIO_ATHENA_REGION} + athena_workgroup: ${env:DUCKGRES_SCENARIO_ATHENA_WORKGROUP} + athena_catalog: AwsDataCatalog + athena_database: ${env:DUCKGRES_SCENARIO_ATHENA_DATABASE} + athena_output_location: ${env:DUCKGRES_SCENARIO_ATHENA_RESULTS_S3_URI} + athena_poll_interval: 500ms + athena_query_timeout: 30m + run_id: ${run_id}-coverage + nightly_run_id: ${run_id} + output_subdir: perf-coverage + suite: coverage + dataset_version: *nightly_dataset + fail_on_query_errors: true + worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST} + worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST} + # Managed Hoglake retains tenant ownership and rejects public deprovision. # The workflow's always-run isolated-stack teardown owns cleanup of the # Duckling, namespace, metadata store and dedicated managed storage prefix. diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_uncached.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_uncached.yaml new file mode 100644 index 000000000..a876ae6dc --- /dev/null +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_uncached.yaml @@ -0,0 +1,84 @@ +name: posthog-frozen-perf-coverage-uncached +run_id_prefix: scenario-frozen-coverage-uncached +required_env: + - DUCKGRES_SCENARIO_ORG_ID + - DUCKGRES_SCENARIO_FROZEN_S3_URI + - DUCKGRES_K8S_WORKER_CPU_REQUEST + - DUCKGRES_K8S_WORKER_MEMORY_REQUEST +steps: + - id: provision + type: provision_warehouse + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + request: + database_name: scenario-frozen-coverage-uncached-${run_id_token} + team_id: 1 + metadata_store: + type: cnpg-shard + ducklake: + enabled: true + data_store: + type: s3bucket + + - id: wait_ready + type: wait_warehouse_ready + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + timeout: 15m + poll_interval: 10s + + - id: setup_frozen_views + type: sql + depends_on: [wait_ready] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/setup_frozen_views.sql + max_attempts: 12 + retry_interval: 10s + + - id: setup_posthog_tables + type: sql + depends_on: [setup_frozen_views] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/setup_posthog_tables.sql + max_attempts: 3 + retry_interval: 10s + + - id: validate_posthog_tables + type: sql + depends_on: [setup_posthog_tables] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/validate_posthog_tables.sql + max_attempts: 3 + retry_interval: 10s + + - id: validate_coverage + type: sql + depends_on: [validate_posthog_tables] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/validate_posthog_coverage.sql + max_attempts: 1 + + - id: coverage_queries + type: perf_queries + depends_on: [validate_coverage] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + catalog_file: ../../../perf/queries/ducklake_posthog_coverage.yaml + targets: [pgwire_uncached] + run_id: ${run_id}-coverage + nightly_run_id: ${run_id} + output_subdir: perf-coverage + suite: coverage + dataset_version: posthog-file-views-v1 + fail_on_query_errors: true + worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST} + worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST} diff --git a/tests/mw-dev/scenario/sql/validate_posthog_coverage.sql b/tests/mw-dev/scenario/sql/validate_posthog_coverage.sql new file mode 100644 index 000000000..43506ec58 --- /dev/null +++ b/tests/mw-dev/scenario/sql/validate_posthog_coverage.sql @@ -0,0 +1,56 @@ +-- Fail rather than record misleading fast results for an empty workload. +-- Checks expose only fixed assertion labels, never fixture identifiers. +SELECT CASE WHEN EXISTS ( + SELECT 1 FROM posthog.events + WHERE timestamp >= TIMESTAMPTZ '2026-03-17 00:00:00+00' + AND timestamp < TIMESTAMPTZ '2026-03-24 00:00:00+00' + AND event = '$pageview' AND person_id IS NOT NULL +) THEN 1 ELSE error('coverage fixture has no first-week pageview actors') END; + +SELECT CASE WHEN EXISTS ( + SELECT 1 FROM posthog.events + WHERE timestamp >= TIMESTAMPTZ '2026-03-17 00:00:00+00' + AND timestamp < TIMESTAMPTZ '2026-03-24 00:00:00+00' + AND event = '$autocapture' AND person_id IS NOT NULL +) THEN 1 ELSE error('coverage fixture has no first-week autocapture actors') END; + +SELECT CASE WHEN EXISTS ( + SELECT 1 FROM posthog.events + WHERE timestamp >= TIMESTAMPTZ '2026-03-24 00:00:00+00' + AND timestamp < TIMESTAMPTZ '2026-03-31 00:00:00+00' + AND person_id IS NOT NULL +) THEN 1 ELSE error('coverage fixture has no return-week actors') END; + +SELECT CASE WHEN EXISTS ( + SELECT 1 FROM posthog.events e + INNER JOIN posthog.persons p ON e.team_id = p.team_id AND e.person_id = p.id + WHERE e.timestamp >= TIMESTAMPTZ '2026-03-17 00:00:00+00' + AND e.timestamp < TIMESTAMPTZ '2026-03-24 00:00:00+00' +) THEN 1 ELSE error('coverage fixture has no first-week event/person overlap') END; + +WITH starts AS ( + SELECT team_id, person_id, MIN(timestamp) AS started_at + FROM posthog.events + WHERE timestamp >= TIMESTAMPTZ '2026-03-17 00:00:00+00' + AND timestamp < TIMESTAMPTZ '2026-03-24 00:00:00+00' + AND event = '$pageview' AND person_id IS NOT NULL + GROUP BY team_id, person_id +) +SELECT CASE WHEN EXISTS ( + SELECT 1 FROM starts s + INNER JOIN posthog.events e ON s.team_id = e.team_id AND s.person_id = e.person_id + WHERE e.event = '$autocapture' AND e.timestamp > s.started_at + AND e.timestamp <= s.started_at + INTERVAL '7' DAY + AND e.timestamp >= TIMESTAMPTZ '2026-03-17 00:00:00+00' + AND e.timestamp < TIMESTAMPTZ '2026-03-31 00:00:00+00' +) THEN 1 ELSE error('coverage fixture has no ordered funnel completions') END; + +SELECT CASE WHEN EXISTS ( + SELECT 1 FROM posthog.events first_week + INNER JOIN posthog.events second_week + ON first_week.team_id = second_week.team_id AND first_week.person_id = second_week.person_id + WHERE first_week.timestamp >= TIMESTAMPTZ '2026-03-17 00:00:00+00' + AND first_week.timestamp < TIMESTAMPTZ '2026-03-24 00:00:00+00' + AND second_week.timestamp >= TIMESTAMPTZ '2026-03-24 00:00:00+00' + AND second_week.timestamp < TIMESTAMPTZ '2026-03-31 00:00:00+00' +) THEN 1 ELSE error('coverage fixture has no retained actors across weeks') END; diff --git a/tests/perf/README.md b/tests/perf/README.md index 90028f147..bc3be3134 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -2,6 +2,12 @@ This package contains the golden-query performance harness. +The frozen suite also includes twelve additional workload shapes in +`queries/ducklake_posthog_coverage.yaml`. They retain separate `perf-coverage/` +artifacts and versioned intent IDs. For the focused uncached DuckDB run, +baseline recording, runtime estimates, and failure recovery, see the +[coverage runbook](../../docs/runbooks/perf-coverage.md). + ## Protocol Drivers Catalogs may target `pgwire`, `pgwire_uncached`, `pgwire_cached`, `trino`, `trino_cached`, @@ -367,7 +373,7 @@ Both suites of one nightly publish under the same dataset version (`posthog-file-views-v1`, shared through a YAML anchor in the scenario), with three columns on `runs` and `query_results`: -- `suite` (`tables` | `properties`, a closed set in `core`, validated when the +- `suite` (`tables` | `properties` | `coverage`, a closed set in `core`, validated when the step starts so a typo fails before hours of measurement); - `nightly_run_id`, the table-suite run's ID, which pairs a nightly's suites explicitly (a standalone run is its own nightly); diff --git a/tests/perf/core/catalog_test.go b/tests/perf/core/catalog_test.go index 6b6875ec0..a1f190df6 100644 --- a/tests/perf/core/catalog_test.go +++ b/tests/perf/core/catalog_test.go @@ -25,7 +25,7 @@ func TestCheckedInCatalogsLoad(t *testing.T) { t.Fatalf("LoadCatalog(%s): %v", path, err) } wantTargets := []Protocol{ProtocolPGWire} - if filepath.Base(path) == "ducklake_posthog_tables.yaml" { + if filepath.Base(path) == "ducklake_posthog_tables.yaml" || filepath.Base(path) == "ducklake_posthog_coverage.yaml" { wantTargets = []Protocol{ProtocolPGWireUncached, ProtocolPGWireCached, ProtocolTrino, ProtocolTrinoCached, ProtocolAthena} } if !reflect.DeepEqual(catalog.Targets, wantTargets) { diff --git a/tests/perf/core/coverage_test.go b/tests/perf/core/coverage_test.go new file mode 100644 index 000000000..5679a0418 --- /dev/null +++ b/tests/perf/core/coverage_test.go @@ -0,0 +1,128 @@ +package core + +import ( + "database/sql" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + _ "github.com/duckdb/duckdb-go/v2" +) + +// Exercise the actual benchmark SQL against a small fixture with out-of-order +// funnel events, person snapshot duplicates, tenant collisions and boundary dates. +func TestCoverageCatalogSemantics(t *testing.T) { + c, err := LoadCatalog(filepath.Join("..", "queries", "ducklake_posthog_coverage.yaml")) + if err != nil { + t.Fatal(err) + } + if len(c.Queries) != 36 { + t.Fatalf("got %d variants, want 12 complete triples", len(c.Queries)) + } + db, err := sql.Open("duckdb", "") + if err != nil { + t.Fatal(err) + } + defer func() { + if err := db.Close(); err != nil { + t.Errorf("close database: %v", err) + } + }() + _, err = db.Exec(`CREATE SCHEMA posthog; + CREATE TABLE posthog.events(team_id BIGINT, person_id VARCHAR, event VARCHAR, timestamp TIMESTAMPTZ, uuid VARCHAR); + CREATE TABLE posthog.persons(team_id BIGINT, id VARCHAR, is_identified BOOLEAN); + INSERT INTO posthog.persons VALUES (1,'a',true),(1,'a',true),(1,'b',false),(2,'a',false); + INSERT INTO posthog.events VALUES + (1,'a','$autocapture','2026-03-17 00:00:00+00','1'), + (1,'a','$pageview','2026-03-17 00:10:00+00','2'), + (1,'a','$autocapture','2026-03-17 00:20:00+00','3'), + (1,'a','$pageview','2026-03-26 00:00:00+00','4'), + (1,'b','$pageview','2026-03-18 00:00:00+00','5'), + (2,'a','$pageview','2026-03-18 00:00:00+00','6'), + (1,NULL,'other','2026-03-18 00:00:00+00','7'), + (1,'c','$autocapture','2026-03-17 00:00:00+00','8'), + (1,'c','$pageview','2026-03-17 00:40:00+00','9');`) + if err != nil { + t.Fatal(err) + } + validation, err := os.ReadFile(filepath.Join("..", "..", "mw-dev", "scenario", "sql", "validate_posthog_coverage.sql")) + if err != nil { + t.Fatal(err) + } + if _, err := db.Exec(string(validation)); err != nil { + t.Fatalf("valid fixture rejected: %v", err) + } + defer func() { + if _, err := db.Exec("DELETE FROM posthog.persons"); err != nil { + t.Fatal(err) + } + if _, err := db.Exec(string(validation)); err == nil || !strings.Contains(err.Error(), "overlap") { + t.Fatalf("missing join population must fail: %v", err) + } + }() + + expected := map[string][]string{ + "q_coverage_daily_uniques_v1": {"2026-03-17 2", "2026-03-18 2", "2026-03-26 1"}, + "q_coverage_event_trends_v1": {"2026-03-17 $autocapture 3", "2026-03-17 $pageview 2", "2026-03-18 $pageview 2", "2026-03-26 $pageview 1"}, + "q_coverage_person_enrichment_v1": {"false 2", "true 3"}, + "q_coverage_person_activity_join_v1": {"3 5"}, + "q_coverage_recent_events_v1": {"7 other ", "6 $pageview a", "5 $pageview b", "9 $pageview c", "3 $autocapture a", "2 $pageview a", "8 $autocapture c", "1 $autocapture a"}, + "q_coverage_active_actors_v1": {"1 a 3", "1 c 2", "1 b 1", "2 a 1"}, + "q_coverage_ordered_funnel_v1": {"4 1"}, + "q_coverage_weekly_retention_v1": {"4 1"}, + "q_coverage_repeat_activity_cohort_v1": {"1"}, + "q_coverage_exclusion_cohort_v1": {"2"}, + "q_coverage_daily_person_model_v1": {"4 7"}, + "q_coverage_session_model_v1": {"5 7"}, + } + for i := 0; i < len(c.Queries); i += 3 { + q := c.Queries[i] + base := q.QueryID[:len(q.QueryID)-len("__ducklake_table")] + t.Run(base, func(t *testing.T) { + want, ok := expected[base] + if !ok { + t.Fatalf("missing semantic expectation for %s", base) + } + rows, err := db.Query(q.PGWireSQL) + if err != nil { + t.Fatal(err) + } + defer func() { + if err := rows.Close(); err != nil { + t.Errorf("close rows: %v", err) + } + }() + cols, err := rows.Columns() + if err != nil { + t.Fatal(err) + } + var got []string + for rows.Next() { + values := make([]any, len(cols)) + pointers := make([]any, len(cols)) + for j := range values { + pointers[j] = &values[j] + } + if err := rows.Scan(pointers...); err != nil { + t.Fatal(err) + } + line := "" + for j, v := range values { + if j > 0 { + line += " " + } + line += fmt.Sprint(v) + } + got = append(got, line) + } + if err := rows.Err(); err != nil { + t.Fatal(err) + } + if fmt.Sprint(got) != fmt.Sprint(want) { + t.Fatalf("got %v, want %v", got, want) + } + }) + } +} diff --git a/tests/perf/core/suite_test.go b/tests/perf/core/suite_test.go new file mode 100644 index 000000000..efa2ccc7d --- /dev/null +++ b/tests/perf/core/suite_test.go @@ -0,0 +1,16 @@ +package core + +import "testing" + +func TestValidateSuite(t *testing.T) { + for _, suite := range []string{"tables", "properties", "coverage"} { + if err := ValidateSuite(suite); err != nil { + t.Errorf("suite %q: %v", suite, err) + } + } + for _, suite := range []string{"", "covergae", "Tables"} { + if err := ValidateSuite(suite); err == nil { + t.Errorf("accepted unknown suite %q", suite) + } + } +} diff --git a/tests/perf/core/types.go b/tests/perf/core/types.go index b32024beb..10d2d2c47 100644 --- a/tests/perf/core/types.go +++ b/tests/perf/core/types.go @@ -157,13 +157,14 @@ type QueryResult struct { const ( SuiteTables = "tables" SuiteProperties = "properties" + SuiteCoverage = "coverage" ) // ValidateSuite rejects a suite outside the closed set, so a typo fails when a // run starts rather than after hours of measurement at publish time. func ValidateSuite(suite string) error { - if suite != SuiteTables && suite != SuiteProperties { - return fmt.Errorf("unknown suite %q (want %q or %q)", suite, SuiteTables, SuiteProperties) + if suite != SuiteTables && suite != SuiteProperties && suite != SuiteCoverage { + return fmt.Errorf("unknown suite %q (want %q, %q, or %q)", suite, SuiteTables, SuiteProperties, SuiteCoverage) } return nil } @@ -171,8 +172,8 @@ func ValidateSuite(suite string) error { type RunSummary struct { RunID string `json:"run_id"` DatasetVersion string `json:"dataset_version"` - // Suite names the comparison family inside a dataset ("tables" or - // "properties"); one nightly publishes several suites under one dataset. + // Suite names the comparison family inside a dataset (tables, properties, + // or coverage); one nightly publishes several suites under one dataset. Suite string `json:"suite,omitempty"` // FixtureVersion identifies the exact fixture a suite measured when it is // finer-grained than the dataset (e.g. the properties object inventory). diff --git a/tests/perf/publisher/publisher_test.go b/tests/perf/publisher/publisher_test.go index 3b422e6f0..0f639285d 100644 --- a/tests/perf/publisher/publisher_test.go +++ b/tests/perf/publisher/publisher_test.go @@ -203,45 +203,50 @@ func TestPublishArtifactsBootstrapsAndReplacesRunData(t *testing.T) { } func TestPublishArtifactsWritesSuiteAndFixtureVersion(t *testing.T) { - runDir := t.TempDir() - writeCustomSummaryFile(t, runDir, map[string]any{ - "run_id": "scenario-dev-posthog-frozen-perf-1-properties", - "dataset_version": "posthog-file-views-v1", - "suite": "properties", - "fixture_version": "properties-sha256-abc", - "nightly_run_id": "scenario-dev-posthog-frozen-perf-1", - "started_at": "2026-03-11T23:43:03Z", - "finished_at": "2026-03-11T23:43:14Z", - }) - writeFixtureCSVFile(t, runDir) - artifacts, err := loadArtifacts(runDir) - if err != nil { - t.Fatalf("loadArtifacts returned error: %v", err) - } - db := &fakeDB{tx: &fakeTx{}} - if err := publishArtifacts(context.Background(), Config{Schema: "duckgres_perf"}, db, artifacts); err != nil { - t.Fatalf("publishArtifacts returned error: %v", err) - } - runInsert := db.tx.execs[2] - if runInsert.args[1] != "posthog-file-views-v1" || runInsert.args[8] != core.SuiteProperties { - t.Fatalf("run dataset/suite = %#v/%#v", runInsert.args[1], runInsert.args[8]) - } - if got, ok := runInsert.args[9].(*string); !ok || got == nil || *got != "properties-sha256-abc" { - t.Fatalf("run fixture_version = %#v", runInsert.args[9]) - } - if got := runInsert.args[10]; got != "scenario-dev-posthog-frozen-perf-1" { - t.Fatalf("run nightly_run_id = %#v", got) - } - for _, result := range db.tx.execs[3:] { - if result.args[11] != "posthog-file-views-v1" || result.args[15] != core.SuiteProperties { - t.Fatalf("result dataset/suite = %#v/%#v", result.args[11], result.args[15]) - } - if got, ok := result.args[16].(*string); !ok || got == nil || *got != "properties-sha256-abc" { - t.Fatalf("result fixture_version = %#v", result.args[16]) - } - if got := result.args[17]; got != "scenario-dev-posthog-frozen-perf-1" { - t.Fatalf("result nightly_run_id = %#v", got) - } + for _, suite := range []string{"properties", "coverage"} { + t.Run(suite, func(t *testing.T) { + runDir := t.TempDir() + writeCustomSummaryFile(t, runDir, map[string]any{ + "run_id": "scenario-dev-posthog-frozen-perf-1-" + suite, + "dataset_version": "posthog-file-views-v1", + "suite": suite, + "fixture_version": suite + "-sha256-abc", + "nightly_run_id": "scenario-dev-posthog-frozen-perf-1", + "started_at": "2026-03-11T23:43:03Z", + "finished_at": "2026-03-11T23:43:14Z", + }) + writeFixtureCSVFile(t, runDir) + artifacts, err := loadArtifacts(runDir) + if err != nil { + t.Fatalf("loadArtifacts returned error: %v", err) + } + db := &fakeDB{tx: &fakeTx{}} + if err := publishArtifacts(context.Background(), Config{Schema: "duckgres_perf"}, db, artifacts); err != nil { + t.Fatalf("publishArtifacts returned error: %v", err) + } + runInsert := db.tx.execs[2] + if runInsert.args[1] != "posthog-file-views-v1" || runInsert.args[8] != suite { + t.Fatalf("run dataset/suite = %#v/%#v", runInsert.args[1], runInsert.args[8]) + } + if got, ok := runInsert.args[9].(*string); !ok || got == nil || *got != suite+"-sha256-abc" { + t.Fatalf("run fixture_version = %#v", runInsert.args[9]) + } + if got := runInsert.args[10]; got != "scenario-dev-posthog-frozen-perf-1" { + t.Fatalf("run nightly_run_id = %#v", got) + } + for _, result := range db.tx.execs[3:] { + if result.args[11] != "posthog-file-views-v1" || result.args[15] != suite { + t.Fatalf("result dataset/suite = %#v/%#v", result.args[11], result.args[15]) + } + if got, ok := result.args[16].(*string); !ok || got == nil || *got != suite+"-sha256-abc" { + t.Fatalf("result fixture_version = %#v", result.args[16]) + } + if got := result.args[17]; got != "scenario-dev-posthog-frozen-perf-1" { + t.Fatalf("result nightly_run_id = %#v", got) + } + } + + }) } } diff --git a/tests/perf/queries/ducklake_posthog_coverage.yaml b/tests/perf/queries/ducklake_posthog_coverage.yaml new file mode 100644 index 000000000..028c6bc70 --- /dev/null +++ b/tests/perf/queries/ducklake_posthog_coverage.yaml @@ -0,0 +1,175 @@ +name: posthog-frozen-coverage-v1 +description: Bounded analytical workload coverage on the frozen events/persons snapshot; background shapes measure SELECT computation, not materialization writes. +seed: 42 +dataset_scale: 1 +targets: [pgwire_uncached, pgwire_cached, trino, trino_cached, athena] +warmup_iterations: 1 +measure_iterations: 4 +relation_variants: + ducklake_table: {events: posthog.events, persons: posthog.persons} + hoglake_table: {events: posthog.events, persons: posthog.persons} + athena_external: {events: events, persons: persons} + +# Fixed UTC windows make comparisons reproducible. Persons are deduplicated by +# tenant and ID before joins because the frozen export contains snapshot rows. +# Identification means observed identified in the snapshot, not as-of event time. +# Funnel: first pageview followed by autocapture within seven days. +# Retention: activity in week one and again in week two (not new-user retention). +# Session model: new session after an inactivity gap strictly greater than 30m. +# Verify date/event/join population before treating a run as representative. +paired_queries: + - query_id_base: q_coverage_daily_uniques_v1 + intent_id: intent_coverage_daily_uniques_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, distinct, trends] + params: {} + sql_template: | + SELECT CAST(day AS VARCHAR) AS day, COUNT(*) AS persons FROM ( + SELECT CAST("timestamp" AT TIME ZONE 'UTC' AS DATE) AS day, team_id, person_id + FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-27 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND person_id IS NOT NULL AND event = '$pageview' + GROUP BY 1, 2, 3 + ) active GROUP BY day ORDER BY day + + - query_id_base: q_coverage_event_trends_v1 + intent_id: intent_coverage_event_trends_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, aggregate, trends] + params: {} + sql_template: | + SELECT CAST(CAST("timestamp" AT TIME ZONE 'UTC' AS DATE) AS VARCHAR) AS day, event, COUNT(*) AS events FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-27 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND event IN ('$pageview', '$autocapture') GROUP BY 1, 2 ORDER BY 1, 2 + + - query_id_base: q_coverage_person_enrichment_v1 + intent_id: intent_coverage_person_enrichment_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, join] + params: {} + sql_template: | + SELECT * FROM ( + WITH deduplicated_persons AS ( + SELECT team_id, id, MAX(CASE WHEN is_identified THEN 1 ELSE 0 END) AS identified + FROM {{ relation "persons" }} GROUP BY team_id, id + ) + SELECT p.identified = 1 AS is_identified, COUNT(*) AS events + FROM (SELECT * FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE)) e JOIN deduplicated_persons p ON e.team_id = p.team_id AND e.person_id = p.id + GROUP BY p.identified ORDER BY p.identified + ) coverage_query ORDER BY 1 + + - query_id_base: q_coverage_person_activity_join_v1 + intent_id: intent_coverage_person_activity_join_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, join] + params: {} + sql_template: | + SELECT * FROM ( + WITH activity AS (SELECT team_id, person_id, COUNT(*) AS events FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND person_id IS NOT NULL GROUP BY team_id, person_id), deduplicated_persons AS (SELECT DISTINCT team_id, id FROM {{ relation "persons" }}) + SELECT COUNT(*) AS matched_persons, COALESCE(SUM(a.events), 0) AS events + FROM activity a JOIN deduplicated_persons p ON a.team_id = p.team_id AND a.person_id = p.id + ) coverage_query + + - query_id_base: q_coverage_recent_events_v1 + intent_id: intent_coverage_recent_events_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, retrieval] + params: {} + sql_template: | + SELECT uuid, event, person_id FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + ORDER BY "timestamp" DESC, uuid DESC LIMIT 100 + + - query_id_base: q_coverage_active_actors_v1 + intent_id: intent_coverage_active_actors_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, retrieval] + params: {} + sql_template: | + SELECT team_id, person_id, COUNT(*) AS events FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND person_id IS NOT NULL GROUP BY team_id, person_id ORDER BY events DESC, team_id, person_id LIMIT 100 + + - query_id_base: q_coverage_ordered_funnel_v1 + intent_id: intent_coverage_ordered_funnel_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, funnel] + params: {} + sql_template: | + SELECT * FROM ( + WITH starts AS ( + SELECT team_id, person_id, MIN("timestamp") AS started_at FROM {{ relation "events" }} + WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND event = '$pageview' AND person_id IS NOT NULL GROUP BY team_id, person_id + ), completions AS ( + SELECT s.team_id, s.person_id, MIN(e."timestamp") AS completed_at + FROM starts s JOIN {{ relation "events" }} e ON s.team_id = e.team_id AND s.person_id = e.person_id + WHERE e.event = '$autocapture' AND e."timestamp" > s.started_at + AND e."timestamp" <= s.started_at + INTERVAL '7' DAY + AND e."timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND e."timestamp" < CAST('2026-03-31 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + GROUP BY s.team_id, s.person_id + ) + SELECT COUNT(*) AS started_persons, COUNT(c.person_id) AS completed_persons + FROM starts s LEFT JOIN completions c ON s.team_id = c.team_id AND s.person_id = c.person_id + ) coverage_query + + - query_id_base: q_coverage_weekly_retention_v1 + intent_id: intent_coverage_weekly_retention_v1 + tags: [nightly, frozen, posthog, coverage, paired, foreground, retention] + params: {} + sql_template: | + SELECT * FROM ( + WITH activity AS ( + SELECT team_id, person_id, + MAX(CASE WHEN "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) THEN 1 ELSE 0 END) AS first_week, + MAX(CASE WHEN "timestamp" >= CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) THEN 1 ELSE 0 END) AS second_week + FROM {{ relation "events" }} + WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-31 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND person_id IS NOT NULL GROUP BY team_id, person_id + ) + SELECT COALESCE(SUM(first_week), 0) AS cohort_persons, + COALESCE(SUM(CASE WHEN first_week = 1 AND second_week = 1 THEN 1 ELSE 0 END), 0) AS retained_persons FROM activity + ) coverage_query + + - query_id_base: q_coverage_repeat_activity_cohort_v1 + intent_id: intent_coverage_repeat_activity_cohort_v1 + tags: [nightly, frozen, posthog, coverage, paired, background, cohort] + params: {} + sql_template: | + SELECT COUNT(*) AS cohort_persons FROM (SELECT team_id, person_id, COUNT(*) AS events FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND person_id IS NOT NULL GROUP BY team_id, person_id HAVING COUNT(*) >= 3) members + + - query_id_base: q_coverage_exclusion_cohort_v1 + intent_id: intent_coverage_exclusion_cohort_v1 + tags: [nightly, frozen, posthog, coverage, paired, background, cohort] + params: {} + sql_template: | + SELECT COUNT(*) AS cohort_persons FROM ( + SELECT team_id, person_id FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND person_id IS NOT NULL + GROUP BY team_id, person_id + HAVING SUM(CASE WHEN event = '$pageview' THEN 1 ELSE 0 END) > 0 + AND SUM(CASE WHEN event = '$autocapture' THEN 1 ELSE 0 END) = 0 + ) members + + - query_id_base: q_coverage_daily_person_model_v1 + intent_id: intent_coverage_daily_person_model_v1 + tags: [nightly, frozen, posthog, coverage, paired, background, modeling] + params: {} + sql_template: | + SELECT COUNT(*) AS person_days, COALESCE(SUM(events), 0) AS events FROM ( + SELECT team_id, person_id, CAST("timestamp" AT TIME ZONE 'UTC' AS DATE) AS day, COUNT(*) AS events + FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND person_id IS NOT NULL GROUP BY 1, 2, 3 + ) daily_activity + + - query_id_base: q_coverage_session_model_v1 + intent_id: intent_coverage_session_model_v1 + tags: [nightly, frozen, posthog, coverage, paired, background, modeling, window] + params: {} + sql_template: | + SELECT * FROM ( + WITH ordered_events AS ( + SELECT team_id, person_id, "timestamp", + LAG("timestamp") OVER (PARTITION BY team_id, person_id ORDER BY "timestamp", uuid) AS previous_at + FROM {{ relation "events" }} WHERE "timestamp" >= CAST('2026-03-17 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) + AND "timestamp" < CAST('2026-03-24 00:00:00+00:00' AS TIMESTAMP WITH TIME ZONE) AND person_id IS NOT NULL + ) + SELECT COALESCE(SUM(CASE WHEN previous_at IS NULL OR "timestamp" > previous_at + INTERVAL '30' MINUTE THEN 1 ELSE 0 END), 0) AS sessions, + COUNT(*) AS events FROM ordered_events + ) coverage_query + From 9c9f5f305f87b7f0d41c0b54d64e5648aeb8ef83 Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 12:11:56 -0400 Subject: [PATCH 02/13] fix(scenario): bound generated job names for coverage runs --- tests/mw-dev/run.sh | 4 +++- tests/mw-dev/run_sh_test.go | 28 ++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/tests/mw-dev/run.sh b/tests/mw-dev/run.sh index dec08940d..307576e20 100755 --- a/tests/mw-dev/run.sh +++ b/tests/mw-dev/run.sh @@ -773,7 +773,9 @@ scenario_name_for_file() { scenario_job_name() { local name="$1" run_hash run_hash="$(printf '%s' "${DUCKGRES_SCENARIO_RUN_ID:?DUCKGRES_SCENARIO_RUN_ID is required}" | cksum | awk '{print $1}')" - printf 'duckgres-scenario-%s-%s\n' "$name" "$run_hash" | tr '_' '-' + # Kubernetes copies the Job name into a label, whose limit is 63 bytes. + # Reserve 18 for the prefix, one separator, and ten for the cksum value. + printf 'duckgres-scenario-%s-%s\n' "${name:0:34}" "$run_hash" | tr '_' '-' } cmd_test_scenario() { diff --git a/tests/mw-dev/run_sh_test.go b/tests/mw-dev/run_sh_test.go index 5f61c4679..f8898286a 100644 --- a/tests/mw-dev/run_sh_test.go +++ b/tests/mw-dev/run_sh_test.go @@ -489,6 +489,34 @@ func TestScenarioRunsSelectedScenarioAgainstIsolatedStack(t *testing.T) { } } +func TestScenarioLongNameProducesValidJobLabel(t *testing.T) { + const scenario = "posthog_frozen_perf_coverage_uncached" + fakes := newRunSHFakes(t) + cmd := runSHCommand(t, fakes.binDir, "test-scenario", + "SCENARIO_RUNNER_IMAGE=example.invalid/duckgres:scenario", "SCENARIO_NAME="+scenario) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("scenario failed: %v\n%s", err, out) + } + match := regexp.MustCompile(`kind: Job\nmetadata:\n name: ([^\n]+)`).FindStringSubmatch(fakes.calls(t)) + if len(match) != 2 { + t.Fatal("missing Job name") + } + name := match[1] + if len(name) > 63 || !regexp.MustCompile(`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`).MatchString(name) { + t.Fatalf("invalid Job label %q (%d bytes)", name, len(name)) + } + runID := "scenario-dev-" + strings.ReplaceAll(scenario, "_", "-") + "-123" + checksum := exec.Command("cksum") + checksum.Stdin = strings.NewReader(runID) + out, err := checksum.Output() + if err != nil { + t.Fatal(err) + } + if !strings.HasSuffix(name, "-"+strings.Fields(string(out))[0]) { + t.Fatalf("Job name %q lost run ID checksum", name) + } +} + func TestScenarioStopsJobWithoutCollectingArtifactsWhenContainerTerminationCannotBeConfirmed(t *testing.T) { fakes := newRunSHFakes(t) From 2ab291e88f6c986d044a9eea274285349cf5381e Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 12:56:36 -0400 Subject: [PATCH 03/13] test(perf): run expanded coverage in separate target dispatches --- .github/workflows/scenario-dev.yml | 13 +- docs/runbooks/perf-coverage.md | 49 +++++-- tests/mw-dev/run.sh | 17 ++- tests/mw-dev/run_sh_test.go | 23 +++ tests/mw-dev/scenario/coverage_test.go | 82 ++++++++++- .../scenarios/posthog_frozen_perf.yaml | 37 ----- .../posthog_frozen_perf_coverage.yaml | 96 +++++++++++++ .../posthog_frozen_perf_coverage_trino.yaml | 131 ++++++++++++++++++ tests/perf/README.md | 2 +- 9 files changed, 390 insertions(+), 60 deletions(-) create mode 100644 tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage.yaml create mode 100644 tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_trino.yaml diff --git a/.github/workflows/scenario-dev.yml b/.github/workflows/scenario-dev.yml index e23a0987a..4dec18050 100644 --- a/.github/workflows/scenario-dev.yml +++ b/.github/workflows/scenario-dev.yml @@ -8,6 +8,12 @@ on: required: true default: full-suite type: string + coverage_target: + description: Target for posthog_frozen_perf_coverage; dispatch once per configuration + required: false + default: pgwire_uncached + type: choice + options: [pgwire_uncached, pgwire_cached, trino, trino_cached, athena] properties_s3_uri: description: Properties Parquet S3 prefix override; blank uses the existing Athena table location required: false @@ -82,7 +88,8 @@ jobs: # Frozen perf uses a namespace-local Trino cell. The dedicated role is # consumed only when E2E_SUITE selects that isolated deployment. TRINO_POD_IDENTITY_ROLE: ${{ secrets.MW_DEV_TRINO_POD_IDENTITY_ROLE }} - E2E_SUITE: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf') && 'trino' || 'neutral' }} + E2E_SUITE: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_coverage' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) && 'trino' || 'neutral' }} + DUCKGRES_SCENARIO_COVERAGE_TARGET: ${{ inputs.coverage_target || 'pgwire_uncached' }} PR_NUMBER: ${{ github.run_id }} NAMESPACE: duckgres-ci-pr-${{ github.run_id }} DUCKGRES_SCENARIO_MAX_RUNTIME: 4h @@ -94,7 +101,7 @@ jobs: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - name: Require a current image for cached comparisons - if: ${{ inputs.duckgres_image != '' && inputs.scenario == 'posthog_frozen_perf' }} + if: ${{ inputs.duckgres_image != '' && (inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_coverage' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) }} run: | echo "::error::Cached Trino comparisons require the control-plane image built from this revision. Omit duckgres_image so an older image cannot silently ignore the cache flag." exit 1 @@ -149,7 +156,7 @@ jobs: run: aws eks update-kubeconfig --name "$CLUSTER_NAME" --region "$AWS_REGION" --alias "$KUBE_CONTEXT" - name: Load frozen perf identity and Athena configuration - if: env.SCENARIO_NAME == 'posthog_frozen_perf' || env.SCENARIO_NAME == 'posthog_frozen_perf_coverage_uncached' + if: env.SCENARIO_NAME == 'posthog_frozen_perf' || env.SCENARIO_NAME == 'posthog_frozen_perf_coverage' || env.SCENARIO_NAME == 'posthog_frozen_perf_coverage_uncached' run: bash scripts/scenario_athena_config.sh >> "$GITHUB_ENV" - name: Load properties fixture override diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md index 44f2b39d7..f088b156e 100644 --- a/docs/runbooks/perf-coverage.md +++ b/docs/runbooks/perf-coverage.md @@ -8,21 +8,42 @@ the SELECT computation, not writes or a complete materialization job. ## Run -The regular `posthog_frozen_perf` scenario includes all five configurations. -For a focused DuckDB baseline, dispatch the same workflow on the PR branch: +The regular `posthog_frozen_perf` nightly retains the original table and +properties workloads. Coverage runs separately, one configuration per dispatch, +so five configurations do not share one four-hour scenario budget: + +```sh +gh workflow run scenario-dev.yml --ref \ + -f scenario=posthog_frozen_perf_coverage \ + -f coverage_target=pgwire_uncached +``` + +`coverage_target` defaults to `pgwire_uncached`; supported choices are +`pgwire_uncached`, `pgwire_cached`, `trino`, `trino_cached`, and `athena`. +Dispatch once per configuration. Trino targets select the declarative +`posthog_frozen_perf_coverage_trino.yaml` backing scenario and deploy the Trino +lane. Other targets use the neutral lane. Both reuse the existing frozen setup; +Athena configuration also supplies the frozen runner identity. + +The original focused baseline entry point remains unchanged: ```sh gh workflow run scenario-dev.yml --ref \ -f scenario=posthog_frozen_perf_coverage_uncached ``` -The focused scenario builds the revision being tested, deploys an isolated -warehouse, registers the same frozen files, and executes only -`pgwire_uncached`. It does not require a running local database. Both scenarios -use one warmup and four measured repetitions per query, sequential execution, -and the workflow's explicit worker resource settings. "Uncached" disables the -DuckDB external file cache; it does not guarantee cold OS or storage caches. -See [cache settings](../../tests/perf/README.md#duckdb-cache-comparison). +Each coverage dispatch executes all twelve queries with one warmup and four +measured repetitions, sequentially, using the same frozen files. The explicit +3 CPU / 12 GiB settings describe DuckDB workers, not equivalent resources across +engines. "Uncached" disables the DuckDB external file cache; it does not +guarantee cold OS or storage caches. See +[cache settings](../../tests/perf/README.md#duckdb-cache-comparison). + +Independent dispatches have different run and nightly IDs. Compare matching +recorded commits, catalog versions, fixture versions, resource settings, and +methodology. Pin the same `trino_image` digest for both Trino configurations. +The workflow serializes dispatches on the same branch; complete one before +starting the next. Artifacts live in `perf-coverage/`, with a `-coverage` run ID suffix. The original `perf/` and `perf-properties/` histories are preserved. Main-branch @@ -62,13 +83,17 @@ five configurations, the equal-speed estimate is: ```text additional query time = sum(uncached means) * 5 repetitions * 5 configurations -full workflow estimate = original full workflow time + additional query time +full campaign estimate = original workflow time + additional query time + + sum(overhead for each coverage dispatch) ``` Report this explicitly as an equal-speed estimate, not a forecast of engine performance. Include a sensitivity range for the other four configurations. -Account separately for fixture validation, setup, collection, and teardown; -the focused workflow's build/deployment time must not be multiplied by five. +Each independent coverage dispatch repeats build, deployment, fixture validation, +collection, and teardown. Estimate that overhead per dispatch; do not count it +as query time. A hypothetical single sequential workflow would share setup, +but is not the supported launch method and may exceed the four-hour scenario +limit. Use the measured coverage phase wall time to check the sum-based estimate, because warmups and harness overhead are not in `query_results.csv`. diff --git a/tests/mw-dev/run.sh b/tests/mw-dev/run.sh index 307576e20..96a62c664 100755 --- a/tests/mw-dev/run.sh +++ b/tests/mw-dev/run.sh @@ -63,7 +63,7 @@ esac # it to a digest pin when TRINO_IMAGE is unset. Regular Trino tests keep a # promoted pin because they require atomic writes. TRINO_MASTER_IMAGE_RESOLVER="${TRINO_MASTER_IMAGE_RESOLVER:-$HERE/../../scripts/resolve_trino_master_image.sh}" -if [ "$SCENARIO_NAME" = posthog_frozen_perf ]; then +if [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage ]; then TRINO_IMAGE="${TRINO_IMAGE:-}" else TRINO_IMAGE="${TRINO_IMAGE:-ghcr.io/posthog/trino:86468a7955788b90fe2072f80d86d548972ff28b@sha256:64927a71d2870802a56b671828c6052e7aa37317a7c3a50bd50a93960402d67b}" @@ -130,7 +130,7 @@ require_pr_identity() { } frozen_perf_scenario() { - [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage_uncached ] + [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage_uncached ] } hoglake_perf_enabled() { @@ -789,8 +789,16 @@ cmd_test_scenario() { ;; esac - scenario_file="tests/mw-dev/scenario/scenarios/${SCENARIO_NAME}.yaml" - if [ ! -f "$HERE/scenario/scenarios/${SCENARIO_NAME}.yaml" ]; then + local backing_scenario="$SCENARIO_NAME" + if [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage ]; then + case "${DUCKGRES_SCENARIO_COVERAGE_TARGET:-}" in + pgwire_uncached|pgwire_cached|athena) ;; + trino|trino_cached) backing_scenario=posthog_frozen_perf_coverage_trino ;; + *) echo "DUCKGRES_SCENARIO_COVERAGE_TARGET must select one coverage protocol" >&2; return 2 ;; + esac + fi + scenario_file="tests/mw-dev/scenario/scenarios/${backing_scenario}.yaml" + if [ ! -f "$HERE/scenario/scenarios/${backing_scenario}.yaml" ]; then echo "Scenario '$SCENARIO_NAME' does not exist at $scenario_file." >&2 return 2 fi @@ -866,6 +874,7 @@ spec: - { name: DUCKGRES_SCENARIO_ORG_ID, value: "ci-pr-${PR_NUMBER}-cnpg" } - { name: DUCKGRES_SCENARIO_OUTPUT_BASE, value: "/artifacts/scenario-dev" } - { name: DUCKGRES_SCENARIO_RUN_ID, value: "$DUCKGRES_SCENARIO_RUN_ID" } + - { name: DUCKGRES_SCENARIO_COVERAGE_TARGET, value: "${DUCKGRES_SCENARIO_COVERAGE_TARGET:-}" } - { name: DUCKGRES_SCENARIO_MAX_RUNTIME, value: "${DUCKGRES_SCENARIO_MAX_RUNTIME:-4h}" } - { name: DUCKGRES_SCENARIO_GO_TEST_TIMEOUT, value: "${DUCKGRES_SCENARIO_GO_TEST_TIMEOUT:-4h15m}" } - { name: GOCACHE, value: "/tmp/go-cache" } diff --git a/tests/mw-dev/run_sh_test.go b/tests/mw-dev/run_sh_test.go index f8898286a..282d5baf4 100644 --- a/tests/mw-dev/run_sh_test.go +++ b/tests/mw-dev/run_sh_test.go @@ -489,6 +489,29 @@ func TestScenarioRunsSelectedScenarioAgainstIsolatedStack(t *testing.T) { } } +func TestCoverageDispatchRoutesSelectedTarget(t *testing.T) { + for _, target := range []string{"pgwire_uncached", "pgwire_cached", "trino", "trino_cached", "athena"} { + t.Run(target, func(t *testing.T) { + fakes := newRunSHFakes(t) + cmd := runSHCommand(t, fakes.binDir, "test-scenario", "SCENARIO_RUNNER_IMAGE=example.invalid/duckgres:scenario", "SCENARIO_NAME=posthog_frozen_perf_coverage", "DUCKGRES_SCENARIO_COVERAGE_TARGET="+target) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("scenario failed: %v\n%s", err, out) + } + file := "posthog_frozen_perf_coverage" + if strings.HasPrefix(target, "trino") { + file += "_trino" + } + calls := fakes.calls(t) + if !strings.Contains(calls, "scenarios/"+file+".yaml") { + t.Fatal("wrong coverage backing file") + } + if !strings.Contains(calls, `name: DUCKGRES_SCENARIO_COVERAGE_TARGET, value: "`+target+`"`) { + t.Fatal("target not passed to runner") + } + }) + } +} + func TestScenarioLongNameProducesValidJobLabel(t *testing.T) { const scenario = "posthog_frozen_perf_coverage_uncached" fakes := newRunSHFakes(t) diff --git a/tests/mw-dev/scenario/coverage_test.go b/tests/mw-dev/scenario/coverage_test.go index 5cdf51967..aa9072903 100644 --- a/tests/mw-dev/scenario/coverage_test.go +++ b/tests/mw-dev/scenario/coverage_test.go @@ -1,19 +1,21 @@ package scenario import ( + "os" "path/filepath" "reflect" "strings" "testing" "github.com/posthog/duckgres/tests/mw-dev/scenario/core" + "gopkg.in/yaml.v3" ) func TestCoverageScenariosPreserveBaselineAndFocusedIsolation(t *testing.T) { for _, focused := range []bool{false, true} { - name := "posthog_frozen_perf" + name := "posthog_frozen_perf_coverage" if focused { - name += "_coverage_uncached" + name += "_uncached" } t.Run(name, func(t *testing.T) { scenario, err := core.LoadScenario(filepath.Join("scenarios", name+".yaml")) @@ -69,8 +71,82 @@ func TestCoverageScenariosPreserveBaselineAndFocusedIsolation(t *testing.T) { t.Fatal("focused run provisions Trino") } } else { - assertPerfTargetsPGWireTrinoAndAthena(t, *coverage) + if !reflect.DeepEqual(coverage.With["targets"], []any{"${env:DUCKGRES_SCENARIO_COVERAGE_TARGET}"}) { + t.Fatal("general coverage must select one target") + } } }) } } + +func TestOriginalNightlyExcludesCoverage(t *testing.T) { + scenario, err := core.LoadScenario(filepath.Join("scenarios", "posthog_frozen_perf.yaml")) + if err != nil { + t.Fatal(err) + } + for _, step := range scenario.Steps { + if strings.Contains(step.ID, "coverage") { + t.Fatal("nightly must not include long-running coverage") + } + } +} + +func TestCoverageTargetTemplatesResolve(t *testing.T) { + for _, target := range []string{"pgwire_uncached", "pgwire_cached", "trino", "trino_cached", "athena"} { + t.Run(target, func(t *testing.T) { + name := "posthog_frozen_perf_coverage" + if strings.HasPrefix(target, "trino") { + name += "_trino" + } + scenario, err := core.LoadScenario(filepath.Join("scenarios", name+".yaml")) + if err != nil { + t.Fatal(err) + } + for _, key := range scenario.RequiredEnv { + t.Setenv(key, "example") + } + t.Setenv("DUCKGRES_SCENARIO_COVERAGE_TARGET", target) + resolved, err := resolveRunTemplates(scenario, "example-run") + if err != nil { + t.Fatal(err) + } + var gotTrino bool + for _, step := range resolved.Steps { + if step.Type == "wait_trino_ready" { + gotTrino = true + } + if step.ID == "coverage_queries" && !reflect.DeepEqual(step.With["targets"], []any{target}) { + t.Fatalf("targets = %v", step.With["targets"]) + } + } + if gotTrino != strings.HasPrefix(target, "trino") { + t.Fatalf("wrong setup for %s", target) + } + }) + } +} + +func TestCoverageWorkflowOffersExactlySupportedTargets(t *testing.T) { + raw, err := os.ReadFile(filepath.Join("..", "..", "..", ".github", "workflows", "scenario-dev.yml")) + if err != nil { + t.Fatal(err) + } + var workflow struct { + On struct { + Dispatch struct { + Inputs map[string]struct { + Default string `yaml:"default"` + Type string `yaml:"type"` + Options []string `yaml:"options"` + } `yaml:"inputs"` + } `yaml:"workflow_dispatch"` + } `yaml:"on"` + } + if err := yaml.Unmarshal(raw, &workflow); err != nil { + t.Fatal(err) + } + input := workflow.On.Dispatch.Inputs["coverage_target"] + if input.Default != "pgwire_uncached" || input.Type != "choice" || !reflect.DeepEqual(input.Options, []string{"pgwire_uncached", "pgwire_cached", "trino", "trino_cached", "athena"}) { + t.Fatalf("coverage workflow input = %#v", input) + } +} diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml index ac76c5860..3dcbce2f3 100644 --- a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf.yaml @@ -146,43 +146,6 @@ steps: worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST} worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST} - - id: validate_coverage - type: sql - depends_on: [perf_queries] - with: - org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} - catalog: ducklake - file: ../sql/validate_posthog_coverage.sql - max_attempts: 1 - - - id: coverage_queries - type: perf_queries - depends_on: [validate_coverage] - with: - org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} - catalog: ducklake - catalog_file: ../../../perf/queries/ducklake_posthog_coverage.yaml - targets: [pgwire_uncached, pgwire_cached, trino, trino_cached, athena] - trino_hoglake_catalog: ${env:DUCKGRES_SCENARIO_ORG_ID}-frozen - trino_ca_cert_file: ${env:DUCKGRES_SCENARIO_TRINO_CA_CERT} - trino_startup_timeout: 2m - trino_startup_poll_interval: 2s - athena_region: ${env:DUCKGRES_SCENARIO_ATHENA_REGION} - athena_workgroup: ${env:DUCKGRES_SCENARIO_ATHENA_WORKGROUP} - athena_catalog: AwsDataCatalog - athena_database: ${env:DUCKGRES_SCENARIO_ATHENA_DATABASE} - athena_output_location: ${env:DUCKGRES_SCENARIO_ATHENA_RESULTS_S3_URI} - athena_poll_interval: 500ms - athena_query_timeout: 30m - run_id: ${run_id}-coverage - nightly_run_id: ${run_id} - output_subdir: perf-coverage - suite: coverage - dataset_version: *nightly_dataset - fail_on_query_errors: true - worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST} - worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST} - # Managed Hoglake retains tenant ownership and rejects public deprovision. # The workflow's always-run isolated-stack teardown owns cleanup of the # Duckling, namespace, metadata store and dedicated managed storage prefix. diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage.yaml new file mode 100644 index 000000000..d8c0ab5aa --- /dev/null +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage.yaml @@ -0,0 +1,96 @@ +name: posthog-frozen-perf-coverage +run_id_prefix: scenario-frozen-coverage +required_env: + - DUCKGRES_SCENARIO_COVERAGE_TARGET + - DUCKGRES_SCENARIO_ATHENA_REGION + - DUCKGRES_SCENARIO_ATHENA_WORKGROUP + - DUCKGRES_SCENARIO_ATHENA_DATABASE + - DUCKGRES_SCENARIO_ATHENA_RESULTS_S3_URI + - DUCKGRES_SCENARIO_ORG_ID + - DUCKGRES_SCENARIO_FROZEN_S3_URI + - DUCKGRES_K8S_WORKER_CPU_REQUEST + - DUCKGRES_K8S_WORKER_MEMORY_REQUEST +steps: + - id: provision + type: provision_warehouse + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + request: + database_name: scenario-frozen-coverage-${run_id_token} + team_id: 1 + metadata_store: + type: cnpg-shard + ducklake: + enabled: true + data_store: + type: s3bucket + + - id: wait_ready + type: wait_warehouse_ready + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + timeout: 15m + poll_interval: 10s + + - id: setup_frozen_views + type: sql + depends_on: [wait_ready] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/setup_frozen_views.sql + max_attempts: 12 + retry_interval: 10s + + - id: setup_posthog_tables + type: sql + depends_on: [setup_frozen_views] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/setup_posthog_tables.sql + max_attempts: 3 + retry_interval: 10s + + - id: validate_posthog_tables + type: sql + depends_on: [setup_posthog_tables] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/validate_posthog_tables.sql + max_attempts: 3 + retry_interval: 10s + + - id: validate_coverage + type: sql + depends_on: [validate_posthog_tables] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/validate_posthog_coverage.sql + max_attempts: 1 + + - id: coverage_queries + type: perf_queries + depends_on: [validate_coverage] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + catalog_file: ../../../perf/queries/ducklake_posthog_coverage.yaml + targets: ["${env:DUCKGRES_SCENARIO_COVERAGE_TARGET}"] + athena_region: ${env:DUCKGRES_SCENARIO_ATHENA_REGION} + athena_workgroup: ${env:DUCKGRES_SCENARIO_ATHENA_WORKGROUP} + athena_catalog: AwsDataCatalog + athena_database: ${env:DUCKGRES_SCENARIO_ATHENA_DATABASE} + athena_output_location: ${env:DUCKGRES_SCENARIO_ATHENA_RESULTS_S3_URI} + athena_poll_interval: 500ms + athena_query_timeout: 30m + run_id: ${run_id}-coverage + nightly_run_id: ${run_id} + output_subdir: perf-coverage + suite: coverage + dataset_version: posthog-file-views-v1 + fail_on_query_errors: true + worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST} + worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST} diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_trino.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_trino.yaml new file mode 100644 index 000000000..004f35cea --- /dev/null +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_trino.yaml @@ -0,0 +1,131 @@ +name: posthog-frozen-perf-coverage-trino +run_id_prefix: scenario-frozen-coverage-trino +required_env: + - DUCKGRES_SCENARIO_COVERAGE_TARGET + - DUCKGRES_SCENARIO_ORG_ID + - DUCKGRES_SCENARIO_HOGLAKE_URI + - DUCKGRES_SCENARIO_FROZEN_S3_URI + - DUCKGRES_SCENARIO_TRINO_CA_CERT + - DUCKGRES_SCENARIO_TRINO_CATALOG_STORE_CELL_ID + - DUCKGRES_SCENARIO_TRINO_CATALOG_STORE_DSN + - DUCKGRES_SCENARIO_TRINO_PERF_URL + - DUCKGRES_SCENARIO_TRINO_PERF_CELL_ID + - DUCKGRES_SCENARIO_TRINO_CACHED_URL + - DUCKGRES_SCENARIO_TRINO_CACHED_CELL_ID + - DUCKGRES_SCENARIO_TRINO_ADMIN_PASSWORD_FILE + - DUCKGRES_SCENARIO_ATHENA_REGION + - DUCKGRES_SCENARIO_ATHENA_WORKGROUP + - DUCKGRES_SCENARIO_ATHENA_DATABASE + - DUCKGRES_SCENARIO_ATHENA_RESULTS_S3_URI + - DUCKGRES_K8S_WORKER_CPU_REQUEST + - DUCKGRES_K8S_WORKER_MEMORY_REQUEST +steps: + - id: provision + type: provision_warehouse + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + request: + database_name: scenario-frozen-perf-${run_id_token} + team_id: 1 + metadata_store: + type: cnpg-shard + ducklake: + enabled: true + data_store: + type: s3bucket + trino: + enabled: true + tier: free + + - id: wait_ready + type: wait_warehouse_ready + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + timeout: 15m + poll_interval: 10s + + - id: wait_trino_ready + type: wait_trino_ready + depends_on: [wait_ready] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + timeout: 15m + poll_interval: 10s + + - id: setup_frozen_views + type: sql + depends_on: [wait_ready] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/setup_frozen_views.sql + max_attempts: 12 + retry_interval: 10s + + - id: setup_posthog_tables + type: sql + depends_on: [setup_frozen_views] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/setup_posthog_tables.sql + max_attempts: 3 + retry_interval: 10s + + - id: validate_posthog_tables + type: sql + depends_on: [setup_posthog_tables] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/validate_posthog_tables.sql + max_attempts: 3 + retry_interval: 10s + + - id: setup_hoglake + type: setup_hoglake + depends_on: [validate_posthog_tables] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + uri: ${env:DUCKGRES_SCENARIO_HOGLAKE_URI} + source: ${env:DUCKGRES_SCENARIO_FROZEN_S3_URI} + hoglake_catalog: ${env:DUCKGRES_SCENARIO_ORG_ID}-frozen + file: ../perf/setup_hoglake.py + + - id: validate_coverage + type: sql + depends_on: [setup_hoglake, wait_trino_ready] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + file: ../sql/validate_posthog_coverage.sql + max_attempts: 1 + + - id: coverage_queries + type: perf_queries + depends_on: [validate_coverage] + with: + org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} + catalog: ducklake + catalog_file: ../../../perf/queries/ducklake_posthog_coverage.yaml + targets: ["${env:DUCKGRES_SCENARIO_COVERAGE_TARGET}"] + trino_hoglake_catalog: ${env:DUCKGRES_SCENARIO_ORG_ID}-frozen + trino_ca_cert_file: ${env:DUCKGRES_SCENARIO_TRINO_CA_CERT} + trino_startup_timeout: 2m + trino_startup_poll_interval: 2s + athena_region: ${env:DUCKGRES_SCENARIO_ATHENA_REGION} + athena_workgroup: ${env:DUCKGRES_SCENARIO_ATHENA_WORKGROUP} + athena_catalog: AwsDataCatalog + athena_database: ${env:DUCKGRES_SCENARIO_ATHENA_DATABASE} + athena_output_location: ${env:DUCKGRES_SCENARIO_ATHENA_RESULTS_S3_URI} + athena_poll_interval: 500ms + athena_query_timeout: 30m + run_id: ${run_id}-coverage + nightly_run_id: ${run_id} + output_subdir: perf-coverage + suite: coverage + dataset_version: posthog-file-views-v1 + fail_on_query_errors: true + worker_cpu: ${env:DUCKGRES_K8S_WORKER_CPU_REQUEST} + worker_memory: ${env:DUCKGRES_K8S_WORKER_MEMORY_REQUEST} + diff --git a/tests/perf/README.md b/tests/perf/README.md index bc3be3134..8c4056e48 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -2,7 +2,7 @@ This package contains the golden-query performance harness. -The frozen suite also includes twelve additional workload shapes in +Separate frozen coverage scenarios exercise twelve additional workload shapes in `queries/ducklake_posthog_coverage.yaml`. They retain separate `perf-coverage/` artifacts and versioned intent IDs. For the focused uncached DuckDB run, baseline recording, runtime estimates, and failure recovery, see the From 3aadf35d6e510b2bf2c55af38aec49fbc73b240e Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 13:58:31 -0400 Subject: [PATCH 04/13] docs(perf): record uncached coverage baseline and runtime estimate --- docs/runbooks/perf-coverage.md | 7 ++ .../README.md | 67 +++++++++++++++++++ .../manifest.json | 56 ++++++++++++++++ .../samples.csv | 49 ++++++++++++++ 4 files changed, 179 insertions(+) create mode 100644 tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md create mode 100644 tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/manifest.json create mode 100644 tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/samples.csv diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md index f088b156e..1009d473b 100644 --- a/docs/runbooks/perf-coverage.md +++ b/docs/runbooks/perf-coverage.md @@ -61,6 +61,13 @@ turning joins or funnels into cheap empty-result benchmarks. ## Record a comparison baseline +The initial uncached DuckDB comparison base is the +[2026-09-25 coverage v1 baseline](../../tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md). +It retains 48 successful measurements and the complete methodology. The query +phase took 87m06s; the workflow took 1h42m45s. The equal-speed expanded campaign +estimate is about 9h37m across separate target dispatches plus the original suite. + + Keep the successful focused run URL, tested commit, catalog version/hash, dataset version, worker resources, cache settings, and repetition counts alongside sanitized per-query timing samples. Do not commit query result diff --git a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md new file mode 100644 index 000000000..d54233993 --- /dev/null +++ b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md @@ -0,0 +1,67 @@ +# Coverage v1: uncached DuckDB baseline + +This is the initial comparison base for the twelve coverage v1 intents on +`pgwire_uncached`. [Successful source run](https://github.com/PostHog/duckgres/actions/runs/36159233201) +measured commit `9c9f5f305f87b7f0d41c0b54d64e5648aeb8ef83` on 2026-09-25. +The focused scenario and catalog are unchanged by subsequent launch-path edits. + +[Samples](samples.csv) retain all 48 successful measurements; +[manifest](manifest.json) records the catalog hash, dataset version, cache +settings, resource requests, methodology, and timing calculations. No query +result values or internal fixture locations are retained. + +## Measured times + +- Coverage phase: **87m06s**, including 12 warmups and 48 measurements; zero errors. +- Complete workflow: **1h42m45s** (creation through completed-run update). +- Time outside the coverage phase: **15m39s**, including build, deployment, + fixture validation, artifact collection, and teardown. Coverage validation + itself took **3m50s**. +- Sum of per-query means: **17m26s**. The two person joins account for **57.1%**. + +| Query family | Mean (s) | Median (s) | Observed min–max (s) | +|---|---:|---:|---:| +| Daily unique users | 61.35 | 61.68 | 57.99–64.03 | +| Event trends | 38.86 | 39.16 | 36.72–40.39 | +| Person enrichment join | 361.08 | 362.75 | 348.63–370.20 | +| Person activity join | 235.86 | 235.82 | 229.27–242.52 | +| Recent events | 72.50 | 72.99 | 68.57–75.46 | +| Active actors | 41.93 | 42.27 | 40.39–42.77 | +| Ordered funnel | 67.24 | 67.42 | 64.07–70.07 | +| Sampled-week retention | 42.17 | 43.28 | 37.92–44.19 | +| Repeat-activity cohort | 21.09 | 21.76 | 18.82–22.01 | +| Exclusion cohort | 27.72 | 28.50 | 25.09–28.80 | +| Daily-person model | 22.81 | 23.04 | 20.99–24.17 | +| Session model | 53.31 | 52.58 | 51.32–56.74 | + +## Expanded campaign estimate + +Keep the original nightly suite separate and dispatch coverage once per +configuration. With one warmup plus four measurements, equal query speeds +would add **7h16m of query work** across five configurations. Using the measured +15m39s overhead for each dispatch adds another **1h18m**. Adding the +[original 63m20s workflow](https://github.com/PostHog/duckgres/actions/runs/36113843696) +gives **about 9h37m total**, or **8h34m additional**. + +This is a serial planning estimate, not a measured result for the other engines. +Their setup, warmup, and query costs may differ. It excludes queue delays. +Holding DuckDB uncached fixed and varying the other four configurations: + +| Other configurations' query durations relative to uncached DuckDB | Estimated campaign | +|---|---:| +| 0.5× | 6h43m | +| 1× | 9h37m | +| 2× | 15h26m | + +These are scenarios, not statistical confidence bounds. Even the equal-speed +query work exceeds the four-hour scenario limit, which is why coverage uses +separate target dispatches. This baseline does not establish Trino or Athena +performance. Resource requests are 3 CPU / 12Gi for DuckDB workers only. + +## Future comparisons + +Compare matching intent IDs and protocol using the same catalog hash, fixture, +resource requests, and cache settings. Report per-query mean/median ratios and +absolute times; a total ratio is dominated by slow queries. A changed query or +window requires a new intent version and baseline. Four repetitions do not +establish tail latency. See the [runbook](../../../../docs/runbooks/perf-coverage.md). diff --git a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/manifest.json b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/manifest.json new file mode 100644 index 000000000..1a740e045 --- /dev/null +++ b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/manifest.json @@ -0,0 +1,56 @@ +{ + "baseline_id": "coverage-v1-duckdb-uncached-2026-09-25", + "source_run_url": "https://github.com/PostHog/duckgres/actions/runs/36159233201", + "tested_commit": "9c9f5f305f87b7f0d41c0b54d64e5648aeb8ef83", + "catalog_path": "tests/perf/queries/ducklake_posthog_coverage.yaml", + "catalog_sha256": "e2971f25433a6067f1c0ac42229bf7dc7c82c0c8ab51d6e476b4848df5c59226", + "dataset_version": "posthog-file-views-v1", + "suite": "coverage", + "protocol": "pgwire_uncached", + "scenario": "posthog_frozen_perf_coverage_uncached", + "worker_resource_requests": { + "cpu": "3", + "memory": "12Gi" + }, + "cache_settings": { + "enable_external_file_cache": false, + "parquet_metadata_cache": false, + "enable_http_metadata_cache": false + }, + "warmup_iterations": 1, + "measure_iterations": 4, + "query_intents": 12, + "measured_queries": 48, + "warmup_queries": 12, + "query_errors": 0, + "timing_seconds": { + "coverage_phase": 5225.503589484, + "coverage_fixture_validation": 230.235161795, + "workflow_created_to_completed_updated_at": 6165, + "workflow_outside_coverage_phase": 939.4964105159997, + "measured_query_total": 4183.687134048, + "sum_query_means": 1045.921783512 + }, + "workflow_created_at": "2026-09-25T16:12:09Z", + "completed_workflow_updated_at": "2026-09-25T17:54:54Z", + "coverage_started_at": "2026-09-25T16:26:19.084523672Z", + "coverage_finished_at": "2026-09-25T17:53:24.58464969Z", + "estimate": { + "original_workflow_url": "https://github.com/PostHog/duckgres/actions/runs/36113843696", + "original_workflow_seconds": 3800, + "repetitions_per_configuration": 5, + "configurations": 5, + "query_seconds_at_equal_speed": 26148.044587800003, + "assumed_overhead_per_dispatch_seconds": 939.4964105159997, + "campaign_seconds_at_equal_speed": 34645.526640380005 + }, + "limitations": [ + "Only uncached DuckDB was measured; peer speeds and setup overhead are assumptions.", + "Resource values are DuckDB worker requests, not equal engine hardware budgets.", + "Uncached disables named DuckDB caches, not OS or storage caches.", + "Frozen fixture contains sampled days; retention is a performance shape, not a production rate.", + "Four repetitions characterize observed variation, not tail latency.", + "Independent coverage starts with different cache history from coverage appended after other suites.", + "Workflow wall time uses createdAt through the completed run updatedAt; no queue time from separate future dispatches is estimated." + ] +} diff --git a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/samples.csv b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/samples.csv new file mode 100644 index 000000000..e4a154e5e --- /dev/null +++ b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/samples.csv @@ -0,0 +1,49 @@ +query_id,intent_id,protocol,measure_iteration,duration_ms +q_coverage_daily_uniques_v1__ducklake_table,intent_coverage_daily_uniques_v1,pgwire_uncached,1,57990.076097 +q_coverage_event_trends_v1__ducklake_table,intent_coverage_event_trends_v1,pgwire_uncached,1,36721.211893 +q_coverage_person_enrichment_v1__ducklake_table,intent_coverage_person_enrichment_v1,pgwire_uncached,1,348630.727166 +q_coverage_person_activity_join_v1__ducklake_table,intent_coverage_person_activity_join_v1,pgwire_uncached,1,229274.438235 +q_coverage_recent_events_v1__ducklake_table,intent_coverage_recent_events_v1,pgwire_uncached,1,68569.479379 +q_coverage_active_actors_v1__ducklake_table,intent_coverage_active_actors_v1,pgwire_uncached,1,40394.218659 +q_coverage_ordered_funnel_v1__ducklake_table,intent_coverage_ordered_funnel_v1,pgwire_uncached,1,64066.620569 +q_coverage_weekly_retention_v1__ducklake_table,intent_coverage_weekly_retention_v1,pgwire_uncached,1,37916.014597 +q_coverage_repeat_activity_cohort_v1__ducklake_table,intent_coverage_repeat_activity_cohort_v1,pgwire_uncached,1,18816.490929 +q_coverage_exclusion_cohort_v1__ducklake_table,intent_coverage_exclusion_cohort_v1,pgwire_uncached,1,25091.107455 +q_coverage_daily_person_model_v1__ducklake_table,intent_coverage_daily_person_model_v1,pgwire_uncached,1,20991.281524 +q_coverage_session_model_v1__ducklake_table,intent_coverage_session_model_v1,pgwire_uncached,1,51323.299029 +q_coverage_daily_uniques_v1__ducklake_table,intent_coverage_daily_uniques_v1,pgwire_uncached,2,60034.007037 +q_coverage_event_trends_v1__ducklake_table,intent_coverage_event_trends_v1,pgwire_uncached,2,39430.339134 +q_coverage_person_enrichment_v1__ducklake_table,intent_coverage_person_enrichment_v1,pgwire_uncached,2,370202.228797 +q_coverage_person_activity_join_v1__ducklake_table,intent_coverage_person_activity_join_v1,pgwire_uncached,2,242519.554305 +q_coverage_recent_events_v1__ducklake_table,intent_coverage_recent_events_v1,pgwire_uncached,2,73454.258391 +q_coverage_active_actors_v1__ducklake_table,intent_coverage_active_actors_v1,pgwire_uncached,2,42441.753842 +q_coverage_ordered_funnel_v1__ducklake_table,intent_coverage_ordered_funnel_v1,pgwire_uncached,2,70066.542500 +q_coverage_weekly_retention_v1__ducklake_table,intent_coverage_weekly_retention_v1,pgwire_uncached,2,43317.646138 +q_coverage_repeat_activity_cohort_v1__ducklake_table,intent_coverage_repeat_activity_cohort_v1,pgwire_uncached,2,21521.700775 +q_coverage_exclusion_cohort_v1__ducklake_table,intent_coverage_exclusion_cohort_v1,pgwire_uncached,2,28393.201396 +q_coverage_daily_person_model_v1__ducklake_table,intent_coverage_daily_person_model_v1,pgwire_uncached,2,23109.121973 +q_coverage_session_model_v1__ducklake_table,intent_coverage_session_model_v1,pgwire_uncached,2,52682.057170 +q_coverage_daily_uniques_v1__ducklake_table,intent_coverage_daily_uniques_v1,pgwire_uncached,3,64029.693248 +q_coverage_event_trends_v1__ducklake_table,intent_coverage_event_trends_v1,pgwire_uncached,3,40387.651561 +q_coverage_person_enrichment_v1__ducklake_table,intent_coverage_person_enrichment_v1,pgwire_uncached,3,362181.664088 +q_coverage_person_activity_join_v1__ducklake_table,intent_coverage_person_activity_join_v1,pgwire_uncached,3,238072.547003 +q_coverage_recent_events_v1__ducklake_table,intent_coverage_recent_events_v1,pgwire_uncached,3,75460.142206 +q_coverage_active_actors_v1__ducklake_table,intent_coverage_active_actors_v1,pgwire_uncached,3,42773.287401 +q_coverage_ordered_funnel_v1__ducklake_table,intent_coverage_ordered_funnel_v1,pgwire_uncached,3,69409.701471 +q_coverage_weekly_retention_v1__ducklake_table,intent_coverage_weekly_retention_v1,pgwire_uncached,3,44191.445753 +q_coverage_repeat_activity_cohort_v1__ducklake_table,intent_coverage_repeat_activity_cohort_v1,pgwire_uncached,3,22001.911651 +q_coverage_exclusion_cohort_v1__ducklake_table,intent_coverage_exclusion_cohort_v1,pgwire_uncached,3,28798.335575 +q_coverage_daily_person_model_v1__ducklake_table,intent_coverage_daily_person_model_v1,pgwire_uncached,3,24172.015999 +q_coverage_session_model_v1__ducklake_table,intent_coverage_session_model_v1,pgwire_uncached,3,52483.776282 +q_coverage_daily_uniques_v1__ducklake_table,intent_coverage_daily_uniques_v1,pgwire_uncached,4,63326.694027 +q_coverage_event_trends_v1__ducklake_table,intent_coverage_event_trends_v1,pgwire_uncached,4,38884.138434 +q_coverage_person_enrichment_v1__ducklake_table,intent_coverage_person_enrichment_v1,pgwire_uncached,4,363320.561007 +q_coverage_person_activity_join_v1__ducklake_table,intent_coverage_person_activity_join_v1,pgwire_uncached,4,233574.135155 +q_coverage_recent_events_v1__ducklake_table,intent_coverage_recent_events_v1,pgwire_uncached,4,72533.884957 +q_coverage_active_actors_v1__ducklake_table,intent_coverage_active_actors_v1,pgwire_uncached,4,42101.277939 +q_coverage_ordered_funnel_v1__ducklake_table,intent_coverage_ordered_funnel_v1,pgwire_uncached,4,65435.443518 +q_coverage_weekly_retention_v1__ducklake_table,intent_coverage_weekly_retention_v1,pgwire_uncached,4,43249.354433 +q_coverage_repeat_activity_cohort_v1__ducklake_table,intent_coverage_repeat_activity_cohort_v1,pgwire_uncached,4,22014.864298 +q_coverage_exclusion_cohort_v1__ducklake_table,intent_coverage_exclusion_cohort_v1,pgwire_uncached,4,28613.186554 +q_coverage_daily_person_model_v1__ducklake_table,intent_coverage_daily_person_model_v1,pgwire_uncached,4,22972.236876 +q_coverage_session_model_v1__ducklake_table,intent_coverage_session_model_v1,pgwire_uncached,4,56741.807622 From b503f633eff089ce8e12e875ffe9fd03ed465d81 Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 14:02:25 -0400 Subject: [PATCH 05/13] docs(perf): label extended coverage as manual only --- .github/workflows/scenario-dev.yml | 2 +- docs/runbooks/perf-coverage.md | 8 +++++--- tests/perf/README.md | 5 +++-- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/.github/workflows/scenario-dev.yml b/.github/workflows/scenario-dev.yml index 4dec18050..8985c937b 100644 --- a/.github/workflows/scenario-dev.yml +++ b/.github/workflows/scenario-dev.yml @@ -9,7 +9,7 @@ on: default: full-suite type: string coverage_target: - description: Target for posthog_frozen_perf_coverage; dispatch once per configuration + description: Manual extended coverage target for posthog_frozen_perf_coverage; excluded from nightly required: false default: pgwire_uncached type: choice diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md index 1009d473b..217b60523 100644 --- a/docs/runbooks/perf-coverage.md +++ b/docs/runbooks/perf-coverage.md @@ -1,4 +1,4 @@ -# Workload coverage benchmark +# Extended workload coverage benchmark The workload coverage catalog supplements the original frozen table and properties comparisons. It measures filtered unique users, grouped trends, @@ -9,8 +9,10 @@ the SELECT computation, not writes or a complete materialization job. ## Run The regular `posthog_frozen_perf` nightly retains the original table and -properties workloads. Coverage runs separately, one configuration per dispatch, -so five configurations do not share one four-hour scenario budget: +properties workloads. The twelve additional queries are an **extended, manual-only +run**: scheduled nightly runs never include them. Start them explicitly through +`workflow_dispatch`, one configuration per dispatch, so five configurations do +not share one four-hour scenario budget: ```sh gh workflow run scenario-dev.yml --ref \ diff --git a/tests/perf/README.md b/tests/perf/README.md index 8c4056e48..1aa95f28d 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -2,9 +2,10 @@ This package contains the golden-query performance harness. -Separate frozen coverage scenarios exercise twelve additional workload shapes in +Manual-only extended coverage scenarios exercise twelve additional workload shapes in `queries/ducklake_posthog_coverage.yaml`. They retain separate `perf-coverage/` -artifacts and versioned intent IDs. For the focused uncached DuckDB run, +artifacts and versioned intent IDs. These queries are excluded from standard +nightly runs and execute only when explicitly dispatched. For the focused uncached DuckDB run, baseline recording, runtime estimates, and failure recovery, see the [coverage runbook](../../docs/runbooks/perf-coverage.md). From fedd735e899f44ef5df5d1d18f28a6ca48084dfe Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 14:11:18 -0400 Subject: [PATCH 06/13] test(perf): remove added benchmark test cases --- tests/mw-dev/run_sh_test.go | 87 +++----------- tests/mw-dev/scenario/coverage_test.go | 152 ------------------------- tests/mw-dev/scenario/runner_test.go | 1 - tests/perf/core/coverage_test.go | 128 --------------------- tests/perf/core/suite_test.go | 16 --- tests/perf/publisher/publisher_test.go | 83 +++++++------- 6 files changed, 54 insertions(+), 413 deletions(-) delete mode 100644 tests/mw-dev/scenario/coverage_test.go delete mode 100644 tests/perf/core/coverage_test.go delete mode 100644 tests/perf/core/suite_test.go diff --git a/tests/mw-dev/run_sh_test.go b/tests/mw-dev/run_sh_test.go index 282d5baf4..001e4916e 100644 --- a/tests/mw-dev/run_sh_test.go +++ b/tests/mw-dev/run_sh_test.go @@ -138,27 +138,21 @@ func TestTrinoDeployStartsWorkloadsWithoutScaleSubresource(t *testing.T) { } } -func TestDeployCreatesDedicatedScenarioPodIdentityForFrozenPerf(t *testing.T) { - for _, scenario := range []string{"posthog_frozen_perf", "posthog_frozen_perf_coverage_uncached"} { - t.Run(scenario, func(t *testing.T) { - fakes := newRunSHFakes(t) - cmd := runSHCommand(t, fakes.binDir, "deploy", - "SCENARIO_DEV_ALLOW_DUCKLING_DELETE=1", - "SCENARIO_NAME="+scenario, - "E2E_SUITE=neutral", - "SCENARIO_POD_IDENTITY_ROLE=arn:aws:iam::123456789012:role/athena-perf", - ) - out, err := cmd.CombinedOutput() - if err != nil { - t.Fatalf("Athena perf deploy failed: %v\n%s", err, out) - } - calls := fakes.calls(t) - want := "aws eks create-pod-identity-association --region us-east-1 --cluster-name test-cluster --namespace duckgres-ci-pr-123 --service-account duckgres-scenario --role-arn arn:aws:iam::123456789012:role/athena-perf" - if !strings.Contains(calls, want) { - t.Fatalf("deploy did not create dedicated scenario Pod Identity; calls:\n%s", calls) - } - - }) +func TestDeployCreatesDedicatedScenarioPodIdentityForAthenaPerf(t *testing.T) { + fakes := newRunSHFakes(t) + cmd := runSHCommand(t, fakes.binDir, "deploy", + "SCENARIO_DEV_ALLOW_DUCKLING_DELETE=1", + "SCENARIO_NAME=posthog_frozen_perf", + "SCENARIO_POD_IDENTITY_ROLE=arn:aws:iam::123456789012:role/athena-perf", + ) + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("Athena perf deploy failed: %v\n%s", err, out) + } + calls := fakes.calls(t) + want := "aws eks create-pod-identity-association --region us-east-1 --cluster-name test-cluster --namespace duckgres-ci-pr-123 --service-account duckgres-scenario --role-arn arn:aws:iam::123456789012:role/athena-perf" + if !strings.Contains(calls, want) { + t.Fatalf("deploy did not create dedicated scenario Pod Identity; calls:\n%s", calls) } } @@ -489,57 +483,6 @@ func TestScenarioRunsSelectedScenarioAgainstIsolatedStack(t *testing.T) { } } -func TestCoverageDispatchRoutesSelectedTarget(t *testing.T) { - for _, target := range []string{"pgwire_uncached", "pgwire_cached", "trino", "trino_cached", "athena"} { - t.Run(target, func(t *testing.T) { - fakes := newRunSHFakes(t) - cmd := runSHCommand(t, fakes.binDir, "test-scenario", "SCENARIO_RUNNER_IMAGE=example.invalid/duckgres:scenario", "SCENARIO_NAME=posthog_frozen_perf_coverage", "DUCKGRES_SCENARIO_COVERAGE_TARGET="+target) - if out, err := cmd.CombinedOutput(); err != nil { - t.Fatalf("scenario failed: %v\n%s", err, out) - } - file := "posthog_frozen_perf_coverage" - if strings.HasPrefix(target, "trino") { - file += "_trino" - } - calls := fakes.calls(t) - if !strings.Contains(calls, "scenarios/"+file+".yaml") { - t.Fatal("wrong coverage backing file") - } - if !strings.Contains(calls, `name: DUCKGRES_SCENARIO_COVERAGE_TARGET, value: "`+target+`"`) { - t.Fatal("target not passed to runner") - } - }) - } -} - -func TestScenarioLongNameProducesValidJobLabel(t *testing.T) { - const scenario = "posthog_frozen_perf_coverage_uncached" - fakes := newRunSHFakes(t) - cmd := runSHCommand(t, fakes.binDir, "test-scenario", - "SCENARIO_RUNNER_IMAGE=example.invalid/duckgres:scenario", "SCENARIO_NAME="+scenario) - if out, err := cmd.CombinedOutput(); err != nil { - t.Fatalf("scenario failed: %v\n%s", err, out) - } - match := regexp.MustCompile(`kind: Job\nmetadata:\n name: ([^\n]+)`).FindStringSubmatch(fakes.calls(t)) - if len(match) != 2 { - t.Fatal("missing Job name") - } - name := match[1] - if len(name) > 63 || !regexp.MustCompile(`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`).MatchString(name) { - t.Fatalf("invalid Job label %q (%d bytes)", name, len(name)) - } - runID := "scenario-dev-" + strings.ReplaceAll(scenario, "_", "-") + "-123" - checksum := exec.Command("cksum") - checksum.Stdin = strings.NewReader(runID) - out, err := checksum.Output() - if err != nil { - t.Fatal(err) - } - if !strings.HasSuffix(name, "-"+strings.Fields(string(out))[0]) { - t.Fatalf("Job name %q lost run ID checksum", name) - } -} - func TestScenarioStopsJobWithoutCollectingArtifactsWhenContainerTerminationCannotBeConfirmed(t *testing.T) { fakes := newRunSHFakes(t) diff --git a/tests/mw-dev/scenario/coverage_test.go b/tests/mw-dev/scenario/coverage_test.go deleted file mode 100644 index aa9072903..000000000 --- a/tests/mw-dev/scenario/coverage_test.go +++ /dev/null @@ -1,152 +0,0 @@ -package scenario - -import ( - "os" - "path/filepath" - "reflect" - "strings" - "testing" - - "github.com/posthog/duckgres/tests/mw-dev/scenario/core" - "gopkg.in/yaml.v3" -) - -func TestCoverageScenariosPreserveBaselineAndFocusedIsolation(t *testing.T) { - for _, focused := range []bool{false, true} { - name := "posthog_frozen_perf_coverage" - if focused { - name += "_uncached" - } - t.Run(name, func(t *testing.T) { - scenario, err := core.LoadScenario(filepath.Join("scenarios", name+".yaml")) - if err != nil { - t.Fatal(err) - } - var coverage *core.Step - var validation *core.Step - for i := range scenario.Steps { - step := &scenario.Steps[i] - if step.ID == "validate_coverage" { - validation = step - } - if step.ID == "coverage_queries" { - coverage = step - } - if focused && (strings.Contains(step.Type, "trino") || step.Type == "setup_hoglake" || step.Type == "properties_comparison") { - t.Fatalf("focused run includes %s", step.Type) - } - } - if validation == nil || validation.With["file"] != "../sql/validate_posthog_coverage.sql" { - t.Fatal("missing coverage fixture validation") - } - if coverage == nil { - t.Fatal("missing coverage phase") - } - if !reflect.DeepEqual(coverage.DependsOn, []string{"validate_coverage"}) { - t.Fatal("coverage must depend on fixture validation") - } - if coverage.With["catalog_file"] != "../../../perf/queries/ducklake_posthog_coverage.yaml" { - t.Fatal("wrong coverage catalog") - } - if coverage.With["output_subdir"] != "perf-coverage" || coverage.With["run_id"] != "${run_id}-coverage" || coverage.With["nightly_run_id"] != "${run_id}" { - t.Fatal("coverage must have separate artifacts and shared nightly identity") - } - if coverage.With["suite"] != "coverage" { - t.Fatal("coverage must have its own comparison suite") - } - if coverage.With["fail_on_query_errors"] != true { - t.Fatal("coverage errors must fail") - } - if focused { - if !reflect.DeepEqual(coverage.With["targets"], []any{"pgwire_uncached"}) { - t.Fatalf("focused targets = %v", coverage.With["targets"]) - } - for _, env := range scenario.RequiredEnv { - if strings.Contains(env, "TRINO") || strings.Contains(env, "ATHENA") || strings.Contains(env, "HOGLAKE") { - t.Fatalf("unnecessary focused requirement %s", env) - } - } - request := scenario.Steps[0].With["request"].(map[string]any) - if _, ok := request["trino"]; ok { - t.Fatal("focused run provisions Trino") - } - } else { - if !reflect.DeepEqual(coverage.With["targets"], []any{"${env:DUCKGRES_SCENARIO_COVERAGE_TARGET}"}) { - t.Fatal("general coverage must select one target") - } - } - }) - } -} - -func TestOriginalNightlyExcludesCoverage(t *testing.T) { - scenario, err := core.LoadScenario(filepath.Join("scenarios", "posthog_frozen_perf.yaml")) - if err != nil { - t.Fatal(err) - } - for _, step := range scenario.Steps { - if strings.Contains(step.ID, "coverage") { - t.Fatal("nightly must not include long-running coverage") - } - } -} - -func TestCoverageTargetTemplatesResolve(t *testing.T) { - for _, target := range []string{"pgwire_uncached", "pgwire_cached", "trino", "trino_cached", "athena"} { - t.Run(target, func(t *testing.T) { - name := "posthog_frozen_perf_coverage" - if strings.HasPrefix(target, "trino") { - name += "_trino" - } - scenario, err := core.LoadScenario(filepath.Join("scenarios", name+".yaml")) - if err != nil { - t.Fatal(err) - } - for _, key := range scenario.RequiredEnv { - t.Setenv(key, "example") - } - t.Setenv("DUCKGRES_SCENARIO_COVERAGE_TARGET", target) - resolved, err := resolveRunTemplates(scenario, "example-run") - if err != nil { - t.Fatal(err) - } - var gotTrino bool - for _, step := range resolved.Steps { - if step.Type == "wait_trino_ready" { - gotTrino = true - } - if step.ID == "coverage_queries" && !reflect.DeepEqual(step.With["targets"], []any{target}) { - t.Fatalf("targets = %v", step.With["targets"]) - } - } - if gotTrino != strings.HasPrefix(target, "trino") { - t.Fatalf("wrong setup for %s", target) - } - }) - } -} - -func TestCoverageWorkflowOffersExactlySupportedTargets(t *testing.T) { - raw, err := os.ReadFile(filepath.Join("..", "..", "..", ".github", "workflows", "scenario-dev.yml")) - if err != nil { - t.Fatal(err) - } - var workflow struct { - On struct { - Dispatch struct { - Inputs map[string]struct { - Default string `yaml:"default"` - Type string `yaml:"type"` - Options []string `yaml:"options"` - } `yaml:"inputs"` - } `yaml:"workflow_dispatch"` - } `yaml:"on"` - } - if err := yaml.Unmarshal(raw, &workflow); err != nil { - t.Fatal(err) - } - input := workflow.On.Dispatch.Inputs["coverage_target"] - if input.Default != "pgwire_uncached" || input.Type != "choice" || !reflect.DeepEqual(input.Options, []string{"pgwire_uncached", "pgwire_cached", "trino", "trino_cached", "athena"}) { - t.Fatalf("coverage workflow input = %#v", input) - } -} diff --git a/tests/mw-dev/scenario/runner_test.go b/tests/mw-dev/scenario/runner_test.go index d4ff98916..74a34a768 100644 --- a/tests/mw-dev/scenario/runner_test.go +++ b/tests/mw-dev/scenario/runner_test.go @@ -182,7 +182,6 @@ func TestFrozenSuccessScenariosUseIsolatedStackWarehouseIdentity(t *testing.T) { for _, scenarioFile := range []string{ "posthog_frozen_metadata.yaml", "posthog_frozen_perf.yaml", - "posthog_frozen_perf_coverage_uncached.yaml", "posthog_frozen_dbt.yaml", "fast-suite.yaml", "full-suite.yaml", diff --git a/tests/perf/core/coverage_test.go b/tests/perf/core/coverage_test.go deleted file mode 100644 index 5679a0418..000000000 --- a/tests/perf/core/coverage_test.go +++ /dev/null @@ -1,128 +0,0 @@ -package core - -import ( - "database/sql" - "fmt" - "os" - "path/filepath" - "strings" - "testing" - - _ "github.com/duckdb/duckdb-go/v2" -) - -// Exercise the actual benchmark SQL against a small fixture with out-of-order -// funnel events, person snapshot duplicates, tenant collisions and boundary dates. -func TestCoverageCatalogSemantics(t *testing.T) { - c, err := LoadCatalog(filepath.Join("..", "queries", "ducklake_posthog_coverage.yaml")) - if err != nil { - t.Fatal(err) - } - if len(c.Queries) != 36 { - t.Fatalf("got %d variants, want 12 complete triples", len(c.Queries)) - } - db, err := sql.Open("duckdb", "") - if err != nil { - t.Fatal(err) - } - defer func() { - if err := db.Close(); err != nil { - t.Errorf("close database: %v", err) - } - }() - _, err = db.Exec(`CREATE SCHEMA posthog; - CREATE TABLE posthog.events(team_id BIGINT, person_id VARCHAR, event VARCHAR, timestamp TIMESTAMPTZ, uuid VARCHAR); - CREATE TABLE posthog.persons(team_id BIGINT, id VARCHAR, is_identified BOOLEAN); - INSERT INTO posthog.persons VALUES (1,'a',true),(1,'a',true),(1,'b',false),(2,'a',false); - INSERT INTO posthog.events VALUES - (1,'a','$autocapture','2026-03-17 00:00:00+00','1'), - (1,'a','$pageview','2026-03-17 00:10:00+00','2'), - (1,'a','$autocapture','2026-03-17 00:20:00+00','3'), - (1,'a','$pageview','2026-03-26 00:00:00+00','4'), - (1,'b','$pageview','2026-03-18 00:00:00+00','5'), - (2,'a','$pageview','2026-03-18 00:00:00+00','6'), - (1,NULL,'other','2026-03-18 00:00:00+00','7'), - (1,'c','$autocapture','2026-03-17 00:00:00+00','8'), - (1,'c','$pageview','2026-03-17 00:40:00+00','9');`) - if err != nil { - t.Fatal(err) - } - validation, err := os.ReadFile(filepath.Join("..", "..", "mw-dev", "scenario", "sql", "validate_posthog_coverage.sql")) - if err != nil { - t.Fatal(err) - } - if _, err := db.Exec(string(validation)); err != nil { - t.Fatalf("valid fixture rejected: %v", err) - } - defer func() { - if _, err := db.Exec("DELETE FROM posthog.persons"); err != nil { - t.Fatal(err) - } - if _, err := db.Exec(string(validation)); err == nil || !strings.Contains(err.Error(), "overlap") { - t.Fatalf("missing join population must fail: %v", err) - } - }() - - expected := map[string][]string{ - "q_coverage_daily_uniques_v1": {"2026-03-17 2", "2026-03-18 2", "2026-03-26 1"}, - "q_coverage_event_trends_v1": {"2026-03-17 $autocapture 3", "2026-03-17 $pageview 2", "2026-03-18 $pageview 2", "2026-03-26 $pageview 1"}, - "q_coverage_person_enrichment_v1": {"false 2", "true 3"}, - "q_coverage_person_activity_join_v1": {"3 5"}, - "q_coverage_recent_events_v1": {"7 other ", "6 $pageview a", "5 $pageview b", "9 $pageview c", "3 $autocapture a", "2 $pageview a", "8 $autocapture c", "1 $autocapture a"}, - "q_coverage_active_actors_v1": {"1 a 3", "1 c 2", "1 b 1", "2 a 1"}, - "q_coverage_ordered_funnel_v1": {"4 1"}, - "q_coverage_weekly_retention_v1": {"4 1"}, - "q_coverage_repeat_activity_cohort_v1": {"1"}, - "q_coverage_exclusion_cohort_v1": {"2"}, - "q_coverage_daily_person_model_v1": {"4 7"}, - "q_coverage_session_model_v1": {"5 7"}, - } - for i := 0; i < len(c.Queries); i += 3 { - q := c.Queries[i] - base := q.QueryID[:len(q.QueryID)-len("__ducklake_table")] - t.Run(base, func(t *testing.T) { - want, ok := expected[base] - if !ok { - t.Fatalf("missing semantic expectation for %s", base) - } - rows, err := db.Query(q.PGWireSQL) - if err != nil { - t.Fatal(err) - } - defer func() { - if err := rows.Close(); err != nil { - t.Errorf("close rows: %v", err) - } - }() - cols, err := rows.Columns() - if err != nil { - t.Fatal(err) - } - var got []string - for rows.Next() { - values := make([]any, len(cols)) - pointers := make([]any, len(cols)) - for j := range values { - pointers[j] = &values[j] - } - if err := rows.Scan(pointers...); err != nil { - t.Fatal(err) - } - line := "" - for j, v := range values { - if j > 0 { - line += " " - } - line += fmt.Sprint(v) - } - got = append(got, line) - } - if err := rows.Err(); err != nil { - t.Fatal(err) - } - if fmt.Sprint(got) != fmt.Sprint(want) { - t.Fatalf("got %v, want %v", got, want) - } - }) - } -} diff --git a/tests/perf/core/suite_test.go b/tests/perf/core/suite_test.go deleted file mode 100644 index efa2ccc7d..000000000 --- a/tests/perf/core/suite_test.go +++ /dev/null @@ -1,16 +0,0 @@ -package core - -import "testing" - -func TestValidateSuite(t *testing.T) { - for _, suite := range []string{"tables", "properties", "coverage"} { - if err := ValidateSuite(suite); err != nil { - t.Errorf("suite %q: %v", suite, err) - } - } - for _, suite := range []string{"", "covergae", "Tables"} { - if err := ValidateSuite(suite); err == nil { - t.Errorf("accepted unknown suite %q", suite) - } - } -} diff --git a/tests/perf/publisher/publisher_test.go b/tests/perf/publisher/publisher_test.go index 0f639285d..3b422e6f0 100644 --- a/tests/perf/publisher/publisher_test.go +++ b/tests/perf/publisher/publisher_test.go @@ -203,50 +203,45 @@ func TestPublishArtifactsBootstrapsAndReplacesRunData(t *testing.T) { } func TestPublishArtifactsWritesSuiteAndFixtureVersion(t *testing.T) { - for _, suite := range []string{"properties", "coverage"} { - t.Run(suite, func(t *testing.T) { - runDir := t.TempDir() - writeCustomSummaryFile(t, runDir, map[string]any{ - "run_id": "scenario-dev-posthog-frozen-perf-1-" + suite, - "dataset_version": "posthog-file-views-v1", - "suite": suite, - "fixture_version": suite + "-sha256-abc", - "nightly_run_id": "scenario-dev-posthog-frozen-perf-1", - "started_at": "2026-03-11T23:43:03Z", - "finished_at": "2026-03-11T23:43:14Z", - }) - writeFixtureCSVFile(t, runDir) - artifacts, err := loadArtifacts(runDir) - if err != nil { - t.Fatalf("loadArtifacts returned error: %v", err) - } - db := &fakeDB{tx: &fakeTx{}} - if err := publishArtifacts(context.Background(), Config{Schema: "duckgres_perf"}, db, artifacts); err != nil { - t.Fatalf("publishArtifacts returned error: %v", err) - } - runInsert := db.tx.execs[2] - if runInsert.args[1] != "posthog-file-views-v1" || runInsert.args[8] != suite { - t.Fatalf("run dataset/suite = %#v/%#v", runInsert.args[1], runInsert.args[8]) - } - if got, ok := runInsert.args[9].(*string); !ok || got == nil || *got != suite+"-sha256-abc" { - t.Fatalf("run fixture_version = %#v", runInsert.args[9]) - } - if got := runInsert.args[10]; got != "scenario-dev-posthog-frozen-perf-1" { - t.Fatalf("run nightly_run_id = %#v", got) - } - for _, result := range db.tx.execs[3:] { - if result.args[11] != "posthog-file-views-v1" || result.args[15] != suite { - t.Fatalf("result dataset/suite = %#v/%#v", result.args[11], result.args[15]) - } - if got, ok := result.args[16].(*string); !ok || got == nil || *got != suite+"-sha256-abc" { - t.Fatalf("result fixture_version = %#v", result.args[16]) - } - if got := result.args[17]; got != "scenario-dev-posthog-frozen-perf-1" { - t.Fatalf("result nightly_run_id = %#v", got) - } - } - - }) + runDir := t.TempDir() + writeCustomSummaryFile(t, runDir, map[string]any{ + "run_id": "scenario-dev-posthog-frozen-perf-1-properties", + "dataset_version": "posthog-file-views-v1", + "suite": "properties", + "fixture_version": "properties-sha256-abc", + "nightly_run_id": "scenario-dev-posthog-frozen-perf-1", + "started_at": "2026-03-11T23:43:03Z", + "finished_at": "2026-03-11T23:43:14Z", + }) + writeFixtureCSVFile(t, runDir) + artifacts, err := loadArtifacts(runDir) + if err != nil { + t.Fatalf("loadArtifacts returned error: %v", err) + } + db := &fakeDB{tx: &fakeTx{}} + if err := publishArtifacts(context.Background(), Config{Schema: "duckgres_perf"}, db, artifacts); err != nil { + t.Fatalf("publishArtifacts returned error: %v", err) + } + runInsert := db.tx.execs[2] + if runInsert.args[1] != "posthog-file-views-v1" || runInsert.args[8] != core.SuiteProperties { + t.Fatalf("run dataset/suite = %#v/%#v", runInsert.args[1], runInsert.args[8]) + } + if got, ok := runInsert.args[9].(*string); !ok || got == nil || *got != "properties-sha256-abc" { + t.Fatalf("run fixture_version = %#v", runInsert.args[9]) + } + if got := runInsert.args[10]; got != "scenario-dev-posthog-frozen-perf-1" { + t.Fatalf("run nightly_run_id = %#v", got) + } + for _, result := range db.tx.execs[3:] { + if result.args[11] != "posthog-file-views-v1" || result.args[15] != core.SuiteProperties { + t.Fatalf("result dataset/suite = %#v/%#v", result.args[11], result.args[15]) + } + if got, ok := result.args[16].(*string); !ok || got == nil || *got != "properties-sha256-abc" { + t.Fatalf("result fixture_version = %#v", result.args[16]) + } + if got := result.args[17]; got != "scenario-dev-posthog-frozen-perf-1" { + t.Fatalf("result nightly_run_id = %#v", got) + } } } From c34216a288edfc25db1d2705334e287a6652010a Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 14:12:21 -0400 Subject: [PATCH 07/13] docs: exempt perftest coverage from added tests --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index f779ee1ff..bd396aaa2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,6 +16,7 @@ ## Testing +- Perftest query and coverage additions are exempt from the TDD requirement: do not add tests solely to test perftests. Only make minimal compatibility updates to existing tests when needed. - Tests should model realistic user behavior, not work around current service limitations. - If a test exposes a platform bug, report the bug with exact failure details instead of fixing the test. From 6bb67c4067b17736f40d284758d4ec66e05b1b25 Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 14:16:55 -0400 Subject: [PATCH 08/13] refactor(perf): name manual scenario extended --- .github/workflows/scenario-dev.yml | 8 ++++---- docs/runbooks/perf-coverage.md | 8 ++++---- tests/mw-dev/run.sh | 8 ++++---- ...rf_coverage.yaml => posthog_frozen_perf_extended.yaml} | 6 +++--- ...trino.yaml => posthog_frozen_perf_extended_trino.yaml} | 4 ++-- ...ed.yaml => posthog_frozen_perf_extended_uncached.yaml} | 6 +++--- .../coverage-v1-duckdb-uncached-2026-09-25/README.md | 4 +++- 7 files changed, 23 insertions(+), 21 deletions(-) rename tests/mw-dev/scenario/scenarios/{posthog_frozen_perf_coverage.yaml => posthog_frozen_perf_extended.yaml} (95%) rename tests/mw-dev/scenario/scenarios/{posthog_frozen_perf_coverage_trino.yaml => posthog_frozen_perf_extended_trino.yaml} (97%) rename tests/mw-dev/scenario/scenarios/{posthog_frozen_perf_coverage_uncached.yaml => posthog_frozen_perf_extended_uncached.yaml} (93%) diff --git a/.github/workflows/scenario-dev.yml b/.github/workflows/scenario-dev.yml index 8985c937b..fc8bd6005 100644 --- a/.github/workflows/scenario-dev.yml +++ b/.github/workflows/scenario-dev.yml @@ -9,7 +9,7 @@ on: default: full-suite type: string coverage_target: - description: Manual extended coverage target for posthog_frozen_perf_coverage; excluded from nightly + description: Manual extended coverage target for posthog_frozen_perf_extended; excluded from nightly required: false default: pgwire_uncached type: choice @@ -88,7 +88,7 @@ jobs: # Frozen perf uses a namespace-local Trino cell. The dedicated role is # consumed only when E2E_SUITE selects that isolated deployment. TRINO_POD_IDENTITY_ROLE: ${{ secrets.MW_DEV_TRINO_POD_IDENTITY_ROLE }} - E2E_SUITE: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_coverage' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) && 'trino' || 'neutral' }} + E2E_SUITE: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_extended' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) && 'trino' || 'neutral' }} DUCKGRES_SCENARIO_COVERAGE_TARGET: ${{ inputs.coverage_target || 'pgwire_uncached' }} PR_NUMBER: ${{ github.run_id }} NAMESPACE: duckgres-ci-pr-${{ github.run_id }} @@ -101,7 +101,7 @@ jobs: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - name: Require a current image for cached comparisons - if: ${{ inputs.duckgres_image != '' && (inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_coverage' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) }} + if: ${{ inputs.duckgres_image != '' && (inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_extended' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) }} run: | echo "::error::Cached Trino comparisons require the control-plane image built from this revision. Omit duckgres_image so an older image cannot silently ignore the cache flag." exit 1 @@ -156,7 +156,7 @@ jobs: run: aws eks update-kubeconfig --name "$CLUSTER_NAME" --region "$AWS_REGION" --alias "$KUBE_CONTEXT" - name: Load frozen perf identity and Athena configuration - if: env.SCENARIO_NAME == 'posthog_frozen_perf' || env.SCENARIO_NAME == 'posthog_frozen_perf_coverage' || env.SCENARIO_NAME == 'posthog_frozen_perf_coverage_uncached' + if: env.SCENARIO_NAME == 'posthog_frozen_perf' || env.SCENARIO_NAME == 'posthog_frozen_perf_extended' || env.SCENARIO_NAME == 'posthog_frozen_perf_extended_uncached' run: bash scripts/scenario_athena_config.sh >> "$GITHUB_ENV" - name: Load properties fixture override diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md index 217b60523..e3e3575e5 100644 --- a/docs/runbooks/perf-coverage.md +++ b/docs/runbooks/perf-coverage.md @@ -16,22 +16,22 @@ not share one four-hour scenario budget: ```sh gh workflow run scenario-dev.yml --ref \ - -f scenario=posthog_frozen_perf_coverage \ + -f scenario=posthog_frozen_perf_extended \ -f coverage_target=pgwire_uncached ``` `coverage_target` defaults to `pgwire_uncached`; supported choices are `pgwire_uncached`, `pgwire_cached`, `trino`, `trino_cached`, and `athena`. Dispatch once per configuration. Trino targets select the declarative -`posthog_frozen_perf_coverage_trino.yaml` backing scenario and deploy the Trino +`posthog_frozen_perf_extended_trino.yaml` backing scenario and deploy the Trino lane. Other targets use the neutral lane. Both reuse the existing frozen setup; Athena configuration also supplies the frozen runner identity. -The original focused baseline entry point remains unchanged: +For the focused uncached DuckDB baseline: ```sh gh workflow run scenario-dev.yml --ref \ - -f scenario=posthog_frozen_perf_coverage_uncached + -f scenario=posthog_frozen_perf_extended_uncached ``` Each coverage dispatch executes all twelve queries with one warmup and four diff --git a/tests/mw-dev/run.sh b/tests/mw-dev/run.sh index 96a62c664..1da92126c 100755 --- a/tests/mw-dev/run.sh +++ b/tests/mw-dev/run.sh @@ -63,7 +63,7 @@ esac # it to a digest pin when TRINO_IMAGE is unset. Regular Trino tests keep a # promoted pin because they require atomic writes. TRINO_MASTER_IMAGE_RESOLVER="${TRINO_MASTER_IMAGE_RESOLVER:-$HERE/../../scripts/resolve_trino_master_image.sh}" -if [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage ]; then +if [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_extended ]; then TRINO_IMAGE="${TRINO_IMAGE:-}" else TRINO_IMAGE="${TRINO_IMAGE:-ghcr.io/posthog/trino:86468a7955788b90fe2072f80d86d548972ff28b@sha256:64927a71d2870802a56b671828c6052e7aa37317a7c3a50bd50a93960402d67b}" @@ -130,7 +130,7 @@ require_pr_identity() { } frozen_perf_scenario() { - [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage_uncached ] + [ "$SCENARIO_NAME" = posthog_frozen_perf ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_extended ] || [ "$SCENARIO_NAME" = posthog_frozen_perf_extended_uncached ] } hoglake_perf_enabled() { @@ -790,10 +790,10 @@ cmd_test_scenario() { esac local backing_scenario="$SCENARIO_NAME" - if [ "$SCENARIO_NAME" = posthog_frozen_perf_coverage ]; then + if [ "$SCENARIO_NAME" = posthog_frozen_perf_extended ]; then case "${DUCKGRES_SCENARIO_COVERAGE_TARGET:-}" in pgwire_uncached|pgwire_cached|athena) ;; - trino|trino_cached) backing_scenario=posthog_frozen_perf_coverage_trino ;; + trino|trino_cached) backing_scenario=posthog_frozen_perf_extended_trino ;; *) echo "DUCKGRES_SCENARIO_COVERAGE_TARGET must select one coverage protocol" >&2; return 2 ;; esac fi diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended.yaml similarity index 95% rename from tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage.yaml rename to tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended.yaml index d8c0ab5aa..a3c0f0f57 100644 --- a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage.yaml +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended.yaml @@ -1,5 +1,5 @@ -name: posthog-frozen-perf-coverage -run_id_prefix: scenario-frozen-coverage +name: posthog-frozen-perf-extended +run_id_prefix: scenario-frozen-extended required_env: - DUCKGRES_SCENARIO_COVERAGE_TARGET - DUCKGRES_SCENARIO_ATHENA_REGION @@ -16,7 +16,7 @@ steps: with: org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} request: - database_name: scenario-frozen-coverage-${run_id_token} + database_name: scenario-frozen-extended-${run_id_token} team_id: 1 metadata_store: type: cnpg-shard diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_trino.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended_trino.yaml similarity index 97% rename from tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_trino.yaml rename to tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended_trino.yaml index 004f35cea..4eb48a6aa 100644 --- a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_trino.yaml +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended_trino.yaml @@ -1,5 +1,5 @@ -name: posthog-frozen-perf-coverage-trino -run_id_prefix: scenario-frozen-coverage-trino +name: posthog-frozen-perf-extended-trino +run_id_prefix: scenario-frozen-extended-trino required_env: - DUCKGRES_SCENARIO_COVERAGE_TARGET - DUCKGRES_SCENARIO_ORG_ID diff --git a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_uncached.yaml b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended_uncached.yaml similarity index 93% rename from tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_uncached.yaml rename to tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended_uncached.yaml index a876ae6dc..aee9b90e5 100644 --- a/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_coverage_uncached.yaml +++ b/tests/mw-dev/scenario/scenarios/posthog_frozen_perf_extended_uncached.yaml @@ -1,5 +1,5 @@ -name: posthog-frozen-perf-coverage-uncached -run_id_prefix: scenario-frozen-coverage-uncached +name: posthog-frozen-perf-extended-uncached +run_id_prefix: scenario-frozen-extended-uncached required_env: - DUCKGRES_SCENARIO_ORG_ID - DUCKGRES_SCENARIO_FROZEN_S3_URI @@ -11,7 +11,7 @@ steps: with: org_id: ${env:DUCKGRES_SCENARIO_ORG_ID} request: - database_name: scenario-frozen-coverage-uncached-${run_id_token} + database_name: scenario-frozen-extended-uncached-${run_id_token} team_id: 1 metadata_store: type: cnpg-shard diff --git a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md index d54233993..1cff5a8f9 100644 --- a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md +++ b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md @@ -3,7 +3,9 @@ This is the initial comparison base for the twelve coverage v1 intents on `pgwire_uncached`. [Successful source run](https://github.com/PostHog/duckgres/actions/runs/36159233201) measured commit `9c9f5f305f87b7f0d41c0b54d64e5648aeb8ef83` on 2026-09-25. -The focused scenario and catalog are unchanged by subsequent launch-path edits. +The catalog and query settings match the measured revision. The focused entry +point was subsequently renamed to `posthog_frozen_perf_extended_uncached`; the +manifest preserves the original scenario name used by the source run. [Samples](samples.csv) retain all 48 successful measurements; [manifest](manifest.json) records the catalog hash, dataset version, cache From a2d8e1a12b238e3ccaaf64945276108084c235cc Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Fri, 25 Sep 2026 14:40:19 -0400 Subject: [PATCH 09/13] feat(perf): default extended runs to isolated target jobs --- .github/workflows/scenario-dev.yml | 90 +++++++++++++------ docs/runbooks/perf-coverage.md | 68 +++++++++----- tests/mw-dev/scenario/script_test.go | 4 +- .../README.md | 10 ++- 4 files changed, 116 insertions(+), 56 deletions(-) diff --git a/.github/workflows/scenario-dev.yml b/.github/workflows/scenario-dev.yml index fc8bd6005..9fe3e4a12 100644 --- a/.github/workflows/scenario-dev.yml +++ b/.github/workflows/scenario-dev.yml @@ -9,11 +9,11 @@ on: default: full-suite type: string coverage_target: - description: Manual extended coverage target for posthog_frozen_perf_extended; excluded from nightly + description: Extended target; all runs each configuration in a separate job (manual only) required: false - default: pgwire_uncached + default: all type: choice - options: [pgwire_uncached, pgwire_cached, trino, trino_cached, athena] + options: [all, pgwire_uncached, pgwire_cached, trino, trino_cached, athena] properties_s3_uri: description: Properties Parquet S3 prefix override; blank uses the existing Athena table location required: false @@ -41,6 +41,51 @@ concurrency: cancel-in-progress: false jobs: + plan: + runs-on: ubuntu-24.04 + outputs: + matrix: ${{ steps.targets.outputs.matrix }} + trino_image: ${{ steps.trino.outputs.image }} + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + - name: Select scenario targets + id: targets + env: + EVENT_NAME: ${{ github.event_name }} + run: | + python3 - <<'PYTHON' + import json, os + with open(os.environ["GITHUB_EVENT_PATH"]) as event_file: + inputs = json.load(event_file).get("inputs") or {} + scenario = "posthog_frozen_perf" if os.environ["EVENT_NAME"] == "schedule" else inputs.get("scenario", "") + targets = ["pgwire_uncached", "pgwire_cached", "trino", "trino_cached", "athena"] + if scenario == "posthog_frozen_perf_extended": + selected = inputs.get("coverage_target") or "all" + if selected not in ["all", *targets]: + raise SystemExit("Unsupported extended coverage target") + jobs = [{"target": target, "id_suffix": str(i + 1), "artifact_suffix": "-" + target} + for i, target in enumerate(targets) if selected in ("all", target)] + else: + jobs = [{"target": "pgwire_uncached", "id_suffix": "", "artifact_suffix": ""}] + needs_trino = scenario == "posthog_frozen_perf" or ( + scenario == "posthog_frozen_perf_extended" and any(j["target"].startswith("trino") for j in jobs)) + if needs_trino and inputs.get("duckgres_image"): + raise SystemExit("Trino comparisons require the control-plane image built from this revision; omit duckgres_image") + with open(os.environ["GITHUB_OUTPUT"], "a") as output: + print("matrix=" + json.dumps({"include": jobs}, separators=(",", ":")), file=output) + print("needs_trino=" + str(needs_trino).lower(), file=output) + PYTHON + - name: Resolve shared Trino image + id: trino + if: steps.targets.outputs.needs_trino == 'true' + env: + TRINO_IMAGE_OVERRIDE: ${{ inputs.trino_image }} + run: | + set -euo pipefail + image="${TRINO_IMAGE_OVERRIDE:-$(scripts/resolve_trino_master_image.sh)}" + echo "image=$image" >> "$GITHUB_OUTPUT" + echo "Trino image: \`$image\`" >> "$GITHUB_STEP_SUMMARY" + scenario-runner-image: uses: ./.github/workflows/_image-build.yml with: @@ -71,10 +116,16 @@ jobs: ecr-role: ${{ vars.AWS_ECR_PRS_PUBLISH_IAM_ROLE }} scenario: - needs: [scenario-runner-image, duckgres-image] - if: ${{ always() && needs.scenario-runner-image.result == 'success' && (needs.duckgres-image.result == 'success' || (github.event_name == 'workflow_dispatch' && inputs.duckgres_image != '')) }} + name: ${{ github.event_name == 'schedule' && 'posthog_frozen_perf' || inputs.scenario }}${{ matrix.artifact_suffix }} + needs: [plan, scenario-runner-image, duckgres-image] + if: ${{ always() && needs.plan.result == 'success' && needs.scenario-runner-image.result == 'success' && (needs.duckgres-image.result == 'success' || (github.event_name == 'workflow_dispatch' && inputs.duckgres_image != '')) }} runs-on: ubuntu-24.04 timeout-minutes: 270 + strategy: + fail-fast: false + # Sequential targets avoid benchmark resource contention. Each gets a fresh timeout. + max-parallel: 1 + matrix: ${{ fromJSON(needs.plan.outputs.matrix) }} env: SCENARIO_NAME: ${{ github.event_name == 'schedule' && 'posthog_frozen_perf' || inputs.scenario }} SCENARIO_RUNNER_IMAGE: ${{ needs.scenario-runner-image.outputs.image }} @@ -88,10 +139,11 @@ jobs: # Frozen perf uses a namespace-local Trino cell. The dedicated role is # consumed only when E2E_SUITE selects that isolated deployment. TRINO_POD_IDENTITY_ROLE: ${{ secrets.MW_DEV_TRINO_POD_IDENTITY_ROLE }} - E2E_SUITE: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_extended' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) && 'trino' || 'neutral' }} - DUCKGRES_SCENARIO_COVERAGE_TARGET: ${{ inputs.coverage_target || 'pgwire_uncached' }} - PR_NUMBER: ${{ github.run_id }} - NAMESPACE: duckgres-ci-pr-${{ github.run_id }} + E2E_SUITE: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_extended' && (matrix.target == 'trino' || matrix.target == 'trino_cached'))) && 'trino' || 'neutral' }} + DUCKGRES_SCENARIO_COVERAGE_TARGET: ${{ matrix.target }} + PR_NUMBER: ${{ github.run_id }}${{ matrix.id_suffix }} + NAMESPACE: duckgres-ci-pr-${{ github.run_id }}${{ matrix.id_suffix }} + TRINO_IMAGE: ${{ needs.plan.outputs.trino_image }} DUCKGRES_SCENARIO_MAX_RUNTIME: 4h DUCKGRES_SCENARIO_GO_TEST_TIMEOUT: 4h15m # Add process headroom for repeated full-dataset pgwire aggregates. @@ -100,24 +152,6 @@ jobs: steps: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - - name: Require a current image for cached comparisons - if: ${{ inputs.duckgres_image != '' && (inputs.scenario == 'posthog_frozen_perf' || (inputs.scenario == 'posthog_frozen_perf_extended' && (inputs.coverage_target == 'trino' || inputs.coverage_target == 'trino_cached'))) }} - run: | - echo "::error::Cached Trino comparisons require the control-plane image built from this revision. Omit duckgres_image so an older image cannot silently ignore the cache flag." - exit 1 - - - name: Resolve Trino image - if: env.E2E_SUITE == 'trino' - env: - TRINO_IMAGE_OVERRIDE: ${{ inputs.trino_image }} - run: | - set -euo pipefail - # Frozen perf follows PostHog/trino master; resolve once so every - # run.sh step deploys and reports the same digest. - image="${TRINO_IMAGE_OVERRIDE:-$(scripts/resolve_trino_master_image.sh)}" - echo "TRINO_IMAGE=$image" >> "$GITHUB_ENV" - echo "Trino image: \`$image\`" >> "$GITHUB_STEP_SUMMARY" - - name: Scenario dev target selected run: echo "Deploying an isolated Duckgres stack for $SCENARIO_NAME." @@ -206,7 +240,7 @@ jobs: if: always() uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: scenario-dev-${{ env.SCENARIO_NAME }}-${{ github.run_id }}-${{ github.run_attempt }} + name: scenario-dev-${{ env.SCENARIO_NAME }}-${{ github.run_id }}-${{ github.run_attempt }}${{ matrix.artifact_suffix }} path: artifacts/scenario-dev/ if-no-files-found: warn retention-days: 14 diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md index e3e3575e5..a26932a2a 100644 --- a/docs/runbooks/perf-coverage.md +++ b/docs/runbooks/perf-coverage.md @@ -11,21 +11,39 @@ the SELECT computation, not writes or a complete materialization job. The regular `posthog_frozen_perf` nightly retains the original table and properties workloads. The twelve additional queries are an **extended, manual-only run**: scheduled nightly runs never include them. Start them explicitly through -`workflow_dispatch`, one configuration per dispatch, so five configurations do -not share one four-hour scenario budget: +`workflow_dispatch`. This command runs all five configurations: ```sh gh workflow run scenario-dev.yml --ref \ - -f scenario=posthog_frozen_perf_extended \ - -f coverage_target=pgwire_uncached + -f scenario=posthog_frozen_perf_extended ``` -`coverage_target` defaults to `pgwire_uncached`; supported choices are -`pgwire_uncached`, `pgwire_cached`, `trino`, `trino_cached`, and `athena`. -Dispatch once per configuration. Trino targets select the declarative -`posthog_frozen_perf_extended_trino.yaml` backing scenario and deploy the Trino -lane. Other targets use the neutral lane. Both reuse the existing frozen setup; -Athena configuration also supplies the frozen runner identity. +`coverage_target` defaults to `all`. To select one configuration, add +`-f coverage_target=pgwire_uncached`, `pgwire_cached`, `trino`, `trino_cached`, +or `athena`. Each selected configuration gets its own job, isolated deployment, +and artifact. Jobs execute sequentially to avoid benchmark resource contention; +a failed target does not cancel the others. Images are built once, and both +Trino configurations use the same resolved image digest. + +Trino targets select the declarative `posthog_frozen_perf_extended_trino.yaml` +backing scenario and deploy the Trino lane. Other targets use the neutral lane. +Both reuse the existing frozen setup; Athena configuration also supplies the +frozen runner identity. + +### Timeouts + +Each target has a four-hour scenario limit and a 270-minute workflow job limit. +The Go process limit is 4h15m, leaving time for artifact collection and cleanup. +These limits apply independently to each job, not to the sum of all target jobs. +A single target can still time out; peer runtimes have not yet been measured. +The measured uncached DuckDB coverage phase took 87m06s. + +Running all five targets sequentially inside one job would likely time out: +the equal-speed query-work estimate alone is 7h16m. The matrix avoids that +combined timeout without reducing queries or repetitions. Under the same +planning assumptions and shared initial builds, allow roughly **8h16m for the +extended workflow alone**, excluding queue delays. Standard nightly queries +are not included; running them as well brings the estimate to about **9h19m**. For the focused uncached DuckDB baseline: @@ -34,18 +52,18 @@ gh workflow run scenario-dev.yml --ref \ -f scenario=posthog_frozen_perf_extended_uncached ``` -Each coverage dispatch executes all twelve queries with one warmup and four +Each target job executes all twelve queries with one warmup and four measured repetitions, sequentially, using the same frozen files. The explicit 3 CPU / 12 GiB settings describe DuckDB workers, not equivalent resources across engines. "Uncached" disables the DuckDB external file cache; it does not guarantee cold OS or storage caches. See [cache settings](../../tests/perf/README.md#duckdb-cache-comparison). -Independent dispatches have different run and nightly IDs. Compare matching -recorded commits, catalog versions, fixture versions, resource settings, and -methodology. Pin the same `trino_image` digest for both Trino configurations. -The workflow serializes dispatches on the same branch; complete one before -starting the next. +Target jobs have distinct benchmark run and nightly IDs. Group an all-target +campaign by its parent workflow URL and target-suffixed artifacts. Compare +matching recorded commits, catalog versions, dataset versions, resource settings, +and methodology. Across separate workflow dispatches, pin the same `trino_image` +digest when comparing Trino configurations. Artifacts live in `perf-coverage/`, with a `-coverage` run ID suffix. The original `perf/` and `perf-properties/` histories are preserved. Main-branch @@ -67,7 +85,9 @@ The initial uncached DuckDB comparison base is the [2026-09-25 coverage v1 baseline](../../tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md). It retains 48 successful measurements and the complete methodology. The query phase took 87m06s; the workflow took 1h42m45s. The equal-speed expanded campaign -estimate is about 9h37m across separate target dispatches plus the original suite. +estimate recorded with that baseline was 9h37m across separate dispatches plus +the original suite. The current shared-build matrix lowers that planning estimate +to about 9h19m; this is not a new measured result. Keep the successful focused run URL, tested commit, catalog version/hash, @@ -93,16 +113,18 @@ five configurations, the equal-speed estimate is: ```text additional query time = sum(uncached means) * 5 repetitions * 5 configurations full campaign estimate = original workflow time + additional query time - + sum(overhead for each coverage dispatch) + + shared build time + sum(non-build overhead for each target job) ``` Report this explicitly as an equal-speed estimate, not a forecast of engine performance. Include a sensitivity range for the other four configurations. -Each independent coverage dispatch repeats build, deployment, fixture validation, -collection, and teardown. Estimate that overhead per dispatch; do not count it -as query time. A hypothetical single sequential workflow would share setup, -but is not the supported launch method and may exceed the four-hour scenario -limit. +The workflow builds images once; each target job repeats deployment, fixture +validation, collection, and teardown. Estimate those separately from query work. +For the initial baseline, the workflow's initial build phase was approximately +272 seconds and total time outside the query phase was 939.50 seconds. Using +`272 + 5 * (939.50 - 272)` for shared-build overhead gives the approximate +8h16m extended-only estimate above. This assumes other engines' setup and warmup +costs match the focused DuckDB run; actual costs can differ. Use the measured coverage phase wall time to check the sum-based estimate, because warmups and harness overhead are not in `query_results.csv`. diff --git a/tests/mw-dev/scenario/script_test.go b/tests/mw-dev/scenario/script_test.go index 63e45b169..ead1dde86 100644 --- a/tests/mw-dev/scenario/script_test.go +++ b/tests/mw-dev/scenario/script_test.go @@ -76,8 +76,8 @@ func TestDevScenarioWorkflowPreservesIsolationCleanupAndPublishing(t *testing.T) workflow := string(raw) for _, required := range []string{ - "PR_NUMBER: ${{ github.run_id }}", - "NAMESPACE: duckgres-ci-pr-${{ github.run_id }}", + "PR_NUMBER: ${{ github.run_id }}${{ matrix.id_suffix }}", + "NAMESPACE: duckgres-ci-pr-${{ github.run_id }}${{ matrix.id_suffix }}", "- name: Teardown\n if: always()\n run: tests/mw-dev/run.sh teardown", "github.ref == 'refs/heads/main'", "--connection-secret-stdin", diff --git a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md index 1cff5a8f9..dcdf91b65 100644 --- a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md +++ b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md @@ -36,9 +36,9 @@ result values or internal fixture locations are retained. | Daily-person model | 22.81 | 23.04 | 20.99–24.17 | | Session model | 53.31 | 52.58 | 51.32–56.74 | -## Expanded campaign estimate +## Original separate-dispatch estimate -Keep the original nightly suite separate and dispatch coverage once per +The original planning estimate assumed a separate workflow dispatch per configuration. With one warmup plus four measurements, equal query speeds would add **7h16m of query work** across five configurations. Using the measured 15m39s overhead for each dispatch adds another **1h18m**. Adding the @@ -57,7 +57,11 @@ Holding DuckDB uncached fixed and varying the other four configurations: These are scenarios, not statistical confidence bounds. Even the equal-speed query work exceeds the four-hour scenario limit, which is why coverage uses -separate target dispatches. This baseline does not establish Trino or Athena +separate target jobs. The current workflow builds images once and runs targets +as sequential matrix jobs, reducing the equal-speed planning estimate to +approximately **8h16m extended-only**, or **9h19m including the original suite**. +The manifest retains the original estimate for provenance; see the runbook for +the shared-build calculation. This baseline does not establish Trino or Athena performance. Resource requests are 3 CPU / 12Gi for DuckDB workers only. ## Future comparisons From d4bb231020bbaaf0e78e54e992bbbd4cb5b83eb5 Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Mon, 28 Sep 2026 09:55:41 -0400 Subject: [PATCH 10/13] perf: match larger Trino and DuckDB worker budgets --- .github/workflows/scenario-dev.yml | 10 +++- docs/runbooks/perf-coverage.md | 46 +++++++++++++-- tests/mw-dev/manifests.tmpl.yaml | 4 +- tests/mw-dev/manifests.trino.tmpl.yaml | 25 ++++----- tests/mw-dev/run.sh | 56 ++++++++++++++----- tests/mw-dev/run_sh_test.go | 7 +++ tests/perf/README.md | 7 ++- .../README.md | 6 +- 8 files changed, 121 insertions(+), 40 deletions(-) diff --git a/.github/workflows/scenario-dev.yml b/.github/workflows/scenario-dev.yml index 9fe3e4a12..d86a66d84 100644 --- a/.github/workflows/scenario-dev.yml +++ b/.github/workflows/scenario-dev.yml @@ -146,9 +146,13 @@ jobs: TRINO_IMAGE: ${{ needs.plan.outputs.trino_image }} DUCKGRES_SCENARIO_MAX_RUNTIME: 4h DUCKGRES_SCENARIO_GO_TEST_TIMEOUT: 4h15m - # Add process headroom for repeated full-dataset pgwire aggregates. - DUCKGRES_K8S_WORKER_CPU_REQUEST: "3" - DUCKGRES_K8S_WORKER_MEMORY_REQUEST: 12Gi + # Match the benchmark Trino fleet: three 16-CPU / 64Gi workers. + # Non-benchmark scenarios retain their existing small resource budget. + DUCKGRES_K8S_WORKER_CPU_REQUEST: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '48' || '3' }} + DUCKGRES_K8S_WORKER_MEMORY_REQUEST: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '192Gi' || '12Gi' }} + # The perf driver passes explicit sizing GUCs: keep their admission caps aligned. + DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '48' || '8' }} + DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '192Gi' || '16Gi' }} steps: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md index a26932a2a..eabff5244 100644 --- a/docs/runbooks/perf-coverage.md +++ b/docs/runbooks/perf-coverage.md @@ -36,7 +36,8 @@ Each target has a four-hour scenario limit and a 270-minute workflow job limit. The Go process limit is 4h15m, leaving time for artifact collection and cleanup. These limits apply independently to each job, not to the sum of all target jobs. A single target can still time out; peer runtimes have not yet been measured. -The measured uncached DuckDB coverage phase took 87m06s. +These runtime estimates use the historical small-worker profile, not the larger +profile below. The measured uncached DuckDB coverage phase took 87m06s. Running all five targets sequentially inside one job would likely time out: the equal-speed query-work estimate alone is 7h16m. The matrix avoids that @@ -53,10 +54,45 @@ gh workflow run scenario-dev.yml --ref \ ``` Each target job executes all twelve queries with one warmup and four -measured repetitions, sequentially, using the same frozen files. The explicit -3 CPU / 12 GiB settings describe DuckDB workers, not equivalent resources across -engines. "Uncached" disables the DuckDB external file cache; it does not -guarantee cold OS or storage caches. See +measured repetitions, sequentially, using the same frozen files. + +### Worker sizing + +Standard frozen perf and extended perf use the following execution budgets: + +| Engine | Workers | CPU per worker | RAM per worker | JVM heap | Query memory per worker | +|---|---:|---:|---:|---:|---:| +| Trino (each cache mode) | 3 | 16 | 64 GiB | 48 GiB | 32 GiB | +| DuckDB (each cache mode) | 1 | 48 | 192 GiB | N/A | Derived by Duckgres from pod memory | + +Both have **48 CPU / 192 GiB of worker resources**, with requests equal to +limits. This compares aggregate execution resources; Trino coordinator and +support services are additional overhead. Athena capacity is service-managed. +Trino's cluster-wide query memory cap is 96 GiB. Its 48 GiB heap leaves 16 GiB +outside the JVM, and its 32 GiB per-node query cap leaves room for JVM headroom. +This is a chosen production-oriented profile consistent with +[Trino's guidance](https://trino.io/docs/current/installation/deployment.html#jvm-config), +not a universal prescribed CPU/RAM size. + +DuckDB's client sizing caps are raised alongside its worker resources so the +explicit benchmark profile is not silently clamped. At 192 GiB, the current +Duckgres headroom policy sets DuckDB's memory limit to 144 GiB; engine memory +limits differ even though container resources match. Other scenario types and +the auxiliary Trino bootstrap cluster retain their small defaults. + +The benchmark node pool must admit nodes large enough for a 48-CPU / 192Gi +DuckDB pod and 16-CPU / 64Gi Trino pods, with additional allocatable capacity for +system services. Deploy the [node-size allowlist change](https://github.com/PostHog/charts/pull/16434) +before running this profile. If pods remain Pending, inspect scheduling events and +node-pool constraints rather than counting setup failure as query latency. + +The retained September 25 baseline used the old 3 CPU / 12 GiB aggregate +profile. Keep it as historical data; it is **not a like-for-like baseline** for +these larger workers. Runtime estimates above also describe the old profile; +measure a fresh run before setting a new baseline or runtime forecast. + +"Uncached" disables the DuckDB external file cache; it does not guarantee cold +OS or storage caches. See [cache settings](../../tests/perf/README.md#duckdb-cache-comparison). Target jobs have distinct benchmark run and nightly IDs. Group an all-target diff --git a/tests/mw-dev/manifests.tmpl.yaml b/tests/mw-dev/manifests.tmpl.yaml index eef280e28..8c86a33b2 100644 --- a/tests/mw-dev/manifests.tmpl.yaml +++ b/tests/mw-dev/manifests.tmpl.yaml @@ -363,9 +363,9 @@ spec: - { name: DUCKGRES_EXPLORATORY_WORKER_MEMORY, value: "2Gi" } - { name: DUCKGRES_EXPLORATORY_WORKER_TTL, value: "10m" } - { name: DUCKGRES_K8S_WORKER_PROFILE_MIN_CPU, value: "1" } - - { name: DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU, value: "8" } + - { name: DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU, value: "${DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU}" } - { name: DUCKGRES_K8S_WORKER_PROFILE_MIN_MEMORY, value: "2Gi" } - - { name: DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY, value: "16Gi" } + - { name: DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY, value: "${DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY}" } # Prod-logic parity (sizes differ, the knobs match prod-us): # - workerQueueTimeout bounds how long a connection waits for its # on-demand cold spawn (binary default 60s is too tight for a diff --git a/tests/mw-dev/manifests.trino.tmpl.yaml b/tests/mw-dev/manifests.trino.tmpl.yaml index 0b91caf8f..b571933bc 100644 --- a/tests/mw-dev/manifests.trino.tmpl.yaml +++ b/tests/mw-dev/manifests.trino.tmpl.yaml @@ -90,9 +90,9 @@ data: http-server.https.keystore.path=/etc/trino/tls/keystore.p12 http-server.https.keystore.key=${ENV:TRINO_TLS_KEYSTORE_PASSWORD} discovery.uri=http://duckgres-trino.${NAMESPACE}.svc:8080 - query.max-memory=6GB + query.max-memory=${TRINO_QUERY_MEMORY} # The coordinator does not execute tasks and retains headroom in its 2G - # heap; execution workers carry the 2GB per-node allowance below. + # heap; execution workers use the selected profile below. query.max-memory-per-node=1GB catalog.management=dynamic # Both targets load the same managers; catalog fs.cache.enabled gates use. @@ -134,7 +134,7 @@ data: node.data-dir=/data/trino jvm.config: | -server - -Xmx3G + -Xmx${TRINO_WORKER_HEAP} -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError -Dfile.encoding=UTF-8 @@ -150,12 +150,11 @@ data: coordinator=false http-server.http.port=8080 discovery.uri=http://duckgres-trino.${NAMESPACE}.svc:8080 - query.max-memory=6GB - query.max-memory-per-node=2GB - # 1-CPU workers: raised from 8/16 to overlap S3 latency (keep task.concurrency default). - # Experiment measured by the targeted frozen perf run; revert if events by-name uncached does not improve. - task.max-worker-threads=24 - task.min-drivers=48 + query.max-memory=${TRINO_QUERY_MEMORY} + query.max-memory-per-node=${TRINO_WORKER_QUERY_MEMORY} + # Worker parallelism follows the rendering profile; task.concurrency stays default. + task.max-worker-threads=${TRINO_WORKER_THREADS} + task.min-drivers=${TRINO_WORKER_MIN_DRIVERS} shutdown.grace-period=10s catalog.management=dynamic # Both targets load the same managers; catalog fs.cache.enabled gates use. @@ -319,11 +318,9 @@ spec: initialDelaySeconds: 20 periodSeconds: 5 resources: - requests: { cpu: "1", memory: 4Gi } - # Three replicas provide the same aggregate 3-CPU/12Gi execution - # budget as the frozen-perf Duckgres worker. Keep requests equal - # to limits so node contention cannot change the comparison. - limits: { cpu: "1", memory: 4Gi } + requests: { cpu: "${TRINO_WORKER_CPU}", memory: "${TRINO_WORKER_MEMORY}" } + # Requests equal limits for the selected three-worker profile. + limits: { cpu: "${TRINO_WORKER_CPU}", memory: "${TRINO_WORKER_MEMORY}" } volumeMounts: - { name: config, mountPath: /etc/trino/node.properties, subPath: node.properties } - { name: config, mountPath: /etc/trino/jvm.config, subPath: jvm.config } diff --git a/tests/mw-dev/run.sh b/tests/mw-dev/run.sh index 1da92126c..7fb0d910f 100755 --- a/tests/mw-dev/run.sh +++ b/tests/mw-dev/run.sh @@ -30,6 +30,8 @@ SCENARIO_NAME="${SCENARIO_NAME:-full-suite}" SCENARIO_ARTIFACTS_DIR="${SCENARIO_ARTIFACTS_DIR:-$HERE/../../artifacts/scenario-dev}" DUCKGRES_K8S_WORKER_CPU_REQUEST="${DUCKGRES_K8S_WORKER_CPU_REQUEST:-750m}" DUCKGRES_K8S_WORKER_MEMORY_REQUEST="${DUCKGRES_K8S_WORKER_MEMORY_REQUEST:-1536Mi}" +DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU="${DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU:-8}" +DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY="${DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY:-16Gi}" E2E_SUITE="${E2E_SUITE:-neutral}" case "$E2E_SUITE" in neutral|duckdb|trino|reshard) ;; @@ -141,6 +143,33 @@ trino_multicell_enabled() { [ "$E2E_SUITE" = trino ] && [ "$SCENARIO_NAME" = full-suite ] } +# Bootstrap and general E2E keep the small fleet. Only comparison clusters +# use production-sized workers; both cache modes use the same profile. +render_trino_template() { + local TRINO_WORKER_CPU=1 TRINO_WORKER_MEMORY=4Gi TRINO_WORKER_HEAP=3G + local TRINO_QUERY_MEMORY=6GB TRINO_WORKER_QUERY_MEMORY=2GB + local TRINO_WORKER_THREADS=24 TRINO_WORKER_MIN_DRIVERS=48 + if [ "${1:-}" = perf ]; then + TRINO_WORKER_CPU=16 TRINO_WORKER_MEMORY=64Gi TRINO_WORKER_HEAP=48G + TRINO_QUERY_MEMORY=96GB TRINO_WORKER_QUERY_MEMORY=32GB + TRINO_WORKER_THREADS=32 TRINO_WORKER_MIN_DRIVERS=64 + fi + export TRINO_WORKER_CPU TRINO_WORKER_MEMORY TRINO_WORKER_HEAP + export TRINO_QUERY_MEMORY TRINO_WORKER_QUERY_MEMORY TRINO_WORKER_THREADS TRINO_WORKER_MIN_DRIVERS + envsubst '$NAMESPACE $PR_NUMBER $TRINO_IMAGE $TRINO_TLS_PASSWORD $TRINO_CA_CERT_B64 $TRINO_SERVER_P12_B64 $CONFIG_STORE_PASSWORD $TRINO_WORKER_CPU $TRINO_WORKER_MEMORY $TRINO_WORKER_HEAP $TRINO_QUERY_MEMORY $TRINO_WORKER_QUERY_MEMORY $TRINO_WORKER_THREADS $TRINO_WORKER_MIN_DRIVERS' < "$HERE/manifests.trino.tmpl.yaml" +} + +trino_perf_modes() { + if [ "$SCENARIO_NAME" = posthog_frozen_perf_extended ]; then + case "${DUCKGRES_SCENARIO_COVERAGE_TARGET:-}" in + trino) echo perf ;; + trino_cached) echo cached ;; + esac + else + echo 'perf cached' + fi +} + render_trino_backend() { local color="$1" local forwarded_config="" @@ -152,8 +181,7 @@ render_trino_backend() { TRINO_IMAGE="$TRINO_IMAGE" TRINO_TLS_PASSWORD="$TRINO_TLS_PASSWORD" \ CONFIG_STORE_PASSWORD="$(cat "$config_store_password_file")" \ NAMESPACE="$TRINO_CELL_NS" PR_NUMBER="$PR_NUMBER" \ - envsubst '$NAMESPACE $PR_NUMBER $TRINO_IMAGE $TRINO_TLS_PASSWORD $TRINO_CA_CERT_B64 $TRINO_SERVER_P12_B64 $CONFIG_STORE_PASSWORD' \ - < "$HERE/manifests.trino.tmpl.yaml" \ + render_trino_template \ | sed -e "s/duckgres-trino-coordinator/duckgres-trino-$color-coordinator/g" \ -e "s/duckgres-trino-worker/duckgres-trino-$color-worker/g" \ -e "s/labels: { app: duckgres-trino, component: coordinator }/labels: { app: duckgres-trino, component: coordinator, posthog.com\/trino-cell: cell-test, posthog.com\/trino-color: $color, app.kubernetes.io\/component: coordinator, app.kubernetes.io\/name: trino-coordinator }/" \ @@ -170,7 +198,7 @@ render_trino_backend() { -e "$forwarded_config" } -# Reuse the baseline template so both clusters retain identical resource budgets. +# Reuse the template with the larger comparison-worker profile for both modes. # Select only cluster-specific resources; TLS, auth, tenant secrets, OPA and the # config-store connection are intentionally shared in the throwaway namespace. render_trino_perf_cluster() { @@ -188,7 +216,7 @@ stringData: shared-secret: "$(cat "$cluster_secret_file")" EOF TRINO_IMAGE="$TRINO_IMAGE" NAMESPACE="$NS" PR_NUMBER="$PR_NUMBER" \ - envsubst '$NAMESPACE $PR_NUMBER $TRINO_IMAGE' < "$HERE/manifests.trino.tmpl.yaml" \ + render_trino_template perf \ | awk ' /^---$/ { if (selected) printf "%s", document; document="---\n"; selected=0; next } { document=document $0 "\n" } @@ -277,9 +305,11 @@ render() { USER_SECRET_KEY="$(cat "$user_secret_key_file")" \ NAMESPACE="$NS" PR_NUMBER="$PR_NUMBER" \ WORKER_IMAGE="$WORKER_IMAGE" CONTROLPLANE_IMAGE="$CONTROLPLANE_IMAGE" \ + DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU="$DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU" \ + DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY="$DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY" \ DUCKGRES_K8S_WORKER_CPU_REQUEST="$DUCKGRES_K8S_WORKER_CPU_REQUEST" \ DUCKGRES_K8S_WORKER_MEMORY_REQUEST="$DUCKGRES_K8S_WORKER_MEMORY_REQUEST" \ - envsubst '$NAMESPACE $PR_NUMBER $WORKER_IMAGE $CONTROLPLANE_IMAGE $INTERNAL_SECRET $INTERNAL_SECRET_FALLBACK $USER_SECRET_KEY $DUCKGRES_K8S_WORKER_CPU_REQUEST $DUCKGRES_K8S_WORKER_MEMORY_REQUEST $CONFIG_STORE_PASSWORD' \ + envsubst '$NAMESPACE $PR_NUMBER $WORKER_IMAGE $CONTROLPLANE_IMAGE $INTERNAL_SECRET $INTERNAL_SECRET_FALLBACK $USER_SECRET_KEY $DUCKGRES_K8S_WORKER_CPU_REQUEST $DUCKGRES_K8S_WORKER_MEMORY_REQUEST $DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU $DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY $CONFIG_STORE_PASSWORD' \ < "$HERE/manifests.tmpl.yaml" if [ "$E2E_SUITE" = "trino" ]; then @@ -289,11 +319,10 @@ render() { TRINO_IMAGE="$TRINO_IMAGE" TRINO_TLS_PASSWORD="$TRINO_TLS_PASSWORD" \ CONFIG_STORE_PASSWORD="$(cat "$config_store_password_file")" \ NAMESPACE="$NS" PR_NUMBER="$PR_NUMBER" \ - envsubst '$NAMESPACE $PR_NUMBER $TRINO_IMAGE $TRINO_TLS_PASSWORD $TRINO_CA_CERT_B64 $TRINO_SERVER_P12_B64 $CONFIG_STORE_PASSWORD' \ - < "$HERE/manifests.trino.tmpl.yaml" + render_trino_template if hoglake_perf_enabled; then - render_trino_perf_cluster perf - render_trino_perf_cluster cached + local mode + for mode in $(trino_perf_modes); do render_trino_perf_cluster "$mode"; done fi HOGLAKE_SERVICE_ACCOUNT=hoglake # Server default (15 min). Frozen perf registers footer-only fixtures and @@ -622,9 +651,8 @@ cmd_deploy() { # Patch the Deployment resources directly. The CI deployer intentionally # cannot patch the deployments/scale subresource, while it already needs # narrowly scoped Deployment patch access for the control-plane config. - # Three 1-CPU/4Gi workers match the frozen-perf Duckgres worker's - # aggregate 3-CPU/12Gi execution budget while exercising Trino's - # distributed path. + # The bootstrap cluster keeps three small workers for catalog admission. + # Comparison clusters below use their own larger worker profile. "${KUBECTL[@]}" -n "$NS" patch deployment duckgres-trino-coordinator \ --type=merge -p '{"spec":{"replicas":1}}' "${KUBECTL[@]}" -n "$NS" patch deployment duckgres-trino-worker \ @@ -632,7 +660,7 @@ cmd_deploy() { "${KUBECTL[@]}" -n "$NS" rollout status deploy/duckgres-trino-coordinator --timeout=300s "${KUBECTL[@]}" -n "$NS" rollout status deploy/duckgres-trino-worker --timeout=300s if hoglake_perf_enabled; then - for mode in perf cached; do + for mode in $(trino_perf_modes); do "${KUBECTL[@]}" -n "$NS" patch deployment "duckgres-trino-$mode-coordinator" --type=merge -p '{"spec":{"replicas":1}}' "${KUBECTL[@]}" -n "$NS" patch deployment "duckgres-trino-$mode-worker" --type=merge -p '{"spec":{"replicas":3}}' "${KUBECTL[@]}" -n "$NS" rollout status "deploy/duckgres-trino-$mode-coordinator" --timeout=300s @@ -1110,7 +1138,7 @@ cmd_diagnostics() { "${KUBECTL[@]}" -n "$NS" logs deploy/duckgres-trino-coordinator -c duckgres-trino-opa --tail=300 || true "${KUBECTL[@]}" -n "$NS" logs deploy/duckgres-trino-worker -c trino-worker --tail=300 || true if hoglake_perf_enabled; then - for mode in perf cached; do + for mode in $(trino_perf_modes); do "${KUBECTL[@]}" -n "$NS" logs "deploy/duckgres-trino-$mode-coordinator" -c trino-coordinator --tail=300 || true "${KUBECTL[@]}" -n "$NS" logs "deploy/duckgres-trino-$mode-coordinator" -c duckgres-trino-opa --tail=300 || true "${KUBECTL[@]}" -n "$NS" logs "deploy/duckgres-trino-$mode-worker" -c trino-worker --tail=300 || true diff --git a/tests/mw-dev/run_sh_test.go b/tests/mw-dev/run_sh_test.go index 001e4916e..1e2c2ff46 100644 --- a/tests/mw-dev/run_sh_test.go +++ b/tests/mw-dev/run_sh_test.go @@ -168,6 +168,13 @@ func TestTrinoWorkersMatchDuckgresAggregateCompute(t *testing.T) { "${TRINO_TLS_PASSWORD}", "test-password", "${TRINO_CA_CERT_B64}", "dGVzdA==", "${TRINO_SERVER_P12_B64}", "dGVzdA==", + "${TRINO_WORKER_CPU}", "1", + "${TRINO_WORKER_MEMORY}", "4Gi", + "${TRINO_WORKER_HEAP}", "3G", + "${TRINO_QUERY_MEMORY}", "6GB", + "${TRINO_WORKER_QUERY_MEMORY}", "2GB", + "${TRINO_WORKER_THREADS}", "24", + "${TRINO_WORKER_MIN_DRIVERS}", "48", ).Replace(string(raw)) decoder := utilyaml.NewYAMLOrJSONDecoder(strings.NewReader(rendered), 4096) diff --git a/tests/perf/README.md b/tests/perf/README.md index 1aa95f28d..885553449 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -7,7 +7,12 @@ Manual-only extended coverage scenarios exercise twelve additional workload shap artifacts and versioned intent IDs. These queries are excluded from standard nightly runs and execute only when explicitly dispatched. For the focused uncached DuckDB run, baseline recording, runtime estimates, and failure recovery, see the -[coverage runbook](../../docs/runbooks/perf-coverage.md). +[coverage runbook](../../docs/runbooks/perf-coverage.md). Frozen standard and extended benchmarks +use three Trino workers at 16 CPU / 64 GiB each and one DuckDB worker at +48 CPU / 192 GiB, matching aggregate worker resources. Trino uses a 48 GiB JVM +heap, 32 GiB per-worker query cap, and 96 GiB cluster-wide query cap. +Historical 3 CPU / 12 GiB results require a new baseline before comparison to +this larger profile; see the runbook for scheduling prerequisites. ## Protocol Drivers diff --git a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md index dcdf91b65..d1e73f117 100644 --- a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md +++ b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md @@ -1,6 +1,6 @@ # Coverage v1: uncached DuckDB baseline -This is the initial comparison base for the twelve coverage v1 intents on +This is the historical small-resource comparison base for the twelve coverage v1 intents on `pgwire_uncached`. [Successful source run](https://github.com/PostHog/duckgres/actions/runs/36159233201) measured commit `9c9f5f305f87b7f0d41c0b54d64e5648aeb8ef83` on 2026-09-25. The catalog and query settings match the measured revision. The focused entry @@ -12,6 +12,10 @@ manifest preserves the original scenario name used by the source run. settings, resource requests, methodology, and timing calculations. No query result values or internal fixture locations are retained. +The current benchmark uses larger workers. These samples and runtime estimates +apply only to the original 3 CPU / 12 GiB aggregate profile; they must not be +used as a like-for-like baseline for the new 48 CPU / 192 GiB profile. + ## Measured times - Coverage phase: **87m06s**, including 12 warmups and 48 measurements; zero errors. From dfc317d4875a31079bf4a3d4a1a4e7f978398cd6 Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Mon, 28 Sep 2026 09:59:56 -0400 Subject: [PATCH 11/13] perf: use dev-sized comparison workers --- .github/workflows/scenario-dev.yml | 10 ++++---- docs/runbooks/perf-coverage.md | 24 +++++++++---------- tests/mw-dev/run.sh | 8 +++---- tests/perf/README.md | 6 ++--- .../README.md | 2 +- 5 files changed, 25 insertions(+), 25 deletions(-) diff --git a/.github/workflows/scenario-dev.yml b/.github/workflows/scenario-dev.yml index d86a66d84..2c4088911 100644 --- a/.github/workflows/scenario-dev.yml +++ b/.github/workflows/scenario-dev.yml @@ -146,13 +146,13 @@ jobs: TRINO_IMAGE: ${{ needs.plan.outputs.trino_image }} DUCKGRES_SCENARIO_MAX_RUNTIME: 4h DUCKGRES_SCENARIO_GO_TEST_TIMEOUT: 4h15m - # Match the benchmark Trino fleet: three 16-CPU / 64Gi workers. + # Match the benchmark Trino fleet: three 7-CPU / 28Gi workers. # Non-benchmark scenarios retain their existing small resource budget. - DUCKGRES_K8S_WORKER_CPU_REQUEST: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '48' || '3' }} - DUCKGRES_K8S_WORKER_MEMORY_REQUEST: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '192Gi' || '12Gi' }} + DUCKGRES_K8S_WORKER_CPU_REQUEST: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '21' || '3' }} + DUCKGRES_K8S_WORKER_MEMORY_REQUEST: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '84Gi' || '12Gi' }} # The perf driver passes explicit sizing GUCs: keep their admission caps aligned. - DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '48' || '8' }} - DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '192Gi' || '16Gi' }} + DUCKGRES_K8S_WORKER_PROFILE_MAX_CPU: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '21' || '8' }} + DUCKGRES_K8S_WORKER_PROFILE_MAX_MEMORY: ${{ (github.event_name == 'schedule' || inputs.scenario == 'posthog_frozen_perf' || inputs.scenario == 'posthog_frozen_perf_extended' || inputs.scenario == 'posthog_frozen_perf_extended_uncached') && '84Gi' || '16Gi' }} steps: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 diff --git a/docs/runbooks/perf-coverage.md b/docs/runbooks/perf-coverage.md index eabff5244..fa2a34653 100644 --- a/docs/runbooks/perf-coverage.md +++ b/docs/runbooks/perf-coverage.md @@ -62,26 +62,26 @@ Standard frozen perf and extended perf use the following execution budgets: | Engine | Workers | CPU per worker | RAM per worker | JVM heap | Query memory per worker | |---|---:|---:|---:|---:|---:| -| Trino (each cache mode) | 3 | 16 | 64 GiB | 48 GiB | 32 GiB | -| DuckDB (each cache mode) | 1 | 48 | 192 GiB | N/A | Derived by Duckgres from pod memory | +| Trino (each cache mode) | 3 | 7 | 28 GiB | 20 GiB | 10 GiB | +| DuckDB (each cache mode) | 1 | 21 | 84 GiB | N/A | Derived by Duckgres from pod memory | -Both have **48 CPU / 192 GiB of worker resources**, with requests equal to +Both have **21 CPU / 84 GiB of worker resources**, with requests equal to limits. This compares aggregate execution resources; Trino coordinator and support services are additional overhead. Athena capacity is service-managed. -Trino's cluster-wide query memory cap is 96 GiB. Its 48 GiB heap leaves 16 GiB -outside the JVM, and its 32 GiB per-node query cap leaves room for JVM headroom. -This is a chosen production-oriented profile consistent with -[Trino's guidance](https://trino.io/docs/current/installation/deployment.html#jvm-config), -not a universal prescribed CPU/RAM size. +Trino's cluster-wide query memory cap is 30 GiB. Its 20 GiB heap leaves 8 GiB +outside the JVM, and its 10 GiB per-node query cap leaves room for JVM headroom. +Worker CPU, RAM, heap and per-node query memory match the configured dev +worker size. The cluster-wide cap scales to the requested three-worker fleet; +this is a benchmark profile, not a copy of the complete dev deployment. DuckDB's client sizing caps are raised alongside its worker resources so the -explicit benchmark profile is not silently clamped. At 192 GiB, the current -Duckgres headroom policy sets DuckDB's memory limit to 144 GiB; engine memory +explicit benchmark profile is not silently clamped. At 84 GiB, the current +Duckgres headroom policy sets DuckDB's memory limit to 63 GiB; engine memory limits differ even though container resources match. Other scenario types and the auxiliary Trino bootstrap cluster retain their small defaults. -The benchmark node pool must admit nodes large enough for a 48-CPU / 192Gi -DuckDB pod and 16-CPU / 64Gi Trino pods, with additional allocatable capacity for +The benchmark node pool must admit nodes large enough for a 21-CPU / 84Gi +DuckDB pod and 7-CPU / 28Gi Trino pods, with additional allocatable capacity for system services. Deploy the [node-size allowlist change](https://github.com/PostHog/charts/pull/16434) before running this profile. If pods remain Pending, inspect scheduling events and node-pool constraints rather than counting setup failure as query latency. diff --git a/tests/mw-dev/run.sh b/tests/mw-dev/run.sh index 7fb0d910f..b94da0543 100755 --- a/tests/mw-dev/run.sh +++ b/tests/mw-dev/run.sh @@ -144,15 +144,15 @@ trino_multicell_enabled() { } # Bootstrap and general E2E keep the small fleet. Only comparison clusters -# use production-sized workers; both cache modes use the same profile. +# use dev-sized workers; both cache modes use the same profile. render_trino_template() { local TRINO_WORKER_CPU=1 TRINO_WORKER_MEMORY=4Gi TRINO_WORKER_HEAP=3G local TRINO_QUERY_MEMORY=6GB TRINO_WORKER_QUERY_MEMORY=2GB local TRINO_WORKER_THREADS=24 TRINO_WORKER_MIN_DRIVERS=48 if [ "${1:-}" = perf ]; then - TRINO_WORKER_CPU=16 TRINO_WORKER_MEMORY=64Gi TRINO_WORKER_HEAP=48G - TRINO_QUERY_MEMORY=96GB TRINO_WORKER_QUERY_MEMORY=32GB - TRINO_WORKER_THREADS=32 TRINO_WORKER_MIN_DRIVERS=64 + TRINO_WORKER_CPU=7 TRINO_WORKER_MEMORY=28Gi TRINO_WORKER_HEAP=20G + TRINO_QUERY_MEMORY=30GB TRINO_WORKER_QUERY_MEMORY=10GB + TRINO_WORKER_THREADS=14 TRINO_WORKER_MIN_DRIVERS=28 fi export TRINO_WORKER_CPU TRINO_WORKER_MEMORY TRINO_WORKER_HEAP export TRINO_QUERY_MEMORY TRINO_WORKER_QUERY_MEMORY TRINO_WORKER_THREADS TRINO_WORKER_MIN_DRIVERS diff --git a/tests/perf/README.md b/tests/perf/README.md index 885553449..3ec2e7905 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -8,9 +8,9 @@ artifacts and versioned intent IDs. These queries are excluded from standard nightly runs and execute only when explicitly dispatched. For the focused uncached DuckDB run, baseline recording, runtime estimates, and failure recovery, see the [coverage runbook](../../docs/runbooks/perf-coverage.md). Frozen standard and extended benchmarks -use three Trino workers at 16 CPU / 64 GiB each and one DuckDB worker at -48 CPU / 192 GiB, matching aggregate worker resources. Trino uses a 48 GiB JVM -heap, 32 GiB per-worker query cap, and 96 GiB cluster-wide query cap. +use three Trino workers at 7 CPU / 28 GiB each and one DuckDB worker at +21 CPU / 84 GiB, matching aggregate worker resources. Trino uses a 20 GiB JVM +heap, 10 GiB per-worker query cap, and 30 GiB cluster-wide query cap. Historical 3 CPU / 12 GiB results require a new baseline before comparison to this larger profile; see the runbook for scheduling prerequisites. diff --git a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md index d1e73f117..7b16aedfb 100644 --- a/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md +++ b/tests/perf/baselines/coverage-v1-duckdb-uncached-2026-09-25/README.md @@ -14,7 +14,7 @@ result values or internal fixture locations are retained. The current benchmark uses larger workers. These samples and runtime estimates apply only to the original 3 CPU / 12 GiB aggregate profile; they must not be -used as a like-for-like baseline for the new 48 CPU / 192 GiB profile. +used as a like-for-like baseline for the new 21 CPU / 84 GiB profile. ## Measured times From b226a2e12b88a5ad65dd3d486fec9c2dc83cfb1b Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Mon, 28 Sep 2026 13:52:14 -0400 Subject: [PATCH 12/13] fix(perf): allow cold worker startup during connection setup --- tests/mw-dev/run.sh | 5 +++++ tests/perf/README.md | 16 ++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/tests/mw-dev/run.sh b/tests/mw-dev/run.sh index b94da0543..b077d4a4d 100755 --- a/tests/mw-dev/run.sh +++ b/tests/mw-dev/run.sh @@ -838,6 +838,10 @@ cmd_test_scenario() { run_scenario() { local scenario_name="$1" scenario_file="$2" job pod api_base pg suffix internal_secret artifact_rc=0 container_rc=0 container_exit_code scenario_rc=0 + local pg_connect_timeout=10 + # Sized benchmark workers may cold-start during the PostgreSQL handshake. + # Allow the server's five-minute worker queue budget plus startup headroom. + if frozen_perf_scenario; then pg_connect_timeout=360; fi api_base="http://duckgres-control-plane.$NS.svc:8080" pg="$("${KUBECTL[@]}" -n "$NS" get svc duckgres-control-plane -o jsonpath='{.spec.clusterIP}')" suffix=".ci.duckgres.local" @@ -903,6 +907,7 @@ spec: - { name: DUCKGRES_SCENARIO_OUTPUT_BASE, value: "/artifacts/scenario-dev" } - { name: DUCKGRES_SCENARIO_RUN_ID, value: "$DUCKGRES_SCENARIO_RUN_ID" } - { name: DUCKGRES_SCENARIO_COVERAGE_TARGET, value: "${DUCKGRES_SCENARIO_COVERAGE_TARGET:-}" } + - { name: DUCKGRES_SCENARIO_PG_CONNECT_TIMEOUT, value: "${DUCKGRES_SCENARIO_PG_CONNECT_TIMEOUT:-$pg_connect_timeout}" } - { name: DUCKGRES_SCENARIO_MAX_RUNTIME, value: "${DUCKGRES_SCENARIO_MAX_RUNTIME:-4h}" } - { name: DUCKGRES_SCENARIO_GO_TEST_TIMEOUT, value: "${DUCKGRES_SCENARIO_GO_TEST_TIMEOUT:-4h15m}" } - { name: GOCACHE, value: "/tmp/go-cache" } diff --git a/tests/perf/README.md b/tests/perf/README.md index 3ec2e7905..0f8b8cccb 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -405,3 +405,19 @@ Scenario Trino connections require `DUCKGRES_SCENARIO_TRINO_CATALOG_STORE_CELL_I (no default), matching the baseline coordinator's `catalog-store.cell-id`. The isolated workflow supplies it; local runs must set it. The public readiness API cell ID is used only for API identity validation, not catalog-store lookups. + +### Benchmark worker connection startup + +Frozen core and extended scenario runs default PostgreSQL connection startup to +360 seconds, allowing the server's five-minute worker-acquisition budget plus +headroom for cold worker startup. Other scenarios retain a 10-second default. +Override with `DUCKGRES_SCENARIO_PG_CONNECT_TIMEOUT` (seconds) when invoking +`tests/mw-dev/run.sh`. This timeout is passed into the scenario Job; query +execution timeouts and measured query durations are unchanged. + +If a run reports `pin pgwire cache-variant connection` timeouts, inspect +control-plane worker-acquisition logs and pod scheduling/startup events. The +cache-variant driver retains its initial setup error, so one failed connection +can mark every query for that target as failed. Resolve worker readiness or +connection-timeout configuration before rerunning; these are not query latency +measurements. From c137b1cea4c850b07c20ae389d1c78651e3e4b20 Mon Sep 17 00:00:00 2001 From: Bill Guowei Yang Date: Mon, 28 Sep 2026 21:38:55 -0400 Subject: [PATCH 13/13] docs(perf): correct comparison and bootstrap worker sizes --- tests/perf/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/perf/README.md b/tests/perf/README.md index 0f8b8cccb..cf4376f94 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -337,8 +337,10 @@ Both clusters load the same Alluxio and memory cache managers. Each node has a 16GB disk-cache budget in a 20Gi ephemeral volume, with 64kB pages and a seven-day TTL. For connectors that use this cache, entries persist between queries/iterations until eviction or teardown; one warm-up does not guarantee -every replica holds the complete working set. Worker CPU/memory limits remain -three workers at 1 CPU/4Gi per cluster; cache volumes reserve additional storage. +every replica holds the complete working set. Each comparison cluster uses +three workers at 7 CPU / 28 GiB each, matching the DuckDB worker's aggregate +21 CPU / 84 GiB budget. The auxiliary bootstrap cluster retains three workers +at 1 CPU / 4 GiB each; cache volumes reserve additional storage. The frozen suite runs the newest PostHog/trino master build, whose Hoglake connector honors `fs.cache.enabled` through Trino's shared filesystem module