Skip to content

perf: use median aggregation for reliable benchmark comparisons - #56

Merged
vm75 merged 5 commits into
mainfrom
median-bench
Oct 11, 2026
Merged

vm75 merged 5 commits into
mainfrom
median-bench

Conversation

@vm75

@vm75 vm75 commented Oct 11, 2026

Copy link
Copy Markdown
Owner

Summary

Improve performance benchmark reliability by replacing arithmetic-mean aggregation with the median across repeated runs, reducing sensitivity to occasional CPU contention and background activity.

Changes

  • Use median aggregation for throughput, p50, and p95 timing metrics.
  • Increase local run-perf.py default from 5 to 9 runs while retaining 5 runs for CI and Makefile defaults.
  • Normalize historical benchmark snapshots from raw measurements to ensure consistent mean-versus-median comparisons.
  • Preserve existing CLI parameters, environment variables, CSV formats, raw measurements, and regression thresholds.
  • Add regression tests for aggregation, CSV compatibility, and historical snapshot normalization.
  • Update performance documentation.

Compatibility

No changes to the transliteration engine, public APIs, or native benchmark workloads. Existing CI and performance tooling remain compatible.

Validation

Added focused regression tests. Test execution and CI validation should be confirmed before merging.

@vm75
vm75 merged commit 441590d into main Oct 11, 2026
7 checks passed
@vm75
vm75 deleted the median-bench branch October 11, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant