see #1389 (comment)
Today the same entry code was measured twice, hours apart, and the baseline 512 row read +8.9% in the first run and -2.7% in the second. Both numbers were published as a delta against the previous run, so both look like a change in the entry, and neither is.
The second run says what really happened: every row moved between -0.5% and -2.7%, echo-ws and echo-ws-pipeline included. Those two cannot be moved by a change to HTTP header writing, since a WebSocket frame writes no header, so the whole sitting was about 1% slower. That is visible only because there happen to be rows the change cannot touch.
The cheap half of a fix needs no extra machine time. RUNS=3 already measures every profile three times and save_result keeps only the winner. If the three rps values were kept in the result JSON, the PR comment could print the delta next to the spread of the runs behind it, and a delta smaller than that spread would read as what it is.
The expensive half, worth it only for a PR that changes an entry: measure base and head in the same session, alternating per profile, and report the paired delta. One arm measuring the same commit twice gives the floor for that sitting.
This protects the board from lucky runs as much as from unlucky ones. It also tells an author when a change is simply too small for this instrument: a 1.2us per response change measures fine with a profiler and cannot be seen here at all.
see #1389 (comment)
Today the same entry code was measured twice, hours apart, and the
baseline512 row read +8.9% in the first run and -2.7% in the second. Both numbers were published as a delta against the previous run, so both look like a change in the entry, and neither is.The second run says what really happened: every row moved between -0.5% and -2.7%,
echo-wsandecho-ws-pipelineincluded. Those two cannot be moved by a change to HTTP header writing, since a WebSocket frame writes no header, so the whole sitting was about 1% slower. That is visible only because there happen to be rows the change cannot touch.The cheap half of a fix needs no extra machine time.
RUNS=3already measures every profile three times andsave_resultkeeps only the winner. If the three rps values were kept in the result JSON, the PR comment could print the delta next to the spread of the runs behind it, and a delta smaller than that spread would read as what it is.The expensive half, worth it only for a PR that changes an entry: measure base and head in the same session, alternating per profile, and report the paired delta. One arm measuring the same commit twice gives the floor for that sitting.
This protects the board from lucky runs as much as from unlucky ones. It also tells an author when a change is simply too small for this instrument: a 1.2us per response change measures fine with a profiler and cannot be seen here at all.