Estimate when a try push will finish

[rcurranmoz]

Adds an ETA to the push list and the push detail view, and ships it as 1.2 (10).

The reason a try push is annoying isn't that it fails, it's not knowing when it lands. This
answers that.

Two numbers, not one

A push doesn't finish smoothly. On a sampled 907-job push, 90% of the jobs were done at 47
minutes and the last one at 144 — and the tail wasn't slow work, it was waiting: those
final jobs ran for 12–25 minutes after queueing for 95–117. A single "done at" time is
therefore dominated by whichever worker pool happens to be backed up.

So the headline is when 90% of your jobs are in, which is both the reliable number and
the one that answers "when will I know if this is green". The full finish sits beside it,
deliberately rendered as an approximation.

Backtested against 369 try pushes / 73,864 jobs, by replaying 77 completed pushes at nine
points each:

	median error	within ±25%	within ±50%	overruns by >30 min
Most results (90% of jobs)	2 min	70%	88%	1%
All done (last job)	13 min	39%	62%	32%

For contrast, naive percent-complete extrapolation — the obvious implementation — has a
84–165 minute median error and a −50 to −105 minute bias, because the completion curve is
nowhere near linear.

How

Run time is the predictable part: a job's duration has a median coefficient of variation of
5%, so a bundled table of per-job-type medians predicts it to within 5%. Queue wait is the
hard part, and it's read live per worker pool from the push itself — jobs in a pool that have
already started tell you what the ones that haven't will wait. On one push, 76 jobs in
macosx1500-aarch64-shippable had a p25, median and p90 queue wait of 140.7, 140.7 and
144.6 minutes, because they're all gated on the same build and released together.

The table is why this works and why it ships (137 KB, BuildWatch/Resources/JobDurations.json,
regenerate with tools/generate-duration-table.py). Building the same estimator from only
the push's own completed jobs lands within ±25% just 12% of the time, against 70% with it.

No new data source and no extra requests — it runs off job rows BuildWatch already fetches.

Three states, because only one supports a countdown

• Firm — countdown, timeline bar, long-pole pool, all-done time.
• Waiting on build — the most common reason a push looks frozen is its tests sitting
  unscheduled behind a build; one live Talos push had 32 of its 37 unresolved jobs in
  exactly that state with no pool data to work from. Rather than a vague push ETA, the card
  shows the precise thing: which stage is running and when tests get released. Gecko's
  shippable pipeline is three stages deep (instrumented-build- → generate-profile- →
  build-), so the chain is walked, not just the running stage. Predicted build finishes
  land within 5 minutes 71% of the time.
• Estimating — no clock times at all, just progress and elapsed. Below 60% pool coverage
  the estimate is wrong by ~113 minutes.

A flat "rough estimate" band between those was measured and dropped: within ±50% only half
the time, and a range honest enough to cover 77% has to span 6× ("somewhere between one hour
and six"). 83% of pushes clear the firm bar, at a median of 30 minutes in, and then stay
clear for a median 72% of what's left.

Three bugs this surfaced, fixed here

• unscheduled was an unmodelled job state. TreeHerder emits it alongside the other
  three and it isn't rare — a live 691-job push had 24. The parser's unknown-state fallback
  filed them as .completed, so they were invisible to isRunning and could fire "Try push
  passed" with jobs not yet started. Since the watch is one-shot, no correction followed.
  Unknown states now default to .pending; guessing "done" is the dangerous direction.
• Absent timestamps arrive as 0, not null, and decoded to a valid 1 January 1970, so
  every queued job looked like one that started 56 years ago.
• A running push with failures hid the ETA behind the failure badge — exactly the pushes
  being watched most closely. Both now show.

Verification

• Debug and Release both build clean; Release confirmed carrying JobDurations.json (without
  it the estimator silently degrades to one 20.8-minute global median for every job).
• Run against live data in the simulator. The list-row pill on a real in-flight Talos push
  read 1h 2m, matching an independent hand calculation of the algorithm to the minute
  (~6 min of generate-profile + ~59 min of build-macosx64-aarch64-shippable).
• Accuracy figures come from a strict train/test split by time, so the duration table never
  sees the pushes it's scored on.

Caveats

• 77 test pushes, and observations within a push are correlated, so treat the percentages as
  ±several points. The effects driving the design are large enough that the conclusions hold.
• Retriggers can't be anticipated — two test pushes had jobs submitted 5 and 21 hours later,
  and they account for the worst errors in the whole backtest. The ETA just recomputes.
• Confidence's thresholds and the 1.25× tail calibration are tuned on three days of try
  traffic. If pool behaviour shifts a lot, they'd want re-measuring.

Not in this PR

NotificationDelegate posts .openPush but nothing observes it, so tapping a completion
notification doesn't deep-link anywhere despite the README saying it does. Pre-existing and
left alone.

🤖 Generated with Claude Code

[rcurranmoz]