Skip to content

fix(derived data): Adjust auto-heal timeouts in response to operational realities - #125155

Merged
kcons merged 1 commit into
masterfrom
kcons/tout
Sep 22, 2026
Merged

kcons merged 1 commit into
masterfrom
kcons/tout

Conversation

@kcons

@kcons kcons commented Sep 21, 2026

Copy link
Copy Markdown
Member

We're working on a more efficient approach here, but in the meantime, we recognize that running these long queries in a task with a relatively short deadline can be a problem.

Here we allow the task to run longer, give the (cachable) hash discovery query enough headroom to reliably suceeed even when not recently vacuumed, and add a timeout that we don't expect to hit to our group id batching.

The total deadline isn't enough for worst-cases everywhere, but due to discovery caching, if we can find one or two stale hashes and cache that, that's enough to make progress.
Similarly, on the group batching side, we need to be able to process at least one, ideally two hashes to make significant forward progress each run.

One key motivation here: we always prefer to hit statement timeouts over task timeouts. Evidence indicates that task timeouts don't abort the running query, not remotely, and not locally, and this seems correlated with task failures going unreported. OperationalErrors within the task deadline seem to report reliably.

@kcons
kcons requested review from a team as code owners September 21, 2026 23:06
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Sep 21, 2026
@kcons
kcons merged commit d3f720e into master Sep 22, 2026
69 checks passed
@kcons
kcons deleted the kcons/tout branch September 22, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants