recovery: remove automatic retry and improve Job failure detection

[tkatila]

Automatic retry has limited benefits. Retrying a failed reset might cause more harm than benefit, e.g. host crash. If admin wants to retry a reset etc. he/she should re-approve the event to retrigger a recovery.

Operator failed to detect Job failures in some scenarios, which then resulted in the event staying in "in-progress" forever.

Remove retryCount from events. pastJobs can be used to count how many retries there has been.

[Copilot]

🟡 Changes recommended

The new overdue-verdict logic derives deadlines from the current plan spec rather than the Job’s own activeDeadlineSeconds, which can incorrectly fail an in-flight Job after a mid-flight timeout edit and prematurely clear the drain taint.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR changes GPU recovery behavior to make failed recovery attempts terminal (no automatic retries) and strengthens the controller’s ability to detect and surface recovery Job failures/hangs so events don’t remain in-progress indefinitely.

Changes:

• Removes automatic retry semantics (spec.maxRetries and status.events[].retryCount) and relies on explicit re-approval to start another attempt.
• Improves Job failure detection by treating missing/no-verdict Jobs as failures once the Job’s deadline (plus grace) is exceeded, and enforces backoffLimit: 0 (one pod per approval).
• Hardens spec persistence logic to avoid reverting consumed approvals when the controller reads a stale spec.

File summaries

File	Description
RECOVERY.md	Updates user-facing behavior docs: failed is terminal; explains re-approval and Job-verdict guarantees.
internal/controller/gpurecoveryplan_helpers.go	Adds Job terminal/verdict helpers and new failure recording via PastJobs; removes retry budget logic.
internal/controller/gpurecoveryplan_controller.go	Uses APIReader for fresh plan reads; persists spec via approval diffs; sets Job backoffLimit: 0; adds “missing/no-verdict Job” handling.
internal/controller/gpurecoveryplan_controller_test.go	Updates/expands tests for terminal failures, approval persistence, attempt history, no Job retries, and overdue/no-verdict behavior.
internal/controller/gpurecoveryplan_const.go	Introduces a grace period constant for delayed Job verdicts.
config/crd/bases/intel.com_gpurecoveryplans.yaml	Removes maxRetries and retryCount from the CRD schema; updates pastJobs description.
charts/gpu-base-operator/crds/gpurecoveryplans.yaml	Mirrors CRD schema updates for Helm packaging.
api/v1alpha1/gpurecoveryplan_types.go	Removes MaxRetries and RetryCount from API types; updates PastJobs comment.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.