fix: isMisconfiguredPdb to properly handle "outdated" currentHealth - #1127
Merged
Conversation
gardener-prow
Bot
added
do-not-merge/work-in-progress
Kumm-Kai
marked this pull request as ready for review
July 30, 2026 09:22
gardener-prow
Bot
removed
the
do-not-merge/work-in-progress
Contributor
Author
|
/kind bug |
gardener-prow
Bot
added
kind/bug
aaronfern
reviewed
Aug 3, 2026
| // Example PDB from etcd-druid (3 replicas) | ||
| pdb = &policyv1.PodDisruptionBudget{ | ||
| Spec: policyv1.PodDisruptionBudgetSpec{ | ||
| // MaxUnavailable: new(intstr.FromInt(1)), |
Member
There was a problem hiding this comment.
Suggested change
| // MaxUnavailable: new(intstr.FromInt(1)), |
| CurrentHealthy: 3, | ||
| DisruptionsAllowed: 1, // `CurrentHealthy` - `DesiredHealthy` | ||
| }, | ||
| // `ExpectedPods` - `MaxUnavailable` |
Member
There was a problem hiding this comment.
Seems to have been kept by error
Suggested change
| // `ExpectedPods` - `MaxUnavailable` |
|
|
||
| return pdb.Status.ExpectedPods > 0 && pdb.Status.CurrentHealthy >= pdb.Status.ExpectedPods && pdb.Status.DisruptionsAllowed == 0 | ||
| hasPods := pdb.Status.ExpectedPods > 0 | ||
| // `>=` instead of `==` (because of an in progress scale down `CurrentHealthy` can be larger than `ExpectedPods` current might be larger than expected |
Member
There was a problem hiding this comment.
Could you please improve this comment? The current might be larger than expected part is a bit redundant
| var pdb *policyv1.PodDisruptionBudget | ||
| Context("normal", func() { | ||
| It("returns false", func() { | ||
| // Example PDB from etcd-druid (3 replicas) |
Member
There was a problem hiding this comment.
This comment isn't required/helpful 🤷🏻
r4mek
reviewed
Aug 5, 2026
gardener-prow
Bot
added
size/L
Member
|
/lgtm |
|
LGTM label has been added. DetailsGit tree hash: bc640978206d858dc90a7090ae5d1190d6daaff8 |
Member
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: aaronfern The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
gardener-prow
Bot
added
the
approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
/kind bug
We see many
pod disruption budget ... is misconfigured and requires zero voluntary evictionsfor PDBs that are (when all pods are healthy) actually not misconfigured.This happens when the MCM gets the PDB between a pod getting evicted and the PDB controller reconciling the PDB.
Currently the MCM only takes into account the perspective of the PDB controller. But the eviction calls also cause changes in the PDB object:
disruptionsAlloweddisruptedPodsMore details about the changes in the PDB object:
Details
Everything is running:
{ "apiVersion": "policy/v1", "kind": "PodDisruptionBudget", "metadata": { "creationTimestamp": "2026-04-27T15:44:24Z", "generation": 1, "labels": { "app": "kubernetes", "role": "apiserver" }, "name": "kube-apiserver", "namespace": "shoot--xxx--xxx", "resourceVersion": "11475386785", "uid": "9931059a-5e59-47ba-b1d1-33211c5f8c2d" }, "spec": { "maxUnavailable": 1, "selector": { "matchLabels": { "app": "kubernetes", "role": "apiserver" } }, "unhealthyPodEvictionPolicy": "AlwaysAllow" }, "status": { "conditions": [ { "lastTransitionTime": "2026-07-25T12:33:41Z", "message": "", "observedGeneration": 1, "reason": "SufficientPods", "status": "True", "type": "DisruptionAllowed" } ], "currentHealthy": 3, "desiredHealthy": 2, "disruptionsAllowed": 1, "expectedPods": 3, "observedGeneration": 1 } }After evict call:
{ "apiVersion": "policy/v1", "kind": "PodDisruptionBudget", "metadata": { "creationTimestamp": "2026-04-27T15:44:24Z", "generation": 1, "labels": { "app": "kubernetes", "role": "apiserver" }, "name": "kube-apiserver", "namespace": "shoot--xxx--xxx", "resourceVersion": "11550070628", "uid": "9931059a-5e59-47ba-b1d1-33211c5f8c2d" }, "spec": { "maxUnavailable": 1, "selector": { "matchLabels": { "app": "kubernetes", "role": "apiserver" } }, "unhealthyPodEvictionPolicy": "AlwaysAllow" }, "status": { "conditions": [ { "lastTransitionTime": "2026-07-28T12:34:25Z", "message": "", "observedGeneration": 1, "reason": "InsufficientPods", "status": "False", "type": "DisruptionAllowed" } ], "currentHealthy": 3, "desiredHealthy": 2, "disruptedPods": { "kube-apiserver-65cbbb5d5d-zmm29": "2026-07-28T12:34:25Z" }, "disruptionsAllowed": 0, "expectedPods": 3, "observedGeneration": 1 }, "timestamp": "2026-07-28T12:34:25Z" }Evict call:
disruptionsAllowedby 1disruptedPodscurrentHealthyremains at3!-> Before my PR: Misconfigured as
currentHealthy>=expectedPods-> After my PR: Not misconfigured as
len(disruptedPods)== 0First PDB controller reconcile after eviction:
{ "apiVersion": "policy/v1", "kind": "PodDisruptionBudget", "metadata": { "creationTimestamp": "2026-04-27T15:44:24Z", "generation": 1, "labels": { "app": "kubernetes", "role": "apiserver" }, "name": "kube-apiserver", "namespace": "shoot--xxx--xxx", "resourceVersion": "11550071136", "uid": "9931059a-5e59-47ba-b1d1-33211c5f8c2d" }, "spec": { "maxUnavailable": 1, "selector": { "matchLabels": { "app": "kubernetes", "role": "apiserver" } }, "unhealthyPodEvictionPolicy": "AlwaysAllow" }, "status": { "conditions": [ { "lastTransitionTime": "2026-07-28T12:34:25Z", "message": "", "observedGeneration": 1, "reason": "InsufficientPods", "status": "False", "type": "DisruptionAllowed" } ], "currentHealthy": 2, "desiredHealthy": 2, "disruptionsAllowed": 0, "expectedPods": 3, "observedGeneration": 1 }, "timestamp": "2026-07-28T12:34:26Z" }PDB controller:
disruptedPodsthat are getting deletedDeletionTimestamp != nilcurrentHealthy-> now2expectedPodsremains at3-> Not misconfigured as not all pods are healthy
Replacement pod healthy and PDB controller reconcile done (same as the first one):
{ "apiVersion": "policy/v1", "kind": "PodDisruptionBudget", "metadata": { "creationTimestamp": "2026-04-27T15:44:24Z", "generation": 1, "labels": { "app": "kubernetes", "role": "apiserver" }, "name": "kube-apiserver", "namespace": "shoot--xxx--xxx", "resourceVersion": "11550090088", "uid": "9931059a-5e59-47ba-b1d1-33211c5f8c2d" }, "spec": { "maxUnavailable": 1, "selector": { "matchLabels": { "app": "kubernetes", "role": "apiserver" } }, "unhealthyPodEvictionPolicy": "AlwaysAllow" }, "status": { "conditions": [ { "lastTransitionTime": "2026-07-28T12:35:24Z", "message": "", "observedGeneration": 1, "reason": "SufficientPods", "status": "True", "type": "DisruptionAllowed" } ], "currentHealthy": 3, "desiredHealthy": 2, "disruptionsAllowed": 1, "expectedPods": 3, "observedGeneration": 1 }, "timestamp": "2026-07-28T12:35:24Z" }Which issue(s) this PR fixes:
None
Special notes for your reviewer:
Release note: