[ardaakman]

What changed

Why

Checks

• make check
• make test-local
• Chart values changed? Doc comment, schema entry, and a default that is
  safe on a cluster that is not ours.
• Nothing here only makes sense in one particular cluster.

---

Note

High Risk
Changes privileged daemon host enrollment/repair logic and operator schedulable capacity; misbehavior could strand nodes or mis-advertise GPUs until rolled back.

Overview
This PR extends the node daemon and operator, bumps thunder-sdk to v0.2.0, and refreshes chart defaults and integration tests.

Helm / daemon configuration: Adds daemon.extraEnv so operators can inject extra container env vars (including secrets via valueFrom). Variables named THUNDERD_ENV_<key> are merged into the host’s /etc/thunder/thunderd.env once at daemon startup (atomic write via stdin, no thunderd restart). Chart schema, DaemonSet template, values, and README document rollout behavior (pod restart required for env/secret changes).

Daemon repair behavior: Reworks unhealthy-node handling so enrolled nodes are never re-enrolled (avoids duplicate hosts and enrollment-token spam). Repairs go through a capped budget (repairAttempt: restart, CLI reinstall without token, dangling thunderd.service symlink sweep, optional thunder update / rollback when distribution checksum changes). Enrollment is reserved for nodes with no auth token on disk.

Operator capacity: Hosts with configurations.cordoned no longer contribute physical GPU capacity; live client counts still set the advertised floor when every host is cordoned. Local kind tests assert cordon/uncordon ResourceSlice and DeviceClass behavior.

Misc: publish-images workflow gains pull-requests: write; kind_bridge_address is more reliable for the registry stub; operator/daemon image tags updated in values.

^{Reviewed by Cursor Bugbot for commit 3b73073. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Bridge lookup aborts past check
  • Added || true to kind_bridge_address call to prevent errexit from aborting before the empty-string check and diagnostic message can run.

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Enrollment budget blocks later recovery
  • Removed repairAttempt wrapper from enrollment path to allow unlimited retries via reconcile loop backoff, preventing permanent enrollment stalls when NVIDIA driver or Thunder API appears after initial failures.

Or push these changes by commenting:

@cursor push db6069dd7d

Preview (db6069dd7d)

diff --git a/internal/daemon/reconcile.go b/internal/daemon/reconcile.go
--- a/internal/daemon/reconcile.go
+++ b/internal/daemon/reconcile.go
@@ -189,15 +189,13 @@ func (r *reconciler) ensureEnrolled(ctx context.Context, cfg Config) error {
 
 	if !r.nodeEnrolled(cfg, status, statusErr) {
 		log.Printf("thunderd unhealthy on node %s after %d check(s); enrolling", cfg.Node, r.unhealthy)
-		attempted, err := r.repairAttempt(ctx, cfg, func(ctx context.Context) error {
-			return r.enroll(ctx, cfg)
-		}, nil, nil)
-		if err != nil {
-			return err
+		if err := r.sweepDanglingSymlinks(ctx); err != nil {
+			log.Printf("node %s: could not sweep dangling thunderd symlinks, enrolling anyway: %v", cfg.Node, err)
 		}
-		if attempted {
-			r.unhealthy, r.transitional = 0, 0
+		if err := r.enroll(ctx, cfg); err != nil {
+			return err
 		}
+		r.unhealthy, r.transitional = 0, 0
 		return nil
 	}
 

@@ -189,15 +189,13 @@ func (r *reconciler) ensureEnrolled(ctx context.Context, cfg Config) error {
 
 	if !r.nodeEnrolled(cfg, status, statusErr) {
 		log.Printf("thunderd unhealthy on node %s after %d check(s); enrolling", cfg.Node, r.unhealthy)
-		attempted, err := r.repairAttempt(ctx, cfg, func(ctx context.Context) error {
-			return r.enroll(ctx, cfg)
-		}, nil, nil)
-		if err != nil {
-			return err
+		if err := r.sweepDanglingSymlinks(ctx); err != nil {
+			log.Printf("node %s: could not sweep dangling thunderd symlinks, enrolling anyway: %v", cfg.Node, err)
 		}
-		if attempted {
-			r.unhealthy, r.transitional = 0, 0
+		if err := r.enroll(ctx, cfg); err != nil {
+			return err
 		}
+		r.unhealthy, r.transitional = 0, 0
 		return nil
 	}

_{You can send follow-ups to the cloud agent here.}

[brianmodel]

prob don't need to have a repair budget, it's ok if it restarts a bunch of times

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Absolute symlink check uses container root
  • Modified danglingThunderdSymlinks to read symlink targets and resolve absolute paths against hostRoot before checking existence, preventing valid host symlinks from being incorrectly deleted.

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 66a20fa. Configure here.}