Backup/restore fails for per-user (common) storage on multi-node clusters due to PVC multi-attach

[rohanKanojia]

Description

DevWorkspace backup/restore fails for workspaces using per-user (common) storage on multi-node clusters. The backup Job cannot mount the shared ReadWriteOnce PVC when it gets scheduled on a different node than the running workspace pod that holds the PVC.

This works correctly on single-node clusters (e.g., CRC) because all pods land on the same node, and Kubernetes allows multiple pods on the same node to mount a ReadWriteOnce PVC. On multi-node clusters (e.g., AWS with 6 nodes), the backup Job pod may be scheduled on a different node, causing a Multi-Attach error or the DWO controller skipping Job creation entirely.

  oc describe pod devworkspace-backup-wn85f-xkbgt -n dwo-backup-test

 ...

  Warning  FailedAttachVolume  29m  attachdetach-controller
  Multi-Attach error for volume
  "pvc-42803733-1cc7-4abb-a06c-4175bee35041" Volume is already used by
  pod(s) workspacec7ccc971e0c84d22-778bf568d4-5rzcj

Per-workspace storage is unaffected because each workspace has its own dedicated PVC — stopping one workspace fully releases its PVC for the backup Job.

How To Reproduce

1. Deploy DWO on a multi-node OpenShift cluster (e.g., AWS with 6 nodes, OCP 4.20)
2. Create two DevWorkspaces using per-user (common) storage in the same namespace
3. Stop one workspace
4. Trigger a backup of the stopped workspace
5. Observe that the backup Job either:
   • Gets stuck in Pending with a Multi-Attach error (PVC is held by the running workspace on another node)
   • Is never created by DWO because the shared PVC is still in use
6. The backup times out

Expected behavior

The backup Job should be scheduled on the same node as the running workspace pod (using pod affinity or nodeSelector targeting the node where the shared PVC is currently mounted), so that both can access the ReadWriteOnce PVC simultaneously.

Additional context

• Passes on CRC (single-node): all pods share the same node, so RWO PVC mounting is not an issue
• Fails on multi-node AWS clusters: tested on OCP 4.20 with 6 worker nodes
• Per-workspace storage backup/restore works correctly on both single and multi-node clusters
• Tested during DWO pre-release testing (v0.42.0)
• Test results pattern:
  • ❌ Test 1-2: Per-User Storage + OCP Internal Registry — FAILED
  • ✅ Test 3-4: Per-Workspace Storage + OCP Internal Registry — PASSED
  • ❌ Test 5-6: Per-User Storage + External Registry (Quay) — FAILED
  • ✅ Test 7-8: Per-Workspace Storage + External Registry (Quay) — PASSED

Suggested fix: When creating a backup Job for per-user (common) storage, add a node affinity or pod affinity constraint to ensure the Job is scheduled on the same node where the shared PVC is currently mounted.