ReplicasScheduled stays True on a partial shortfall, so a drain that can't finish is not visible on the condition

[nicknikolakakis]

What happened?

docs/content/platform/drain-cluster.md (lines 53-56) says:

When no other cluster can take a replica, because every candidate is full or tainted, the deployment runs below its spec.replicas until capacity frees up. Its ReplicasScheduled condition reports the shortfall, so a drain that can't finish is visible rather than silent.

The condition's message reports the shortfall. Its status does not. I had two deployments in the same namespace on one control plane, with one of the two inference clusters tainted NoExecute, and read both at the same moment:

mock-demo     status=True   reason=ReplicasCreated       msg=Scheduled 1 of 2 replicas
mock-demo-v2  status=False  reason=InsufficientCapacity  msg=0 of 1 replicas scheduled (checked 2 clusters)

So:

Shortfall	status	reason
Some replicas placed, not all	True	ReplicasCreated
Zero replicas placed	False	InsufficientCapacity

The partial case is the one drain-cluster.md is describing, and it is the one where the boolean reads healthy. A deployment running at half its requested replicas reports ReplicasScheduled=True.

The practical consequence is that "visible rather than silent" invites an alert on the condition status, and that alert will not fire for the partial drain. Today you have to parse the message string or compare status.replicas.ready against spec.replicas.

I don't have a strong opinion on which end should move. Either the docs should say the condition is only False when nothing could be placed, or the condition should go False whenever status.replicas.ready < spec.replicas. The second is more useful to alert on but is a behaviour change.

Related to #35, which raises the general problem of surfacing scheduling failures. This is a specific, measured instance of it.

How can we reproduce it?

1. Register two InferenceClusters, one schedulable node each, both publishing the same InferenceClass.
2. Apply a ModelDeployment with spec.replicas: 2. It places one replica per cluster.
3. Taint one cluster:

   kubectl patch inferencecluster <name> --type merge \
     -p '{"spec":{"taints":[{"key":"modelplane.ai/maintenance","effect":"NoExecute"}]}}'

4. The evicted replica has nowhere to go, since the other cluster's single node is occupied. Read the condition:

   kubectl -n <ns> get modeldeployment -o jsonpath='{range .items[*]}{.metadata.name}  {range .status.conditions[?(@.type=="ReplicasScheduled")]}status={.status}  reason={.reason}  msg={.message}{end}{"\n"}{end}'

5. Observe status=True with Scheduled 1 of 2 replicas.

The False case appears when a deployment can place nothing at all. In my run that came from a second ModelDeployment with spec.replicas: 1 that was already on the cluster for unrelated reasons while the taint was in place, which is how both rows above came from a single read. I have not re-run the two cases as a scripted sequence from a clean cluster, so treat step 5 as the reproduction and the second deployment as the contrast I happened to have.

Workaround: alert on status.replicas.ready vs spec.replicas rather than on the condition.

What environment did it happen in?

Modelplane version: v0.3.1 (xpkg.upbound.io/modelplane/modelplane:v0.3.1)
Crossplane: 2.4.0
Kubernetes: kind, kindest/node:v1.34.0, control plane plus two workload clusters
Inference clusters: source: Existing (BYO, registered by kubeconfig)
Providers: provider-helm and provider-kubernetes from the upbound org, as shipped with v0.3.1
GPUs: none. dra-example-driver publishing fake gpu.example.com devices