fix(infra): raise MLflow memory limit; cut uvicorn workers to 2

[Enskc05]

fix(infra): raise MLflow memory limit; cut uvicorn workers to 2

MLflow was OOMKilled while receiving the first model artifact and entered
CrashLoopBackOff (4 restarts).

Cause

Artifacts are proxied through the tracking server
(--artifacts-destination with the default --serve-artifacts behaviour), so
the model file passes through MLflow's memory on its way to MinIO. A 2Gi limit
did not cover that, and MLflow 3.x starts four uvicorn workers by default —
each loading the app — so the baseline was already high before any upload.

The failure was self-sustaining: the training client retried the upload, and
each retry killed MLflow again as soon as it came back up. Deleting
The failure was self-sustaining: the training client retried the upl
each retry killed MLflow again as soon as it came back up. Deleting the
training Job is what breaks the loop; raising the limit alone would not have.

Changes

• Memory limit 2Gi → 6Gi, request 512Mi → 1Gi.
• --workers 2. A single-tenant tracking server does not need four; this
  roughly halves idle memory.
• Added a troubleshooting row to docs/runbooks/ML-TRAINING.md covering the
  symptom (RemoteDisconnected → Connection refused after training
  completes) and the order of recovery: delete the Job first.

Notes

6Gi is deliberately generous — the pod was deleted before the checkpoint size
could be measured. Once an upload succeeds, kubectl top pod -l app=mlflow
will show real peak usage and the limit can be tightened.

The underlying issue is architectural: proxying artifacts through the tracking
server does not scale with model size. The alternative is having cli
directly to MinIO, which docs/DEVOPS.md §10 deliberately avoided so the ML
side would not need S3 credentials. Worth revisiting as models grow; raising
the limit is the right move for now.