fix(infra): consolidate MinIO training secret, put output PVC under G…

[Enskc05]

fix(infra): consolidate MinIO training secret, put output PVC under GitOps

Cause

Two secrets in deephorizon-ml were carrying MinIO credentials for the same
purpose: minio-ml (user ml-team, referenced by the ML squad's Job
manifests) and minio-ml-credentials (user ml-trainer, referenced by our
runbook template). Carrying one credential under two names guarantees a
rotation eventually misses one.

The user mattered more than the name. ml-team is an account people also use
directly with mc/boto3; it is on the rotation list. The day its password
changes, every training Job breaks — with nothing in the failure tying it back
to the rotation.

Separately, training-outputs-pvc had been applied by hand and existed
nowhere in Git, so it would not survive a cluster rebuild.

Changes

• Runbook and example Job now reference minio-ml with
  access_key/secret_key, matching the naming the ML squad's manifests
  already use. Adopting their names rather than ours means fewer places change.
• The credential is ml-trainer — a read-only account (ml-read policy)
  opened solely for training. The runbook now explains why service identity is
  kept separate from human identity, and includes the three-way MinIO check
  (read allowed, write denied, out-of-scope denied) from DEVOPS.md Add UNet encoder and decoder blocks #8.
• training-outputs-pvc moves into infra/k8s/ml-training/, tracked by a new
  apps/ml-training.yaml Application. The PVC carries
  argocd.argoproj.io/sync-options: Prune=false — deleting it would take out
  the intermediate state of a run in progress.
• Training Jobs deliberately stay out of GitOps. Each run is ad-hoc with
  its own name and hyperparameters; handing them to Argo CD would recreate
  finished runs or prune one still going. This is now stated in both the
  Application manifest and the runbook.

Testing

ml-trainer verified against MinIO with the three-way check before the secret
was switched. A 100-epoch baseline run is currently training on the L40S using
these credentials — GPU at 100%, 7.7 GiB VRAM.

The PVC spec is copied verbatim from what is already applied in the cluster, so
Argo CD adopts the existing object rather than replacing it.

Note for whoever merges ml/feature

That branch predates the technology-based manifest reorganisation and still
carries infra/k8s/ml/kustomization.yaml, which lists namespace.yaml,
postgres.yaml, and mlflow.yaml. Merging it as-is resurrects
infra/k8s/ml/ alongside the current infra/k8s/mlflow/ and
infra/k8s/postgresql/mlflow/ — a second, stale copy of the MLflow and
Postgres definitions. No Application points there, so nothing applies it
automatically, but anyone running kubectl apply -k infra/k8s/ml/ would
overwrite the live MLflow with the old version. That directory should be
deleted on ml/feature before the merge.