Skip to content

fix(infra): right-size MLflow memory limit to 4Gi from measurement - #33

Merged
Enskc05 merged 1 commit into
mainfrom
devops/fix
Aug 7, 2026
Merged

fix(infra): right-size MLflow memory limit to 4Gi from measurement#33
Enskc05 merged 1 commit into
mainfrom
devops/fix

Conversation

@Enskc05

@Enskc05 Enskc05 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

fix(infra): right-size MLflow memory limit to 4Gi from measurement

Follow-up to #32, which raised the limit to 6Gi. That number was a guess — the
pod had already been deleted, so the checkpoint size and peak memory could not
be measured at the time.

Measurement

With the fix from #32 in place (2 uvicorn workers) and a model upload that
completed successfully, Prometheus reports:

max_over_time(container_memory_working_set_bytes{
namespace="deephorizon-ml", container="mlflow"}[45m]) = 1.71 GiB

For comparison, the four pods that were OOMKilled under the old config all
peaked at 1.94–1.97 GiB — pinned against the 2Gi ceiling. So the limit was
genuinely the binding constraint, not a symptom of something else.

Change

  • Memory limit 6Gi4Gi (~2.3× the observed peak). Request stays at 1Gi.
  • Recorded the measured figure and the re-measurement query as a comment in the
    manifest. kubectl top does not work on this cluster — the metrics-server
    addon is not enabled — so the Prometheus query is the method, and
    next to the number it justifies.

Notes

This caps a chain that started with the training smoke test: the art
upload path now works end to end (params → Postgres, best_model.pt
MinIO through the server proxy), with the limit sized from data rath
guessed.

Proxying artifacts through the tracking server remains the scaling c
flagged in #32 — 1.71 GiB for a U-Net checkpoint sets the baseline t
models grow.

#32'de limit 6Gi'ye cikarilmisti; o sayi olculmeden, guvenli tarafta kalmak
icin secilmisti. Basarili model yuklemesi sonrasi Prometheus'tan olculdu:

  max_over_time(container_memory_working_set_bytes{
    namespace="deephorizon-ml",container="mlflow"}[45m]) = 1.71 GiB

4Gi bunun ~2.3 kati; tek GPU sunucusunda bosa rezervasyon yapmadan buyuyen
artifact'lara pay birakiyor. Olcum ve yeniden olcme sorgusu manifest'e
yorum olarak islendi — kubectl top bu kumede calismiyor (metrics-server
addon'u kapali), o yuzden yontem yaziyla kalsin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Enskc05
Enskc05 merged commit eef5d08 into main Aug 7, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant