fix(infra): right-size MLflow memory limit to 4Gi from measurement - #33
Merged
Conversation
#32'de limit 6Gi'ye cikarilmisti; o sayi olculmeden, guvenli tarafta kalmak icin secilmisti. Basarili model yuklemesi sonrasi Prometheus'tan olculdu: max_over_time(container_memory_working_set_bytes{ namespace="deephorizon-ml",container="mlflow"}[45m]) = 1.71 GiB 4Gi bunun ~2.3 kati; tek GPU sunucusunda bosa rezervasyon yapmadan buyuyen artifact'lara pay birakiyor. Olcum ve yeniden olcme sorgusu manifest'e yorum olarak islendi — kubectl top bu kumede calismiyor (metrics-server addon'u kapali), o yuzden yontem yaziyla kalsin. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(infra): right-size MLflow memory limit to 4Gi from measurement
Follow-up to #32, which raised the limit to 6Gi. That number was a guess — the
pod had already been deleted, so the checkpoint size and peak memory could not
be measured at the time.
Measurement
With the fix from #32 in place (2 uvicorn workers) and a model upload that
completed successfully, Prometheus reports:
max_over_time(container_memory_working_set_bytes{
namespace="deephorizon-ml", container="mlflow"}[45m]) = 1.71 GiB
For comparison, the four pods that were
OOMKilledunder the old config allpeaked at 1.94–1.97 GiB — pinned against the 2Gi ceiling. So the limit was
genuinely the binding constraint, not a symptom of something else.
Change
6Gi→4Gi(~2.3× the observed peak). Request stays at 1Gi.manifest.
kubectl topdoes not work on this cluster — themetrics-serveraddon is not enabled — so the Prometheus query is the method, and
next to the number it justifies.
Notes
This caps a chain that started with the training smoke test: the art
upload path now works end to end (
params→ Postgres,best_model.pt→MinIO through the server proxy), with the limit sized from data rath
guessed.
Proxying artifacts through the tracking server remains the scaling c
flagged in #32 — 1.71 GiB for a U-Net checkpoint sets the baseline t
models grow.