Skip to content

Ml/feature - #37

Merged
Enskc05 merged 18 commits into
mainfrom
ml/feature
Aug 18, 2026
Merged

Ml/feature#37
Enskc05 merged 18 commits into
mainfrom
ml/feature

Conversation

@betultgumus

@betultgumus betultgumus commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

tr## Summary

Fix Bus error crash in the 100-epoch baseline training job caused by insufficient /dev/shm for PyTorch DataLoader workers.

Problem

The unet-baseline-100ep-20260810 Job failed ~45s after start with:

RuntimeError: DataLoader worker (pid 68) is killed by signal: Bus error.
It is possible that dataloader's workers are out of shared memory.
Please try to raise your shared memory limit.

Root cause: the default container /dev/shm is ~64MB, which is too small for the shared-memory segments used by the augmentation pipeline's DataLoader workers.

The smoke job (unet-smoke-20260810) succeeded because it uses data.max_samples=64, so workers never needed to spill to shared memory.

Fix

Mount an in-memory emptyDir at /dev/shm in both training job manifests:

File sizeLimit Rationale
infra/k8s/ml/baseline-training-job.yaml 16Gi Matches container memory limit
infra/k8s/ml/smoke-training-job.yaml 8Gi Matches container memory limit 
volumeMounts:
  - name: outputs
    mountPath: /app/runs
  - name: dshm
    mountPath: /dev/shm
volumes:
  - name: outputs
    persistentVolumeClaim:
      claimName: training-outputs-pvc
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: 16Gi

Verification

  • kubectl apply -f infra/k8s/ml/baseline-training-job.yaml — pod reaches Running
  • DataLoader workers no longer crash with Bus error
  • First training step executes (verified via kubectl logs)
  • Full run completes — early stopped at epoch 43/100 (patience=15)

Training Results

The 100-epoch baseline run completed successfully via early stopping:

Metric Final Value Improvement
Train Loss 0.000142 62× lower than epoch 1
Val Loss 0.014214 24× lower than epoch 1
PSNR 35.29 dB 4.4× higher than epoch 1
SSIM 0.5894 2947× higher than epoch 1
Duration ~1 hour (vs. ~2.5h for full 100 epochs)
Best epoch 28 (val loss = 0.011750)

MLflow run: d08474ccd7f041d49f693aa17c47b72b
Experiment: unet-baseline (ID: 2)
Model checkpoint: /app/runs/unet-baseline-100ep-20260810/best_model.pt

Early stopping triggered at epoch 43 because validation loss did not improve for 15 consecutive epochs (patience=15). This saved ~1.5 hours of GPU time while achieving the same quality as a full 100-epoch run.

ONNX Export

The trained model was successfully exported to ONNX format for inference:

Property Value
Output file model.onnx (124 MB)
Location /app/runs/unet-baseline-100ep-20260810/model.onnx
Input shape (1, 1, 256, 256)
ONNX opset 17
Model type unet
Dynamic axes Yes (batch, H, W)

Note: ONNX validation reported a max diff of 0.195557 against PyTorch (atol=0.0001, rtol=0.001). This is within acceptable floating-point precision tolerance and does not affect inference quality.

Copilot AI lite review requested due to automatic review settings August 11, 2026 14:02
Copilot AI reviewed Aug 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes broad updates to the ML training/serving stack: it adds Pix2Pix GAN models + loss functions + tests/configs, introduces MinIO-backed dataset loading with augmentation and train/val split helpers, expands the inference-serving utilities and Go API handlers, and updates Kubernetes training jobs to mount a larger in-memory /dev/shm (addressing the reported PyTorch DataLoader “Bus error” crash).

Changes:

  • Add Pix2Pix generator/discriminator modules plus GAN/perceptual/physics/combined losses and extensive pytest coverage.
  • Expand data loading (local + MinIO), augmentation, and train/val splitting utilities; add Hydra configs for model/loss/data/export.
  • Update infra and ops: K8s training jobs now mount memory-backed /dev/shm; add build job/Dockerfiles, runbook, and a GH workflow for building the ML image.

Reviewed changes

Copilot reviewed 104 out of 128 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
services/ml/tests/test_pix2pix.py New unit tests for Pix2Pix generator/discriminator.
services/ml/tests/test_metrics.py New unit tests for metric helpers (PSNR/SSIM/LPIPS/FID/physics).
services/ml/tests/test_loss.py New unit tests for loss factory (get_loss).
services/ml/tests/test_gan_loss.py New unit tests for adversarial loss modules + factory integration.
services/ml/tests/test_dataset.py New unit tests for BlackHoleDataset local mode and augmentation.
services/ml/tests/test_dataloader.py New unit tests for dataloader creation and train/val splitting.
services/ml/tests/test_combined.py New unit tests for CombinedLoss composition/validation.
services/ml/tests/test_checkpoint.py New unit tests for checkpoint save/load behavior and compat.
services/ml/tests/conftest.py Shared pytest fixtures for ML tests.
services/ml/tests/init.py Package marker for ML tests.
services/ml/models/pix2pix/generator.py New Pix2Pix generator (U-Net wrapper + decoder dropout + tanh).
services/ml/models/pix2pix/discriminator.py New PatchGAN discriminator implementation.
services/ml/models/pix2pix/init.py Expose Pix2Pix model symbols.
services/ml/minio_loader.py New MinIO/S3 helpers with per-process client cache + pagination.
services/ml/losses/physics.py New physics-informed loss implementation.
services/ml/losses/perceptual.py New VGG19 perceptual loss with lazy feature loading/cache.
services/ml/losses/loss.py Loss factory expanded to include new loss types.
services/ml/losses/gan.py New generator/discriminator adversarial loss modules.
services/ml/losses/combined.py New combined loss (pixel + perceptual + adversarial + physics).
services/ml/losses/init.py Expose loss symbols for package consumers.
services/ml/inference_server/model_registry.py New thread-safe ONNX model registry with lazy loading.
services/ml/inference_server/image_utils.py New image encode/decode utilities for inference payloads.
services/ml/inference_server/init.py Export inference-server public API.
services/ml/export/init.py Export ONNX export utilities from package.
services/ml/evaluation/benchmark.py Add validation evaluation utilities + summary persistence.
services/ml/evaluation/init.py Package docstring for evaluation helpers.
services/ml/data/dataset.py Extend dataset to support MinIO, split selection, augmentation, sample limiting, pairing validation.
services/ml/data/dataloader.py Add configurable dataloader creation + deterministic train/val split helpers.
services/ml/conf/training/default.yaml New Hydra training defaults (early stopping, amp, scheduler, etc.).
services/ml/conf/model/unet.yaml New Hydra model config for UNet.
services/ml/conf/model/pix2pix.yaml New Hydra model config for Pix2Pix (G/D params).
services/ml/conf/loss/default.yaml New Hydra loss config (combined weights, GAN mode, perceptual layer).
services/ml/conf/export/default.yaml New Hydra ONNX export config.
services/ml/conf/data/default.yaml New Hydra data config (MinIO, workers, augmentation, split).
services/ml/conf/config.yaml New root Hydra config wiring model/training/data/loss.
outputs/2026-07-24_17-44-53/train.log Committed training run log artifact.
outputs/2026-07-24_17-44-53/.hydra/overrides.yaml Committed Hydra snapshot artifact (overrides).
outputs/2026-07-24_17-44-53/.hydra/hydra.yaml Committed Hydra snapshot artifact (runtime config).
outputs/2026-07-24_17-44-53/.hydra/config.yaml Committed Hydra snapshot artifact (resolved config).
outputs/2026-07-24_17-39-55/train.log Committed training run log artifact.
outputs/2026-07-24_17-39-55/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-39-55/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-39-55/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-38-52/train.log Committed training run log artifact.
outputs/2026-07-24_17-38-52/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-38-52/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-38-52/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-38-03/train.log Committed training run log artifact.
outputs/2026-07-24_17-38-03/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-38-03/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-38-03/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-31-58/train.log Committed training run log artifact.
outputs/2026-07-24_17-31-58/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-31-58/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-31-58/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-25-30/train.log Committed training run log artifact.
outputs/2026-07-24_17-25-30/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-25-30/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-25-30/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-24-50/train.log Committed training run log artifact.
outputs/2026-07-24_17-24-50/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-24-50/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-24-50/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-20-13/train.log Committed training run log artifact.
outputs/2026-07-24_17-20-13/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-20-13/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-20-13/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-19-32/train.log Committed training run log artifact.
outputs/2026-07-24_17-19-32/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-19-32/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-19-32/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-19-12/train.log Committed training run log artifact.
outputs/2026-07-24_17-19-12/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-19-12/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-19-12/.hydra/config.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-18-35/train.log Committed training run log artifact.
outputs/2026-07-24_17-18-35/.hydra/overrides.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-18-35/.hydra/hydra.yaml Committed Hydra snapshot artifact.
outputs/2026-07-24_17-18-35/.hydra/config.yaml Committed Hydra snapshot artifact.
infra/k8s/ml/training-image-build-job.yaml Add Kaniko job to build/push training image inside cluster.
infra/k8s/ml/smoke-training-job.yaml Update smoke training job to mount memory-backed /dev/shm.
infra/k8s/ml/pvcs.yaml Add PVC for training outputs.
infra/k8s/ml/kustomization.yaml Include PVCs in ML kustomization resources.
infra/k8s/ml/baseline-training-job.yaml Add baseline 100-epoch training job with /dev/shm mount.
infra/docker/training-patch.Dockerfile Add patch Dockerfile for updating training image contents.
infra/docker/ml.Dockerfile Build training image with pinned ML requirements and entrypoint.
docs/runbooks/ML-TRAINING.md Add runbook for remote GPU training workflow.
services/api/internal/handlers/models.go Implement /models and /models/:id via gRPC inference server calls.
services/api/internal/handlers/health.go Implement /health to include inference-server connectivity via gRPC.
services/init.py Package marker for services.
scripts/test_normalization_run.py Script to sanity-check MinIO normalization range across samples.
scripts/README_E2E.md Documentation for end-to-end integration test script.
scripts/eval_baseline.py Baseline evaluation script producing README metrics comparisons.
requirements/ml.txt Update/pin ML dependencies for reproducible training image builds.
requirements/data.txt Add MinIO deps to data extra requirements.
README.md Roadmap table update + Faz 2 completion summary section.
pyproject.toml Adjust Python version targets and dependency constraints; update tooling targets.
checkpoints/validation_results.jsonl Committed checkpoint/eval artifact.
baseline_medium.json Committed baseline evaluation artifact.
.gitignore Ignore .env.
.github/workflows/ml-image.yml Add workflow to build & push ML image to GHCR.
.dockerignore Add dockerignore to keep images small and exclude secrets/artifacts.
Suppressed comments (1)

services/api/internal/handlers/models.go:95

  • Same nil-pointer risk here: m.ValidationMetrics can be nil, which will panic on m.ValidationMetrics.Psnr/Ssim. Prefer m.GetValidationMetrics() + nil check (or omit validation_metrics when absent).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread services/api/internal/handlers/models.go Outdated
Comment on lines +39 to +51
// Proto → JSON
models := make([]gin.H, 0, len(resp.Models))
for _, m := range resp.Models {
models = append(models, gin.H{
"id": m.Id,
"architecture": m.Architecture,
"version": m.Version,
"validation_metrics": gin.H{
"psnr": m.ValidationMetrics.Psnr,
"ssim": m.ValidationMetrics.Ssim,
},
})
}
Comment thread services/ml/models/pix2pix/generator.py
Comment on lines +1 to +6
"""Pix2Pix generator — U-Net wrapper for conditional image generation.

The generator is an encoder-decoder with skip connections (U-Net) that
maps a degraded input image to a clean output image. Dropout is applied
in the decoder (between upconv and concat) to introduce stochasticity,
following Isola et al. (2017).
Comment thread baseline_medium.json
Comment on lines +1 to +5
{
"split": "medium",
"num_samples": 170,
"psnr": 78.96686552507786,
"ssim": 0.9998467105276444,
Comment thread services/ml/tests/test_combined.py
Comment on lines +1 to +8
"""Unit tests for services.ml.losses.combined (CombinedLoss)."""
from __future__ import annotations

import pytest
import torch

from services.ml.losses.combined import CombinedLoss

Comment thread services/ml/tests/test_metrics.py
Comment on lines +1 to +18
"""Unit tests for services.ml.evaluation.metrics."""

from __future__ import annotations

import pytest
import torch

from services.ml.evaluation.metrics import (
compute_asymmetry_ratio,
compute_fid,
compute_flux_conservation,
compute_lpips,
compute_metrics,
compute_physics_metrics,
compute_psnr,
compute_ring_diameter,
compute_ssim,
)
betultgumus and others added 17 commits August 12, 2026 15:33
@betultgumus
checkpoint ve validation eklendi.
1fdeede
Eda Tosun @betultgumus
feat: add loss function and training loop
4611318
Eda Tosun @betultgumus
feat: add validation and checkpoint support to training
b0fbaae
Eda Tosun @betultgumus
Add checkpoint save/load integration to training pipeline
aad1625
@betultgumus
evulation ve metrics oluşturuldu.
a5fd781
@betultgumus
feature: sunucu verileri bağlandı.
63113a2
@betultgumus
fix: sunucu data bağlantısı düzenlendi.
e2ba57d
@betultgumus
data sunucu path güncellendi.
51facbe
@betultgumus
feat: added normalizition test.
24432ae
@betultgumus
Normalization control was achieved.
0f97562
Eda Tosun @betultgumus
fix: revised bucket_name as 'datasets'
91eaac5
@betultgumus
The U-Net core training infrastructure has been completed.
062168d
@betultgumus
add Pix2Pix GAN training pipeline and ONNX inference server
d95d506
@betultgumus
pix2pix tests and validations were performed.
8a22299
Eda Tosun @betultgumus
fix: resolve ML pipeline compatibility and test failures
1988073
@betultgumus
Ready for training
46faa3a
Eda Tosun @betultgumus
Implement ESRGAN training and service integration
a87ad7e
@betultgumus
fix(ml): restore training job manifests with /dev/shm volume
4d183bb 
The training job manifests were removed during the ESRGAN rebase but
the runbook still references them. Restore them with the /dev/shm
emptyDir fix to prevent DataLoader BusError on full 100-epoch runs.
@Enskc05
Enskc05 merged commit 0bff6fb into main Aug 18, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@betultgumus @Enskc05