Phase 2 Person 4: full-scale ablation fold-in + DeepLab multi-seed

Phase2/Lasana-Person4/README.md

@@ -0,0 +1,92 @@
+# Person 4 — Evaluation Lead (Phase 2)
+
+Role: **Evaluation Lead**. Proposal §6.2.2, Weeks 7–10: implement AAMO
+(done in Phase 1), run the complete ablation across all model configurations,
+and report results with multiple seeds (mean ± std).
+
+| Weeks | Deliverable | Status |
+|---|---|---|
+| 7–9 | Fold full-scale U-Net / SegFormer-B0 / SegFormer-B0+L_att into one table | Done — `results/baseline_comparison_full_scale.md` |
+| 7–9 | Attention-consistency row uses Dinura's λ2 sweep winner (`l2_1_mse`) | Done |
+| 7–9 | Multi-seed mean±std reporting | Done for DeepLabV3+ (seeds 42/43/44); U-Net/SegFormer remain Seeds=1 pending GPU reruns |
+| 10 | Boundary Loss ablation row | Pending — blocked on Person 5 integration |
+
+## Layout
+
+```
+Phase2/Lasana-Person4/
+  README.md
+  fold_full_scale_results.py   consolidates Table-1 rows (no model loading)
+  train_deeplab_multiseed.py   DeepLabV3+ seeds 42/43/44 + ablation tables
+  checkpoints/                 Deeplab seed 43/44 weights (gitignored *.pt)
+  results/
+    baseline_comparison_full_scale.csv / .md
+    ablation_per_seed.csv
+    ablation_mean_std.csv / .md
+    deeplab_multiseed.json
+  tests/
+    test_fold_results.py
+```
+
+Phase 1 code under `Phase1/Lasana-Person4_Evaluation/` is a **frozen short-paper
+snapshot** (CONTRIBUTING.md). This Phase 2 folder imports its metrics /
+adapters via `sys.path` and does not edit those files. Multi-seed aggregation
+uses a local `aggregate_mean_std` (sample std, ddof=1) rather than the Phase 1
+helper.
+
+## Quick start / Reproducing
+
+```bash
+cd Phase2/Lasana-Person4
+
+# Unit tests (no GPU, no weights)
+python tests/test_fold_results.py
+
+# Fold Chanupa + Kalana + Dinura winner + DeepLab into one table
+python fold_full_scale_results.py
+
+# Train DeepLab seeds 43/44 (seed 42 reuses Phase 1 ckpt), evaluate all three,
+# write ablation_per_seed + ablation_mean_std
+python train_deeplab_multiseed.py
+
+# Or aggregate only, if seed 43/44 checkpoints already exist
+python train_deeplab_multiseed.py --skip-train
+```
+
+Smoke defaults (override with env vars): `DEEPLAB_MAX_SAMPLES=400`,
+`DEEPLAB_EPOCHS=5`, `DEEPLAB_BATCH=2`.
+
+## Dependencies on teammates
+
+| Direction | What |
+|---|---|
+| I need | Kalana's full-scale SegFormer vanilla metrics (`Phase2/Kalana-Person2/results/`) |
+| I need | Dinura's λ2 sweep winner `l2_1_mse` (`Phase2/Dinura-Person3/results/`) |
+| I need | Chanupa's U-Net full-scale row (already in Phase 1 Person 4 results) |
+| I hand off | `results/baseline_comparison_full_scale.md` + `results/ablation_mean_std.md` for Person 5's Week 11–12 paper assembly |
+
+## Results
+
+U-Net / SegFormer / L_att rows share the 3576/766/766 seed-42 test set; DeepLabV3+
+is a 400-sample CPU-smoke subset (seed-42 Dice 0.7821 from `deeplab_multiseed.json`).
+
+| Model | Dice | IoU | AAMO | Seeds |
+|---|---|---|---|---|
+| U-Net (CNN baseline) | 0.8615 | 0.7568 | n/a | 1 |
+| SegFormer-B0 (no attention loss) | 0.8743 | 0.7766 | 0.0334 | 1 |
+| SegFormer-B0 + Attention Consistency (λ2=1.0 MSE) | 0.8577 | 0.7508 | 0.7476 | 1 |
+| SegFormer-B0 + Attention + Boundary Loss | — | — | pending | 0 |
+| DeepLabV3+ (MobileNetV3) — extra baseline | 0.7862 ± 0.0193 | 0.6481 ± 0.0264 | n/a | 3 |
+
+Selection rule for the attention row (Dinura): max test AAMO, then max Dice.
+Winner run tag `l2_1_mse` supersedes Kalana's default-λ2=0.3 attention numbers.
+
+DeepLabV3+ is CPU smoke-scale (400 samples / 5 epochs) and exists to exercise
+the multi-seed aggregation pipeline. U-Net / SegFormer full-scale weights live
+on Drive (GitHub 100 MiB limit); extra seeds for those rows need Colab GPU
+reruns from Person 1/2/3.
+
+## Cross-folder edits
+
+**None.** No file outside this folder was modified. Phase 1 Person 4 results
+are left untouched on purpose.

Phase2/Lasana-Person4/checkpoints/.gitignore

@@ -0,0 +1 @@
+*.pt

Phase2/Lasana-Person4/checkpoints/README.md

@@ -0,0 +1,12 @@
+# Checkpoints
+
+DeepLabV3+ multi-seed smoke weights land here as:
+
+- `deeplabv3_mobilenet_seed43_best.pt`
+- `deeplabv3_mobilenet_seed44_best.pt`
+
+Seed 42 reuses the Phase 1 checkpoint at
+`Phase1/Lasana-Person4_Evaluation/checkpoints/deeplabv3_mobilenet_best.pt`
+(not re-copied here).
+
+Do **not** commit `.pt` files. Commit metrics under `results/` instead.

Phase2/Lasana-Person4/fold_full_scale_results.py

@@ -0,0 +1,250 @@
+"""
+Fold full-scale Table-1 rows into Phase 2 Person 4 results.
+
+Sources (no model loading — table joins only):
+  - U-Net:          Phase1/Lasana-Person4_Evaluation/results/baseline_comparison.csv
+  - SegFormer-B0:   Phase2/Kalana-Person2/results/baseline_comparison.csv (vanilla)
+  - SegFormer+Att:  Phase2/Dinura-Person3/results/runs/l2_1_mse/  (sweep winner λ2=1.0 MSE)
+  - Boundary Loss:  pending (Dhinanjaya still integrating) — see TODO below
+  - DeepLabV3+:     results/deeplab_multiseed.json seed-42 entry (single eval path)
+
+Writes:
+  results/baseline_comparison_full_scale.csv
+  results/baseline_comparison_full_scale.md
+"""
+from __future__ import annotations
+
+import csv
+import json
+from pathlib import Path
+from typing import Any, Dict, List, Optional
+
+ROOT = Path(__file__).resolve().parent
+PROJECT = ROOT.parent.parent  # DNN_Project
+RESULTS = ROOT / "results"
+RESULTS.mkdir(parents=True, exist_ok=True)
+
+PHASE1_P4 = PROJECT / "Phase1" / "Lasana-Person4_Evaluation" / "results"
+KALANA = PROJECT / "Phase2" / "Kalana-Person2" / "results"
+DINURA_WIN = PROJECT / "Phase2" / "Dinura-Person3" / "results" / "runs" / "l2_1_mse"
+WINNING_CFG = PROJECT / "Phase2" / "Dinura-Person3" / "results" / "winning_config.json"
+DEEPLAB_MULTI = RESULTS / "deeplab_multiseed.json"
+
+# TODO(Person 5 / Dhinanjaya): once Boundary Refinement is wired into
+# SegFormer+L_att training and a baseline_comparison.csv row exists, replace
+# the pending boundary _row(...) below with a real source, e.g.:
+#   BOUNDARY = PROJECT / "Phase2" / "Dhinanjaya-Person5" / "results" / "baseline_comparison.csv"
+#   bound = _find_row(_read_csv_rows(BOUNDARY), "Boundary")
+# (exact path TBD when Dhinanjaya hands off the checkpoint / CSV.)
+
+FIELDS = [
+    "model",
+    "dice",
+    "iou",
+    "f1",
+    "precision",
+    "recall",
+    "pixel_acc",
+    "aamo",
+    "params",
+    "gflops",
+    "fps",
+    "ms_per_image",
+    "source",
+    "n_seeds",
+    "notes",
+]
+
+
+def _read_csv_rows(path: Path) -> List[Dict[str, str]]:
+    with open(path, newline="", encoding="utf-8") as f:
+        return list(csv.DictReader(f))
+
+
+def _find_row(rows: List[Dict[str, str]], substr: str) -> Optional[Dict[str, str]]:
+    for r in rows:
+        if substr in (r.get("model") or ""):
+            return r
+    return None
+
+
+def _deeplab_seed42_row() -> Dict[str, Any]:
+    """Single evaluation path: seed-42 entry from train_deeplab_multiseed.py."""
+    if not DEEPLAB_MULTI.exists():
+        raise FileNotFoundError(
+            f"Missing {DEEPLAB_MULTI}\n"
+            "Run: python train_deeplab_multiseed.py --skip-train"
+        )
+    entries = json.loads(DEEPLAB_MULTI.read_text(encoding="utf-8"))
+    for e in entries:
+        if int(e.get("seed", -1)) == 42:
+            return e
+    raise ValueError(f"No seed=42 entry in {DEEPLAB_MULTI}")
+
+
+def _row(
+    model: str,
+    src: Optional[Dict[str, str]],
+    *,
+    source: str,
+    n_seeds: int = 1,
+    notes: str = "",
+    overrides: Optional[Dict[str, Any]] = None,
+) -> Dict[str, Any]:
+    out: Dict[str, Any] = {k: "" for k in FIELDS}
+    out["model"] = model
+    out["source"] = source
+    out["n_seeds"] = n_seeds
+    out["notes"] = notes
+    if src:
+        for k in FIELDS:
+            if k in ("model", "source", "n_seeds", "notes"):
+                continue
+            if k in src and src[k] not in (None, ""):
+                out[k] = src[k]
+    if overrides:
+        out.update(overrides)
+    return out
+
+
+def fold() -> List[Dict[str, Any]]:
+    p4 = _read_csv_rows(PHASE1_P4 / "baseline_comparison.csv")
+    kalana = _read_csv_rows(KALANA / "baseline_comparison.csv")
+    dinura = _read_csv_rows(DINURA_WIN / "baseline_comparison.csv")
+
+    winner_meta = {}
+    if WINNING_CFG.exists():
+        winner_meta = json.loads(WINNING_CFG.read_text(encoding="utf-8")).get("winner", {})
+
+    unet = _find_row(p4, "U-Net")
+    vanilla = _find_row(kalana, "no attention")
+    att = _find_row(dinura, "Attention Consistency")
+    dl42 = _deeplab_seed42_row()
+
+    att_notes = (
+        f"Dinura sweep winner {winner_meta.get('run_tag', 'l2_1_mse')} "
+        f"(λ2={winner_meta.get('lambda2', 1.0)}, {winner_meta.get('att_mode', 'mse')}); "
+        "selection: max test AAMO then max Dice. "
+        "Supersedes Kalana default-λ2=0.3 attention row."
+    )
+
+    rows = [
+        _row(
+            "U-Net (CNN baseline)",
+            unet,
+            source="Phase1/Lasana-Person4_Evaluation (Chanupa PyTorch ckpt)",
+            notes="Full-scale 3576/766/766 seed 42; dataset-wide Dice/IoU.",
+        ),
+        _row(
+            "SegFormer-B0 (no attention loss)",
+            vanilla,
+            source="Phase2/Kalana-Person2 (full-scale Colab)",
+            notes="Full-scale 3576/766/766 seed 42; Person 4 metrics/aamo formulas.",
+        ),
+        _row(
+            "SegFormer-B0 + Attention Consistency Loss (λ2=1.0 MSE)",
+            att,
+            source="Phase2/Dinura-Person3/results/runs/l2_1_mse",
+            notes=att_notes,
+        ),
+        # TODO(Person 5): replace this pending stub when Boundary Refinement
+        # lands. Expected upstream CSV (path TBD at handoff):
+        #   Phase2/Dhinanjaya-Person5/results/baseline_comparison.csv
+        # Then: bound = _find_row(_read_csv_rows(BOUNDARY), "Boundary")
+        # and drop overrides / n_seeds=0.
+        _row(
+            "SegFormer-B0 + Attention Consistency + Boundary Loss",
+            None,
+            source="pending",
+            n_seeds=0,
+            notes="Blocked until Dhinanjaya wires Boundary Refinement into training.",
+            overrides={
+                "dice": "-",
+                "iou": "-",
+                "f1": "-",
+                "aamo": "pending",
+                "params": "-",
+                "gflops": "-",
+                "fps": "-",
+            },
+        ),
+        _row(
+            "DeepLabV3+ (MobileNetV3) — extra baseline",
+            None,
+            source="Phase2/Lasana-Person4/train_deeplab_multiseed.py (seed 42, 400-sample smoke)",
+            notes=(
+                "CPU smoke (400 samples / 5 epochs), seed-42 eval from "
+                "deeplab_multiseed.json. Multi-seed mean±std in ablation_mean_std.md."
+            ),
+            overrides={
+                "dice": dl42["dice"],
+                "iou": dl42["iou"],
+                "f1": dl42["f1"],
+                "precision": dl42.get("precision", ""),
+                "recall": dl42.get("recall", ""),
+                "pixel_acc": dl42.get("pixel_acc", ""),
+                "aamo": dl42.get("aamo", "n/a"),
+                "params": dl42.get("params", ""),
+                "gflops": dl42.get("gflops", "n/a"),
+            },
+        ),
+    ]
+    return rows
+
+
+def write_tables(rows: List[Dict[str, Any]]) -> None:
+    csv_path = RESULTS / "baseline_comparison_full_scale.csv"
+    md_path = RESULTS / "baseline_comparison_full_scale.md"
+
+    with open(csv_path, "w", newline="", encoding="utf-8") as f:
+        w = csv.DictWriter(f, fieldnames=FIELDS)
+        w.writeheader()
+        w.writerows(rows)
+
+    lines = [
+        "# Full-scale baseline comparison (Phase 2 / Lasana-Person4)",
+        "",
+        "Folded from Chanupa (U-Net), Kalana (SegFormer-B0 vanilla), Dinura",
+        "(`l2_1_mse` attention winner), and Person 4's DeepLabV3+ extra baseline.",
+        "U-Net / SegFormer / L_att rows share the 3576/766/766 seed-42 test set;",
+        "the DeepLabV3+ row is a 400-sample CPU-smoke subset evaluated by",
+        "`train_deeplab_multiseed.py`.",
+        "",
+        "| Model | Dice | IoU | F1 | AAMO | Params | GFLOPs | FPS | Source |",
+        "|-------|------|-----|----|------|--------|--------|-----|--------|",
+    ]
+    for r in rows:
+        lines.append(
+            f"| {r['model']} | {r.get('dice', '-')} | {r.get('iou', '-')} | "
+            f"{r.get('f1', '-')} | {r.get('aamo', '-')} | {r.get('params', '-')} | "
+            f"{r.get('gflops', '-')} | {r.get('fps', '-')} | {r.get('source', '')} |"
+        )
+    lines.extend(
+        [
+            "",
+            "## Notes",
+            "",
+            "- Attention-consistency row uses Dinura's sweep winner "
+            "(`λ2=1.0`, MSE, run tag `l2_1_mse`: Dice 0.8577 / IoU 0.7508 / "
+            "AAMO 0.7476), not Kalana's default-λ2=0.3 attention run.",
+            "- Boundary Loss row stays pending until Person 5 finishes integration.",
+            "- DeepLabV3+ Dice/IoU come from `deeplab_multiseed.json` seed 42 "
+            "(same path as the multi-seed ablation); see `ablation_mean_std.md` "
+            "for seeds 42/43/44 mean±std.",
+            "- Phase 1 `Lasana-Person4_Evaluation/results/` is left untouched "
+            "(frozen short-paper snapshot per CONTRIBUTING.md).",
+            "",
+        ]
+    )
+    md_path.write_text("\n".join(lines), encoding="utf-8")
+    print(f"Wrote {csv_path}")
+    print(f"Wrote {md_path}")
+
+
+def main() -> None:
+    rows = fold()
+    write_tables(rows)
+
+
+if __name__ == "__main__":
+    main()

Phase2/Lasana-Person4/results/ablation_mean_std.csv

@@ -0,0 +1,6 @@
+model,n_seeds,dice,iou,f1,aamo,params,gflops
+U-Net (CNN baseline),1,0.8615,0.7568,0.8615,n/a,31037698,109.48
+SegFormer-B0 (no attention loss),1,0.8743,0.7766,0.8743,0.0334,3714658,1.692
+SegFormer-B0 + Attention Consistency Loss (λ2=1.0 MSE),1,0.8577,0.7508,0.8577,0.7476,3714658,1.692
+SegFormer-B0 + Attention Consistency + Boundary Loss,0,-,-,-,pending,-,-
+DeepLabV3+ (MobileNetV3) — extra baseline,3,0.7862 ± 0.0193,0.6481 ± 0.0264,0.7862 ± 0.0193,n/a,11020594,n/a

Phase2/Lasana-Person4/results/ablation_mean_std.md

@@ -0,0 +1,25 @@
+# Ablation results (mean ± std) — Phase 2 / Lasana-Person4
+
+Std is **sample** standard deviation (ddof=1). Single-seed rows
+(U-Net, SegFormer-B0, SegFormer-B0+L_att λ2=1.0) show the bare value
+pending additional GPU-trained seeds from Person 1/2/3. DeepLabV3+
+extra baseline has genuine 3-seed mean±std (seeds 42/43/44, CPU smoke:
+400 samples / 5 epochs).
+
+| Model | Seeds | Dice | IoU | F1 | AAMO |
+|-------|-------|------|-----|----|------|
+| U-Net (CNN baseline) | 1 | 0.8615 | 0.7568 | 0.8615 | n/a |
+| SegFormer-B0 (no attention loss) | 1 | 0.8743 | 0.7766 | 0.8743 | 0.0334 |
+| SegFormer-B0 + Attention Consistency Loss (λ2=1.0 MSE) | 1 | 0.8577 | 0.7508 | 0.8577 | 0.7476 |
+| SegFormer-B0 + Attention Consistency + Boundary Loss | 0 | - | - | - | pending |
+| DeepLabV3+ (MobileNetV3) — extra baseline | 3 | 0.7862 ± 0.0193 | 0.6481 ± 0.0264 | 0.7862 ± 0.0193 | n/a |
+
+## Notes
+
+- Attention-consistency config = Dinura `l2_1_mse` (λ2=1.0, MSE).
+- Boundary Loss row pending Person 5 integration.
+- DeepLabV3+ smoke numbers are not paper-scale; they demonstrate
+  the multi-seed aggregation pipeline Person 4 owns.
+- U-Net / SegFormer full-scale checkpoints live on Drive (not in git);
+  re-running extra seeds requires Colab GPU access from teammates.
+- Teammate GPU multi-seed reporting must also use sample std (ddof=1).