Skip to content

fix: correct category labels, detector evaluation and IMM timing - #161

Merged
montge merged 8 commits into
developfrom
feature/training-pipeline-corrections
Sep 22, 2026
Merged

montge merged 8 commits into
developfrom
feature/training-pipeline-corrections

Conversation

@montge

@montge montge commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Correct the full OpenSky numeric-category mapping using mocked responses; keep the five-class taxonomy unchanged.
  • Run detector outputs through tracking on shared synthetic point clouds with independent trajectory truth; forward model paths/features and fail explicitly for invalid learned modes.
  • Introduce the shared 13-feature elapsed-time IMM contract, observation-only histories, accumulated missed-update gaps, and explicit legacy-model/data rejection.
  • Regenerate only synthetic/random fixtures, document provenance, and complete OpenSpec tasks 9.3 and 12.1–12.3 (65/73 complete overall).

Verification

  • Combined Python suite: 222 passed; Ruff and Pyright clean.
  • Final evaluator regressions: 7 passed; ONNX CLI integration: 8 passed; cross-language ONNX parity suite: 7 passed. Maximum output difference 0.0 at 0.1, 1.0 and 0.3 seconds.
  • Focused Rust filter 96, training/export 18, inference 69 (one existing ignored), tracker 109 passed. Independent timing-contract source review found no concrete correctness issues.
  • Formatting and feature-enabled Clippy pass; adapted fixed-size chunk calls for CI Clippy 1.98.1 (local toolchain is 1.97).
  • Strict OpenSpec validation: 52 live items and 26 archives pass.

Local broad-suite limitations

The feature-enabled workspace run excludes thresh-py. Its existing debug throughput gate failed at 3.8 Hz; unchanged baseline also fails locally at 5.2 Hz, below the unchanged 10 Hz threshold. Rerunning the remaining tests reached an existing cache-directory test denied by the sandbox for a home-directory write. Neither test nor threshold was weakened. Focused changes pass; full standard CI and optional-runtime checks remain authoritative merge gates.

Scope boundaries

No live OpenSky data fetched/stored, no trained checkpoint release, no crates.io publication. Representative trained-model acceptance remains open. Short CLI tests are smoke only; default synthetic duration stays 30 seconds. Targets develop and depends on cleanup PR #159, which should merge first.

Copilot AI lite review requested due to automatic review settings September 22, 2026 13:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 21 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 6ec39152-8984-4078-89ef-73972f7debef

📥 Commits

Reviewing files that changed from the base of the PR and between 5362265 and 174e51e.

⛔ Files ignored due to path filters (1)
  • test-data/training/imm-classifier/imm-samples.parquet is excluded by !**/*.parquet
📒 Files selected for processing (53)
  • .github/workflows/ci.yml
  • .pre-commit-config.yaml
  • TRAINING.md
  • crates/thresh-filter/README.md
  • crates/thresh-filter/src/imm.rs
  • crates/thresh-filter/src/imm_adapter.rs
  • crates/thresh-filter/tests/learned_imm.rs
  • crates/thresh-inference/src/detection.rs
  • crates/thresh/README.md
  • crates/thresh/src/bin/eval-tracker.rs
  • crates/thresh/src/bin/onnx-infer.rs
  • crates/thresh/src/eval_harness.rs
  • crates/thresh/src/training/mod.rs
  • crates/thresh/src/training/parquet_export.rs
  • crates/thresh/tests/eval_tracker_cli.rs
  • docs/eval/flight-data-training-pipeline.md
  • openspec/changes/archive/2026-05-18-cubature-kalman-filter/tasks.md
  • openspec/changes/crates-io-publishing/.openspec.yaml
  • openspec/changes/crates-io-publishing/design.md
  • openspec/changes/crates-io-publishing/proposal.md
  • openspec/changes/crates-io-publishing/specs/publishing-workflow/spec.md
  • openspec/changes/crates-io-publishing/tasks.md
  • openspec/changes/flight-data-training-pipeline/.openspec.yaml
  • openspec/changes/flight-data-training-pipeline/design.md
  • openspec/changes/flight-data-training-pipeline/proposal.md
  • openspec/changes/flight-data-training-pipeline/specs/flight-data-acquisition/spec.md
  • openspec/changes/flight-data-training-pipeline/specs/learned-detector/spec.md
  • openspec/changes/flight-data-training-pipeline/specs/learned-tracker-components/spec.md
  • openspec/changes/flight-data-training-pipeline/tasks.md
  • openspec/changes/learned-imm-tracker-integration/.openspec.yaml
  • openspec/changes/learned-imm-tracker-integration/proposal.md
  • openspec/changes/learned-imm-tracker-integration/specs/learned-imm-tracker/spec.md
  • openspec/changes/stonesoup-reference-review/.openspec.yaml
  • openspec/changes/stonesoup-reference-review/design.md
  • openspec/changes/stonesoup-reference-review/proposal.md
  • openspec/changes/stonesoup-reference-review/tasks.md
  • python/acquisition/opensky.py
  • python/eval/onnx_parity.py
  • python/eval/run_tracker.py
  • python/export/export_imm.py
  • python/tests/test_imm_dataset.py
  • python/tests/test_imm_training.py
  • python/tests/test_onnx_parity.py
  • python/tests/test_opensky.py
  • python/tests/test_run_tracker.py
  • python/training/imm_dataset.py
  • python/training/imm_model.py
  • scripts/generate_eval_detector_fixture.py
  • scripts/generate_imm_classifier_stub.py
  • test-data/models/MODEL_CARD.md
  • test-data/models/eval_single_detection.onnx
  • test-data/models/imm_mode_classifier.onnx
  • test-data/training/imm-classifier/README.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@montge
montge changed the base branch from feature/openspec-review-cleanup to develop September 22, 2026 13:41
@codecov

codecov Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.62385% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/thresh/src/eval_harness.rs 98.39% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@sonarqubecloud

Copy link
Copy Markdown

@montge
montge merged commit 3b8259b into develop Sep 22, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants