Skip to content

Add optional ultrafast backend for online and offline COCO evaluation - #620

Open
developer0hye wants to merge 1 commit into
facebookresearch:mainfrom
developer0hye:feat/ultrafast-coco-backend
Open

developer0hye wants to merge 1 commit into
facebookresearch:mainfrom
developer0hye:feat/ultrafast-coco-backend

Conversation

@developer0hye

Copy link
Copy Markdown

This adds an optional backend="ultrafast" to the online CocoEvaluator and offline CocoEvaluatorOfflineWithPredFileEvaluators. Existing configurations keep pycocotools. The ultrafast extra installs ultrafast-pycocotools >=0.1.11,<0.2 alongside the reference utilities that SAM3 still uses. I maintain ultrafast-pycocotools.

The online adapter gathers prepared predictions through the existing collective/filesystem transports, keeps the first occurrence of duplicate image IDs, and evaluates on rank zero. It includes GT images missing from predictions and preserves SAM3's normalized segmentation areas instead of recomputing pixel areas. Existing custom summaries, exhaustive filtering, rarity averages, reset, prediction dumps, and optional per-image metric dumps remain available. Distinct image subsets are evaluated separately. Detection IDs in metric dumps are assigned over merged predictions instead of restarting per update batch.

The offline adapter explicitly applies COCOevalCustom's positive-split filter before native matching; changing the superclass alone would bypass this filter. TIDE analysis remains on its existing path. Separate Demo/F1 and tracking evaluators are unchanged.

While testing rarity/subset evaluation, I reproduced an existing default-backend bug: accumulate(imgIds=...) slices evalImgs but leaves _paramsEval.imgIds at its original length. The reference accumulator then uses the wrong per-category/per-area stride and raises IndexError. This PR updates the parameter snapshot for the sliced records, restores state afterward, and verifies the result against an independent pycocotools evaluation of the selected images.

Validation: 32 passed, 2 skipped on Linux, Python 3.12, PyTorch 2.10.0+cpu / torchvision 0.25.0+cpu, NumPy 1.26.4, pycocotools 2.0.11, ultrafast-pycocotools 0.1.11. The tests import the real SAM3 package, without model/package stubs or checkpoints.

  • Full precision/recall/score arrays and custom summaries for bbox/segmentation (plus category-aware keypoints), normalized and pixel area ranges, custom caps, ties, crowd annotations, missing/empty predictions, duplicates, and two epochs.
  • All six normalized size buckets have valid objects; each bucket is checked, not only the overall AP.
  • Rarity/exhaustive evaluation, lazy GT file loading, subset/full restoration, and JSON dump readback.
  • Offline positive splits and the public prediction-file API.
  • Actual two-rank Gloo and shared-filesystem gather, including a rank with no predictions. Two category-agnostic keypoint combinations are explicitly skipped because reference computeOks does not support that mode.
  • Added focused CPU CI; repository-pinned ufmt checks and focused lint pass (pre-existing unused-import/type-comparison findings excluded).

No framework-level speed or peak-memory claim is made. A performance comparison must include the complete lifecycle, including gathering, subset evaluation and optional dumps. The existing GPU-capable mask encoder is retained.

Signed-off-by: Yonghye Kwon <developer.0hye@gmail.com>
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant