Add optional ultrafast backend for online and offline COCO evaluation - #620
Open
developer0hye wants to merge 1 commit into
Open
developer0hye wants to merge 1 commit into
developer0hye wants to merge 1 commit into
Conversation
Signed-off-by: Yonghye Kwon <developer.0hye@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds an optional
backend="ultrafast"to the onlineCocoEvaluatorand offlineCocoEvaluatorOfflineWithPredFileEvaluators. Existing configurations keep pycocotools. Theultrafastextra installs ultrafast-pycocotools >=0.1.11,<0.2 alongside the reference utilities that SAM3 still uses. I maintain ultrafast-pycocotools.The online adapter gathers prepared predictions through the existing collective/filesystem transports, keeps the first occurrence of duplicate image IDs, and evaluates on rank zero. It includes GT images missing from predictions and preserves SAM3's normalized segmentation areas instead of recomputing pixel areas. Existing custom summaries, exhaustive filtering, rarity averages, reset, prediction dumps, and optional per-image metric dumps remain available. Distinct image subsets are evaluated separately. Detection IDs in metric dumps are assigned over merged predictions instead of restarting per update batch.
The offline adapter explicitly applies
COCOevalCustom's positive-split filter before native matching; changing the superclass alone would bypass this filter. TIDE analysis remains on its existing path. Separate Demo/F1 and tracking evaluators are unchanged.While testing rarity/subset evaluation, I reproduced an existing default-backend bug:
accumulate(imgIds=...)slicesevalImgsbut leaves_paramsEval.imgIdsat its original length. The reference accumulator then uses the wrong per-category/per-area stride and raisesIndexError. This PR updates the parameter snapshot for the sliced records, restores state afterward, and verifies the result against an independent pycocotools evaluation of the selected images.Validation: 32 passed, 2 skipped on Linux, Python 3.12, PyTorch 2.10.0+cpu / torchvision 0.25.0+cpu, NumPy 1.26.4, pycocotools 2.0.11, ultrafast-pycocotools 0.1.11. The tests import the real SAM3 package, without model/package stubs or checkpoints.
computeOksdoes not support that mode.No framework-level speed or peak-memory claim is made. A performance comparison must include the complete lifecycle, including gathering, subset evaluation and optional dumps. The existing GPU-capable mask encoder is retained.