DINO / RT-DETR: replace dense global encoder attention with multi-scale deformable attention and AIFI

[ooples]

Summary

DINO and RT-DETR run dense global self-attention over every pyramid token in their encoders. At the default 640x640 input that is 80² + 40² + 20² = 8,400 tokens: an 8,400 x 8,400 score matrix per head per layer, several GB in double precision. Running at the default size exhausts memory; the model-family fixtures pin InputSize = 64x64 to fit (#2154).

DINO's encoder layer says so in code: // Self-attention (simplified - would use deformable attention in full implementation).

What the papers do

• DINO (Zhang et al. 2022) and Deformable DETR (Zhu et al. 2021): multi-scale deformable attention (MSDeformAttn). Each query attends to a small fixed number of sampling points per head per level (K = 4, 4 levels), at learned offsets around a reference point, sampled bilinearly. Cost is linear in the token count.
• RT-DETR (Zhao et al. 2023): a hybrid encoder. AIFI runs self-attention on S5 only (400 tokens at 640), and CCFM fuses scales with convolutions. The decoder uses deformable cross-attention.

Work

• An MSDeformAttn operator built from engine ops so it is tape-visible: learned sampling offsets and attention weights, and bilinear sampling of each level at the offset points (grid-sample style, differentiable in the features and the offsets).
• DINO encoder and decoder cross-attention on MSDeformAttn.
• RT-DETR: AIFI on S5 plus the CCFM cross-scale fusion, and deformable decoder cross-attention.
• An equivalence test against a reference MSDeformAttn (the Deformable DETR PyTorch reference), and a tape-gradient check.
• Unpin the fixture input size for these two models once they fit at their defaults.

🤖 Generated with Claude Code

https://claude.ai/code/session_016jgqTmscEnkgmAp1TkNFpG