Summary
Ten of the fourteen detection/OCR models still contain forward-pass helpers written as scalar NumOps loops. Each one allocates a fresh tensor and copies elements through NumOps.ToDouble / NumOps.FromDouble, which severs the autodiff tape: any trainable layer upstream of such a call receives no gradient and never trains, silently.
This is the residual half of the work started in the detection/OCR test-family PR. That PR converted the elementwise helpers (ApplyReLU ×10, ApplySigmoid ×4, ApplySiLU/ApplySwish ×5) and NeckBase.Conv1x1/Add to engine ops, which restored the gradient path through every FPN / PANet / BiFPN — previously the backbone of every detector using a neck received nothing at all. The remaining helpers manipulate shape, not just arithmetic, so they are a real rewrite rather than a one-line swap and were deliberately deferred.
Current state, per model
Measured by scanning each model file for methods that return Tensor<T>, take a Tensor<T>, and contain a for loop using NumOps.ToDouble/FromDouble:
| model |
remaining tape-severing transforms |
| YOLOv8 |
— none |
| YOLOv9 |
— none |
| YOLOv10 |
— none |
| FasterRCNN |
— none |
| YOLOv11 |
MaxPool |
| DETR |
ApplyFFN |
| DINO |
ApplyFFN |
| RTDETR |
ApplyFFN |
| CascadeRCNN |
RefineBoxes |
| CRAFT |
BilinearUpsample |
| DBNet |
ApplyDifferentiableBinarization, BilinearUpsample |
| EAST |
ApplyBatchNormReLU, BilinearUpsample |
| CRNN |
ApplyBatchNorm, ConvertToGrayscale, MaxPool2D |
| TrOCR |
AddPositionalEncoding, ApplyFFN, ComputeCausalAttention, ComputeCrossAttention, ComputeMultiHeadAttention |
13 distinct helpers. Note CRNN.ConvertToGrayscale runs before any trainable layer, so it severs nothing that matters — it is listed for completeness, not as work.
ApplyFFN appears in four models (DETR, DINO, RTDETR, TrOCR) and is the highest-leverage single fix: it sits between every transformer block, so in those models the gradient currently reaches only the last projection.
Why it matters
Train on these bases is now a real tape-recorded step (it used to be an empty method body — see the linked PR), so the models do learn. But they learn only in the layers downstream of the last severing call. A user fine-tuning DETR today updates its output head and nothing else, and gets no error or warning saying so.
The family conformance fixtures assert Train_ShouldChangeParameters, which passes as soon as any parameter moves. That is deliberately the weakest honest claim: it catches a no-op Train, and it must not be mistaken for evidence that a model trains end to end.
Proposed work
Convert each helper to engine ops, pairing every conversion with a numerical-equivalence test that pins current output against the rewritten one before the swap. The arithmetic must not change; only tape visibility.
Rough order by leverage:
ApplyFFN — 4 models, sits between every transformer blockBilinearUpsample — 3 models, the entire text-detection decoder pathMaxPool / MaxPool2D — YOLOv11, CRNNApplyBatchNorm / ApplyBatchNormReLU — CRNN, EAST- TrOCR attention helpers (
ComputeMultiHeadAttention, ComputeCausalAttention, ComputeCrossAttention)
ApplyDifferentiableBinarization (DBNet), RefineBoxes (CascadeRCNN), AddPositionalEncoding (TrOCR)
Acceptance
- No model file in
src/ComputerVision/Detection/ or src/ComputerVision/OCR/ contains a Tensor<T>-in/Tensor<T>-out helper with a scalar NumOps conversion loop.
- Each conversion has an equivalence test comparing pre- and post-rewrite output.
- The family bases can then be strengthened from
Train_ShouldChangeParameters to asserting that loss actually decreases, and that mAP / H-mean / CER improve after training — using the metrics added alongside the family work (ObjectDetectionMetrics, TextDetectionMetrics, TextRecognitionMetrics).
Summary
Ten of the fourteen detection/OCR models still contain forward-pass helpers written as scalar
NumOpsloops. Each one allocates a fresh tensor and copies elements throughNumOps.ToDouble/NumOps.FromDouble, which severs the autodiff tape: any trainable layer upstream of such a call receives no gradient and never trains, silently.This is the residual half of the work started in the detection/OCR test-family PR. That PR converted the elementwise helpers (
ApplyReLU×10,ApplySigmoid×4,ApplySiLU/ApplySwish×5) andNeckBase.Conv1x1/Addto engine ops, which restored the gradient path through every FPN / PANet / BiFPN — previously the backbone of every detector using a neck received nothing at all. The remaining helpers manipulate shape, not just arithmetic, so they are a real rewrite rather than a one-line swap and were deliberately deferred.Current state, per model
Measured by scanning each model file for methods that return
Tensor<T>, take aTensor<T>, and contain aforloop usingNumOps.ToDouble/FromDouble:MaxPoolApplyFFNApplyFFNApplyFFNRefineBoxesBilinearUpsampleApplyDifferentiableBinarization,BilinearUpsampleApplyBatchNormReLU,BilinearUpsampleApplyBatchNorm,ConvertToGrayscale,MaxPool2DAddPositionalEncoding,ApplyFFN,ComputeCausalAttention,ComputeCrossAttention,ComputeMultiHeadAttention13 distinct helpers. Note
CRNN.ConvertToGrayscaleruns before any trainable layer, so it severs nothing that matters — it is listed for completeness, not as work.ApplyFFNappears in four models (DETR, DINO, RTDETR, TrOCR) and is the highest-leverage single fix: it sits between every transformer block, so in those models the gradient currently reaches only the last projection.Why it matters
Trainon these bases is now a real tape-recorded step (it used to be an empty method body — see the linked PR), so the models do learn. But they learn only in the layers downstream of the last severing call. A user fine-tuning DETR today updates its output head and nothing else, and gets no error or warning saying so.The family conformance fixtures assert
Train_ShouldChangeParameters, which passes as soon as any parameter moves. That is deliberately the weakest honest claim: it catches a no-opTrain, and it must not be mistaken for evidence that a model trains end to end.Proposed work
Convert each helper to engine ops, pairing every conversion with a numerical-equivalence test that pins current output against the rewritten one before the swap. The arithmetic must not change; only tape visibility.
Rough order by leverage:
ApplyFFN— 4 models, sits between every transformer blockBilinearUpsample— 3 models, the entire text-detection decoder pathMaxPool/MaxPool2D— YOLOv11, CRNNApplyBatchNorm/ApplyBatchNormReLU— CRNN, EASTComputeMultiHeadAttention,ComputeCausalAttention,ComputeCrossAttention)ApplyDifferentiableBinarization(DBNet),RefineBoxes(CascadeRCNN),AddPositionalEncoding(TrOCR)Acceptance
src/ComputerVision/Detection/orsrc/ComputerVision/OCR/contains aTensor<T>-in/Tensor<T>-out helper with a scalarNumOpsconversion loop.Train_ShouldChangeParametersto asserting that loss actually decreases, and that mAP / H-mean / CER improve after training — using the metrics added alongside the family work (ObjectDetectionMetrics,TextDetectionMetrics,TextRecognitionMetrics).