Sequence-level (hard) knowledge distillation of reasoning traces into small LLMs. Paired MATH-500 evaluation: the teacher's style decides the damage, not the student's size — the same 1.7B loses 3.4 points on a weak teacher's traces and 30.8 on a strong one's.
nlp benchmark research math evaluation transformers knowledge-distillation reasoning mcnemar fine-tuning model-distillation empirical-study llm qwen sequence-level-kd
-
Updated
Oct 12, 2026 - Python