A closed-form, noise-aware controller that adaptively balances SFT and RL — no hand-tuned μ schedule required.
Yuelin Hu¹ · Wei Liu² · Zhenbo Yu¹,³ · Zhengxue Cheng¹ · Li Song¹ ¹Shanghai Jiao Tong University · ²Shanghai Maritime University · ³Novacore
- 🔥 News
- 💡 What is GAC?
- ✨ Key Highlights
- 📊 Results
- 🚀 Quick Start
- 🔍 Evaluation & Release
- 📐 Method
- 🗂️ Repository Layout
- 📝 Citation
- 🙏 Acknowledgements
- 📮 Contact
- [2026/09] 🤗 GAC-Qwen3.5-4B checkpoint released on Hugging Face with an evaluation snapshot and model card.
- [2026/09] 📊 Public checkpoint evaluator released with dataset provenance, fixed-subset validation, and isolated code scoring.
- [2026/08] 📄 Camera-ready released; the reference implementation was open-sourced under Apache-2.0.
- [2026/08] 🎉 GAC was accepted to the EMNLP 2026 Main Conference (Budapest).
Hybrid SFT–RL post-training is the standard recipe for aligning LLMs — but the mixing weight μ between the two signals is almost always a fixed, hand-tuned schedule. This is fragile: when RL noise spikes, or when the SFT expert starts pulling the policy in a wrong direction, a static μ over-commits to whichever side is currently less reliable.
GAC solves this by treating μ as an estimation problem.
We derive a closed-form optimal μ that minimizes the MSE of the composite gradient under a signal-vs-noise decomposition:
μ* = ( αtgt·Δg² + σr² ) / ( Δg² + σs² + σr² )
where σs², σr² are SFT and RL noise variances and Δg² is the SFT–RL disagreement. Since gradient-level quantities are prohibitively expensive at every step, we deploy three coefficient-space proxies estimated online from tensors any GRPO/PPO trainer already computes, wrapped in EMA smoothing, a cosine-schedule prior, and per-step change capping.
Paper result: +3.8 pp over HPT (the previous best hybrid post-training method) averaged over math, code, science, and logic benchmarks, with < 1% wall-time overhead, 28% lower KL-drift area, and gains that grow with model scale from 1.5B → 14B.
- 🎯 Closed-form μ*, derived from MSE-optimal gradient estimation — no meta-learning, no bilevel optimization, no extra hyperparameter grid.
- 📉 Coefficient-space proxies: three cheap signals (
σ_s²,σ_r²,Δg̃²) that any PPO/GRPO trainer already has on hand. - 🛡️ Guarded controller stack: EMA smoothing → cosine-prior blend → per-step change cap → hard clipping. Robust to mid-training regime shifts.
- ⚡ < 1% wall-time overhead: proxies are computed inside the existing forward pass, no extra network eval.
- 📈 Consistent gains across scales: +2.2 pp @ 1.5B, +3.8 pp @ 7B, +3.3 pp @ 14B on AMC vs. HPT (paper Table 3).
- 🔬 Length-invariant by construction: each sequence contributes exactly one scalar per proxy, so longer rollouts don't get extra vote weight.
To keep GAC aligned with progress in capable compact models, we retrained the method on Qwen3.5-4B and release the resulting checkpoint for research and evaluation. The table compares the Qwen3.5-4B base checkpoint with the GAC release under a fixed single-snapshot protocol.
| Domain | Benchmark | Eval size (N) | Qwen3.5-4B Base | GAC-Qwen3.5-4B |
|---|---|---|---|---|
| Mathematics | AMC | 83 | 19.3% | 67.5% |
| Mathematics | AIME24 | 30 | 0.0% | 26.7% |
| Mathematics | AIME25 | 30 | 3.3% | 20.0% |
| Knowledge | MMLU-Pro | 1,000 | 58.3% | 74.2% |
| Science | GPQA-Diamond | 198 | 29.3% | 64.1% |
| Science | SciBench | 692 | 11.4% | 60.1% |
| Code | MBPP | 500 | 67.6% | 74.6% |
| Code | HumanEval | 164 | 67.7% | 81.1% |
| Logic | BBH Logical Deduction | 750 | 86.3% | 93.1% |
| Logic | BBH Object Counting | 250 | 93.2% | 92.8% |
| Logic | BBH Tracking | 750 | 97.6% | 95.3% |
| Logic | BBH average (macro) | 3 slices / 1,750 | 92.4% | 93.7% |
The GAC release is higher on 9 of 11 task slices in this snapshot, with gains across mathematics, knowledge, science, and code. These are release metrics; use the public evaluator for exact per-example counts and reruns.
For the original paper's experiments and tables, see the OpenReview paper.
git clone https://github.com/deepnovacore/GAC.git
cd GAC
pip install -r requirements.txt
pip install -e .The controller has no framework-specific dependencies beyond PyTorch. torch.distributed is used only if you have multi-GPU.
Drop the controller into any hybrid SFT–RL loop:
from gac import AdaptiveMuController
from gac.sft_loss import SFTPhiLossFn
from gac.hybrid_loss import compute_hybrid_loss
from gac.controller import mu_schedule
controller = AdaptiveMuController(
ema_beta=0.99,
mu_change_cap=0.01,
trim_ratio_sft=0.10,
mu_update_freq=10,
alpha_static=0.5,
)
sft_loss_fn = SFTPhiLossFn(token_level=True)
def training_step(batch, global_step, rl_loss_fn):
# cosine-schedule prior on μ (peak → valley over training)
prior = mu_schedule(
global_step,
warmup_steps=200, decay_steps=800,
mu_peak=0.85, mu_valley=0.15,
)
# μ is inferred inside compute_hybrid_loss via the controller
loss, logs = compute_hybrid_loss(
logprob=batch["logprob"],
action_mask=batch["action_mask"],
advantages=batch["advantages"],
expert_mask=batch["expert_mask"],
rl_loss_fn=rl_loss_fn, # your favorite PPO / GRPO surrogate
sft_loss_fn=sft_loss_fn,
controller=controller,
global_step=global_step,
schedule_prior=prior,
blend_weight=0.5, # λ in the paper — cosine-prior blend
)
return loss, logsThat's it. GAC only touches the weighting between the SFT and RL losses; it does not modify either loss itself. You keep your PPO / GRPO implementation, your RL infrastructure, and your reward model unchanged.
The public eval/ harness covers 11 task slices across 9 dataset
families: AMC, AIME24, AIME25, MMLU-Pro, GPQA-Diamond, SciBench, MBPP,
HumanEval, and three BBH logic subsets. It includes dataset provenance,
fixed-subset validation, aggregation, and isolated code scoring.
cd eval
pip install -r requirements.txt
bash run_all.sh --model_path /path/to/your-checkpoint \
--output_dir ./results --tp_size 4MMLU-Pro uses a pinned revision and a checked-in 1,000-ID manifest. GPQA is a
gated dataset: authenticate with Hugging Face or provide a complete, lawfully
obtained CSV through --gpqa_csv. See eval/README.md for
benchmark commands, scoring details, Slurm usage, and the code-execution
safety boundary.
The GAC-Qwen3.5-4B checkpoint contains the compact-backbone release and its evaluation protocol. The original paper's experimental tables remain available in the OpenReview paper.
| Version | Target | Contents |
|---|---|---|
| ✅ v0.1.0 | 2026-08 | Reference AdaptiveMuController, hybrid-loss integration, unit tests, and evaluator covering 11 task slices |
| ✅ v0.1.1 | 2026-09 | Fixed-subset validation, isolated code scoring, provenance checks, and Qwen3.5-4B checkpoint release |
| 🚧 v0.2.0 | 2026-09 | Training pipeline (VeRL fork), public training recipe, and gac-core packaging |
| 🚧 v0.4.0 | 2026-11 | Docker image, 1-command run.sh reproduction, external verification runs |
The controller performs three operations per update:
1. Proxy estimation (Sec. 3.2 of the paper)
| Symbol | Meaning | Length-invariant? |
|---|---|---|
| σr² | Variance of sequence-level GRPO-normalized advantages across the batch. Post-normalization dispersion, not raw reward variance. | ✅ |
| σs² | Tail-trimmed variance of length-normalized per-sequence NLL on expert samples. | ✅ |
| Δg̃² | Mean squared coefficient mismatch between SFT and RL token-level gradient coefficients on shared response tokens, after within-batch z-normalization. | ✅ |
2. Closed-form aggregation into μ*:
μ* = ( αtgt·Δg̃² + σr² ) / ( Δg̃² + σs² + σr² )
3. Guarded update:
raw_μ* → EMA(β) → blend(λ) with cosine prior → clip |Δμ| ≤ c̄ → clip to [μ_min, μ_max]
Detailed derivation (including the MSE minimization, the proxy-vs-oracle validation, and length-invariance proofs) is in docs/method.md and Appendices A–D of the paper.
GAC/
├── gac/ # Reference implementation
│ ├── controller.py # AdaptiveMuController (paper Sec. 3)
│ ├── sft_loss.py # SFT loss + CHORD φ(p)=p(1-p) variant
│ ├── hybrid_loss.py # Reference hybrid loss (μ·L_SFT + (1-μ)·L_RL)
│ ├── utils.py # Length-invariant masked reductions
│ └── __init__.py
├── eval/ # Public checkpoint evaluation harness
│ ├── generate_vllm.py # Unified vLLM generator
│ ├── math_bench.py # AMC / AIME24 / AIME25 (math-verify scoring)
│ ├── knowledge_bench.py # MMLU-Pro / GPQA / SciBench
│ ├── code_bench.py # MBPP / HumanEval (bigcode-eval scoring)
│ ├── bbh_logic.py # BBH logical subsets
│ ├── aggregate.py # Multi-seed mean±std aggregation
│ ├── manifests/ # Pinned benchmark subset manifests
│ ├── run_all.sh # One-command reproduction
│ └── README.md # Detailed usage & data provenance
├── configs/
│ └── gac_default.yaml # Default hyperparameters (paper Sec. 4.1)
├── docs/
│ ├── method.md # Long-form derivation and proxy semantics
│ └── index.html # Project page (served by GitHub Pages)
├── tests/
│ ├── test_controller.py # Closed-form estimator unit tests
│ └── test_evaluation_data.py # Dataset and fixed-subset regression tests
├── assets/ # Figures used in this README
├── CITATION.cff
├── LICENSE # Apache 2.0
├── README.md
├── requirements.txt
└── setup.py
If you use GAC in your research, please cite:
@inproceedings{hu2026gac,
title = {GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training},
author = {Hu, Yuelin and Liu, Wei and Yu, Zhenbo and Cheng, Zhengxue and Song, Li},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year = {2026},
address = {Budapest, Hungary},
url = {https://openreview.net/forum?id=VhBpT4iq60},
}Machine-readable metadata is also provided in CITATION.cff.
We thank Novacore for their collaboration on this work. This work is a joint academic collaboration with Novacore; no commercial rights are asserted.
This research was supported in part by the NSFC (62431015, 62571317, 62501387), the Fundamental Research Funds for the Central Universities, Shanghai Key Laboratory of Digital Media Processing and Transmission under Grant 22DZ2229005, 111 project BP0719010, and the Ant Group Research Fund (academic grant; no IP claim).
The GAC design builds on ideas explored by prior hybrid SFT–RL literature — in particular the token-wise reweighting introduced by CHORD and the hybrid post-training framework of HPT. Our reference implementation integrates cleanly into GRPO / PPO trainers such as VeRL and Trinity-RFT. We thank the anonymous ARR reviewers and the EMNLP 2026 Area Chair for the detailed feedback that shaped the camera-ready version of the paper.
For questions, feedback, or collaboration opportunities:
- Yuelin Hu — huyuelin51717221@sjtu.edu.cn (first author, code maintainer)
- Zhenbo Yu — yuzhenbo@sjtu.edu.cn (co-corresponding, Novacore)
- Li Song — songli@sjtu.edu.cn (co-corresponding, SJTU)
Issues & PRs are very welcome. If you're integrating GAC into a specific trainer (TRL, OpenRLHF, VeRL, Trinity-RFT, custom) and hit rough edges, please open an issue with your integration snippet — it directly helps others.