Evidence-first SFT and DPO alignment evaluation for customer-service language models.
Live alignment workbench | Repository | Evidence card
This project began as a Qwen3-0.6B LoRA SFT plus preference-training exercise on a 4GB laptop GPU. The important result was not a convenient DPO win. A data audit showed that the historical random development splits were strongly coupled to training: 384 of 600 SFT development answers appeared in training records, and 191 of 200 preference-development chosen answers appeared during SFT training.
The response was to separate evidence levels, freeze a 36-case post-training holdout, keep Base/SFT/DPO outputs paired by case, report uncertainty and multiple-testing correction, and leave human preference claims disabled until two independent reviewers actually submit labels.
| Evidence | Result | Boundary |
|---|---|---|
| Training pipeline | 5,800 SFT rows; 2,000 DPO pairs | Raw rows are not published |
| LoRA rank sweep | rank 4 / 16 / 64 | Same-source development loss only |
| Independent automatic holdout | 36 cases x 3 models = 108 outputs | Deterministic rubric, not human preference |
| SFT - Base score | +8.39; 95% CI [+2.08, +14.78] | Post-hoc; Holm p=0.102 |
| DPO - SFT score | -1.58; 95% CI [-6.58, +3.89] | Inconclusive; no DPO advantage |
| Human blind review | 0 valid decisions; 72 ratings missing | INCOMPLETE |
- Base:
Qwen/Qwen3-0.6Bat pinned revisionc1899de.... - Data provenance: 6,000 candidates came from a revision-pinned JDDC transform; the other 2,500 FAQ-style and 3,500 SOP-style candidates were hand-authored template expansions. They are not described as scraped public FAQ records because no item-level public-URL provenance exists.
- SFT: TRL SFTTrainer, LoRA
q_proj/v_proj, ranks 4/16/64, 300 steps. - Historical preference training: rank-16 SFT policy initialization, beta 0.05/0.1/0.3, 200 steps. The locked runtime was TRL 0.13.0 / Transformers 4.57.6 / PEFT 0.7.1; with
ref_model=None, TRL 0.13.0 useddisable_adapter()for reference log-probabilities, so the effective reference was Base rather than separately frozen SFT. - Noise: chosen/rejected flips while preserving sample count. Full-pool nominal 10%/30% files became effective 11.25%/31.25% in the first 800 pairs actually consumed by training.
- Holdout: 36 handwritten cases across six scenarios, authored after existing adapters were frozen; exact and character-trigram audits found no recorded training overlap at threshold 0.8.
The public repository contains dependency-light implementations of contamination auditing, paired bootstrap, sign-flip permutation, exact McNemar, Holm adjustment, and human-review completion gates. It also contains sanitized configurations, aggregate evidence, and synthetic examples.
It excludes raw conversations, derived training rows, holdout text, raw model completions, reviewer packets, blind keys, model weights, adapters, and machine-local paths.
Python 3.10-3.12 is supported. No model or dataset download is required.
python -m venv .venv
# Windows: .venv\Scripts\python -m pip install .
# Linux/macOS: .venv/bin/python -m pip install .
python -m unittest discover -s tests -p "test_*.py"
python scripts/run_synthetic_demo.py
python scripts/verify_public_release.pyBuild wheel and source distribution:
python -m pip install build
python -m buildThe synthetic demo proves evaluation-code behaviour only. It does not recreate the private model runs or the frozen aggregate values in evidence/public_metrics.json.
Rank-64 reached the lowest development loss but used about four times the trainable parameters of rank-16. Rank-16 remained the controlled DPO baseline, not a claim of globally optimal rank.
All three beta runs achieved reward accuracy 1.0 on the same-source development split. Reward margin also scales with beta, and the historical effective reference was Base rather than frozen SFT, so these values cannot be read as directly comparable business quality or a standard SFT-reference DPO result.
The automatic holdout scored Base/SFT/preference-model at 54.92/63.31/61.72. Paired post-hoc analysis gave SFT - Base +8.39, but the recorded continuous-family Holm p-value was 0.102. Preference-model - SFT was -1.58 with an interval crossing zero. Deltas use unrounded per-case values, not subtraction of displayed two-decimal means. Bootstrap intervals are unadjusted; p-values are Holm-adjusted by outcome family. The correct conclusion is that SFT had a positive point estimate under a small deterministic rubric, while the earlier preference-development advantage did not transfer.
Frozen generation used deterministic decoding and max_new_tokens=64. The public snapshot records the setting but does not claim to rerun the private model benchmark in CI.
Built a Qwen3-0.6B LoRA SFT and preference-training pipeline with rank, beta, and effective label-noise ablations; audited same-source development leakage and the historical Base-reference training contract, then froze a balanced 36-case holdout and evaluated 108 paired Base/SFT/preference-model outputs with bootstrap, sign-flip, McNemar, and Holm correction. SFT had a +8.39 automatic-rubric point estimate over Base, but Holm p=0.102; the preference model did not improve over SFT, so human preference and production-quality claims remain gated.
Do not claim standard frozen-SFT-reference DPO, a DPO human win rate, statistically significant gain, online resolution/CSAT improvement, production SLA, SOTA result, or that the historical 13/30 development top-count represents user preference. Its tie-aware recount is 5/9/7 with 9 ties, not 10/13/7 decisive wins.
MIT for original source in this sanitized snapshot. Third-party data and model rights are not redistributed; see THIRD_PARTY_NOTICES.md.