Implementation of the CBAM-based algorithm for Source Device Identification (SDI) leveraging Grad-CAM explanations
-
Updated
Sep 27, 2025 - Python
Implementation of the CBAM-based algorithm for Source Device Identification (SDI) leveraging Grad-CAM explanations
Python implementation of the McNemar's mid-p statistical significance test.
LLM and agent evaluation as measurement: calibrate the judge, test agents through their tool calls, import existing test cases, find untested model calls, and gate CI only on significant regressions. CLI, Python library, npm package and GitHub Action.
Sequence-level (hard) knowledge distillation of reasoning traces into small LLMs. Paired MATH-500 evaluation: the teacher's style decides the damage, not the student's size — the same 1.7B loses 3.4 points on a weak teacher's traces and 30.8 on a strong one's.
Is your AI system really better, or is the eval difference just luck? Paired confidence intervals, permutation tests and minimum detectable effect for eval results. CLI, Python API and GitHub Action.
Find the items a new version of a system broke that the old version got right: flip table, negative flip rate, exact McNemar and bootstrap intervals.
Empirical study of what determines IMDB test accuracy under frozen GloVe + shallow neural head or linear baseline: architecture (SNN/CNN/LSTM), sequence length, padding masking, EarlyStopping, and threshold tuning. A TF-IDF + logistic regression baseline beats every neural condition. Reanalysis of a 2022 project.
Paired non-inferiority verdict: can model A replace model B on the same cases? (Newcombe 1998, McNemar)
Is your fine-tuned LLM really better? Paired statistics, regression slices, trap-question and judge-calibrated evaluation with Markdown/HTML reports
把「点估计」升级成「带不确定性的结论」:LLM 应用评估的统计层、指标层、评估集设计与报告
LoRA 微调训练计划 + 自动评测工具链:置信区间 / 配对检验 / 多重比较校正,13/16 个模块零第三方依赖
v5: Five-cell parallel constraint-reasoning experiment on Claude Haiku 4.5. Phase D closed with a 4-tier representational hierarchy as the headline finding (4/5 cells significant past Bonferroni). See STATUS.md / MEMO.md.
Decide in hours whether to migrate/deprecate an LLM: golden set, old-vs-new comparison, go/no-go, rollback (synthetic data)
Eval harness that knows when a difference is noise
Reliability metrics and statistically sound CI gates for AI agents evaluated with repeated runs (pass^k, McNemar, Wilson, SPRT) - pure Python, zero dependencies
McNemar's paired test for NLP model comparison. Exact binomial fallback when b+c < 25. Single-file, runs in browser.
W8A8 vs BF16 accuracy evaluation on Ascend NPU: official sampling protocol, paired per-item significance testing, and a PPL sentinel.
Paired LLM evals graded by execution: exact McNemar, Bradley-Terry rank bands, Wilson intervals.
Reproducible LLM evaluation, capability tracking and regression detection. Tells you whether a model actually got worse - with paired statistical tests, not vibes.
To associate your repository with the mcnemar topic, visit your repo's landing page and select "manage topics."