Repository navigation
Measure how much machine-written text the score separates, on RAID - #104
Merged
Merged
Conversation
The calibration measures how often the tool is wrong about a human and says outright that it leaves out the other half: how much AI writing it catches. RAID measures that half. These are the predictions, committed before any RAID text is scored so the order can be checked in the history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The calibration measures the half that needs only human writing. This adds the tooling for the other half: `signsofai-calibrate raid` scores JSON lines with the shipped engine and evaluates nothing. RAID's own run_evaluation picks every threshold and counts every hit, so the scorer cannot shape the number. tools/raid/ draws a seeded, fingerprinted sample of RAID-train, evaluates it, applies the eight RAID attacks that need no language model (strengths read from RAID's attack.py, not copied), and writes Docs/DETECTION.md from the resulting JSON. Parallel and sequential scoring were checked byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At a 5% false-positive rate on RAID's human texts the score catches 26.6% of machine-written text: 37.5% from chat-tuned generators, 12.6% from base models, whose AUROC sits near 0.5. Five of the six pre-registered predictions held as written; P4 failed (machine verse is caught well, through sentence rhythm) and P6 held by its letter only. Found along the way, and labelled as such: no machine text in RAID reaches the 649-word floor, so the product as shipped would have answered none of it; and 9 of 110 long news articles cross the 30/100 boundary, a register the calibration corpus never contained. The published rate is not changed. Nothing is submitted to the RAID leaderboard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Eleven live surfaces said, in one wording or another, that how much machine writing the tool catches was never measured. As of DETECTION.md that is false. Each now says what was measured and where, without typing a figure that could drift: the README, SKILL.md, both languages of /why, the evidence report in English and Spanish (re-pinned to its new English source), the CALIBRATION and PARAPHRASE pages through their generators, the corpus README, and the comments and test message that restated the old claim. CALIBRATION.md and PARAPHRASE.md were edited to match their generators and checked against a regenerated copy: only the changed sentences match, and the engine and run lines were left alone because neither page was re-run. Re-running the calibration with 0.8.1 moved no number. Published blog articles are dated and were not touched. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the committee (Codex, which could read but not execute): the mean machine score drops by 0.53 points under the attack, not "less than half a point", and lower-case "moreover" is restored (four Latin letters, four lookalikes, a tie the scanner accepts); the capitalised "Moreover" is not. Checking the second point found a third case the page now names: a one-letter word such as "a" has no Latin letter left and is never restored. The page no longer proposes a specific fix. Closing these is a change of behaviour and needs its own measurement against human text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot reproduced every headline number independently (26.55%, chat 37.5% vs base 12.6%, AUROC 0.629) and a full sequential rescore was byte-identical. What it found was in the presentation: - The domain table printed RAID's cutoffs rounded to one decimal, the same step as the engine's scores, which carries whole ties across the cutoff: at the printed 40.5, news would flag 5.11% of humans. The table now gives the lowest score each cutoff counts, using RAID's own comparison. - RAID-train repeats some human texts verbatim (13,272 distinct in 13,371 rows). The headline still counts rows, as RAID does; the long-text check, which is ours, now counts each text once: 14 of 705, news 9 of 108. - The repetition-penalty sentence compared pools with different generators and named a mechanism nobody measured. It now compares only the six generators run both ways, and says it is an observed difference. - attack.py and evaluate.py now refuse incomplete score files instead of counting a missing score as a miss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fable read the page as a teacher with no statistics and found that it would be quoted as "it catches a quarter of AI text, 60% of ChatGPT", which is not what the product does: on this benchmark the shipped product gives its verdict on no machine text and on 14 human texts, and at the lengths where it speaks, detection was never measured. Every number on the page was right; the page did not say that in one place. - A "read this first" block for teachers, and the headline now gives both halves of the as-shipped result and the news finding. - News: 9 of 108 long articles cross 30/100 (4.4-15.1%), an interval wholly above the published one, so by the rule that set the boundary 30/100 is not supported for news. Said plainly, with the post-hoc caveat, and now also on CALIBRATION.md (through its generator), on both languages of /why and in the README, without typed figures. - New data instead of asserted causes: the human rate at 30/100 by length (14.5% under 200 words to 2.0% at 649+), stat.burstiness on 74% of long news against 34% of all long human text and 32% of the calibration, the rate re-weighted to RAID's own generator mix (25.3%), and the predictions re-judged at the single cutoff (P3 holds there by 0.7 points). - P6 is "held, weakly", not a third category; P5 carries its interval, which touches 1%. Both are judged from attacks.json, not in the generator. - Causal claims without a measurement were cut or marked as observations; the leaderboard block no longer singles out vendors, says which files were read and that FastDetectGPT was skipped, and no longer attributes behaviour to detectors this project did not run. - The evidence report links a real URL, and the corpus README says RAID answers one of its two objections, not both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fable re-read the page from scratch. The first misreading was gone; the one left was "a long essay over 30 is probably AI, since it's only wrong 0.7% of the time", which the page set up and never closed. On this benchmark every long text that crossed 30 was written by a person, and the page now says so where the teacher reads. - The teacher block says what the calibration corpus holds, that ordinary school essays are outside it, puts 9 of 108 against the promise (about one in twelve against one in twenty), counts people instead of fractions of a student, and says what the tool is still for. - The re-run of the calibration with 0.8.1 is now on file (Docs/Detection/calibration-rerun.json: only engine and date moved), and the page prints the sentence from it instead of asserting it. - Scanner misses are listed by class with a reason computed from the text (3 machine, 2 human; 3 in another script, 2 with no Latin letter left). One "degenerate output" was a human Reddit post of nothing but A's. - The news finding is in the past tense, as one sample, with its cause unmeasured, on every surface that carries it; "low" is now "too low to stand out from human writing" in the report, /why and SKILL.md. - One name for the other leaderboard entries, the review claim reduced to what the repository shows, and the AUROC sentence limited to the data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README now includes the calibration re-run that calibration-rerun.json comes from, and the two arguments attack.py evaluate gained for explaining scanner misses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The page keeps the measured effect (vowel-heavy words under full substitution are neither reported nor restored) and drops the detail that belongs with a fix not yet made. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
peopleworks
marked this pull request as ready for review
November 18, 2026 05:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs/CALIBRATION.mdmeasures how often this tool is wrong about a person and says what it leaves out: nothing about how much AI writing it catches, because a tool that flags nothing has a perfect false-positive rate. This measures that other half, on RAID (Dugan et al., ACL 2024), a public benchmark nobody here assembled. Draft until the maintainer decides; nothing has been submitted to the RAID leaderboard.The order, checkable in the history
e007adc— the predictions, committed and pushed before any RAID text was scored (Docs/Detection/prediction.md), with a note added later the same day, still before measuring, saying P6 was probably wrong and why.c2925e2—signsofai-calibrate raidandtools/raid/. The engine only produces scores; RAID's ownrun_evaluationpicks every cutoff and counts every hit.986319f— the measurement andDocs/DETECTION.md, generated fromDocs/Detection/*.json.ab41f6a— eleven live surfaces that said detection was never measured.74433f8to the last commit: the committee's corrections, each commit saying which finding it answers.The numbers
Predictions: P1, P2, P3, P5 and the overall range held; P4 failed (machine verse is caught well, through sentence rhythm); P6 held by its letter only (the analyzer restores disguised letters before reading, as the later note expected).
Found along the way, and labelled as such on the page
stat.burstinessfires on 74% of them against 32% of the calibration corpus). Overall 14 of 705 long human texts. The published false-positive rate is not changed; that goes through the calibration corpus and the committee. It is now stated, without typed figures, onCALIBRATION.md,/whyand the README.rhet.em-dash,lex.actuallyandlex.literallyfire more on RAID's humans than on its 2023 chat models. No rule was changed.Behaviour this changes
Only text. The evidence report's limitation paragraph (English and Spanish, re-pinned to its new English
sourceHash),/whyin both languages,README.md,SKILL.md, the corpus README, and the generated sentences inCALIBRATION.mdandPARAPHRASE.md. Both generated pages were checked against a regenerated copy: only the changed sentences match, and their engine and run lines were left alone because neither was re-run. Re-running the calibration with 0.8.1 moved no number. Published blog articles are dated and were not touched.Committee
Three lenses, three disjoint sets of findings, all verified before acting.
helper_sandbox_lock_failed, and it exited 0 without a verdict. Its log still held two correct findings (the homoglyph drop is 0.53 points, and lower-case moreover is restored)./why, the README and the calibration page. Those surfaces now say it in the past tense, as one sample, with the cause unmeasured, but without typed figures, because figures copied onto surfaces are how this project's self-description has gone stale before. The figures live onDETECTION.md, generated from data.Not in this PR, on purpose
results.jsonthere is public for good; that is the maintainer's call.signsofai-verdict-story). It was true when published; it is left as it was.Verification
dotnet test SignsOfAI.slnx: 393 + 72 + 7 pass;SignsOfAI.Desktop.slnx: 31 pass.Docs/Detection/prediction.mdis unchanged sincee007adc(git diff e007adc HEADon it is empty).--threads 1(2,000 texts here; the full 30,971 and the 16,000 attacked texts by the committee).tools/raid/evaluate.py.🤖 Generated with Claude Code