diff --git a/Docs/CALIBRATION.md b/Docs/CALIBRATION.md
index 868297a..77bc445 100644
--- a/Docs/CALIBRATION.md
+++ b/Docs/CALIBRATION.md
@@ -89,9 +89,9 @@ Across these groups the median score runs from 9.5 (**en-second-language-learner
## What the product does with this number
-The tool speaks at **30/100** and nowhere else, taking the boundary from the table above rather than from anybody's judgement. Below it a document gets its score and the reason it gets nothing more: a low score is not evidence that a person wrote something, since a detector that detects nothing also returns a low score, and this project has deliberately never measured how much machine writing it catches. The boundary moves when this page moves — including upward if a larger corpus turns out to be less flattering.
+The tool speaks at **30/100** and nowhere else, taking the boundary from the table above rather than from anybody's judgement. Below it a document gets its score and the reason it gets nothing more: a low score is not evidence that a person wrote something, since a detector that detects nothing also returns a low score, and on the public benchmark in DETECTION.md most machine-written text scores low as well. The boundary moves when this page moves — including upward if a larger corpus turns out to be less flattering.
-Above it there is **one** verdict, not a scale of them. This corpus can place a boundary and can say nothing whatever about how much further past it a score has travelled: no text known to be human came close to the upper reaches, and grading "moderate" against "strong" would need machine-written text, which the opening of this page argues against collecting. Interfaces do shade a high score more urgently than a low one, and those shades are a display convention — they are not on this page because nothing measured them.
+Above it there is **one** verdict, not a scale of them. This corpus can place a boundary and can say nothing whatever about how much further past it a score has travelled: no text known to be human came close to the upper reaches, and grading "moderate" against "strong" would need machine-written text, which the opening of this page argues against collecting; the benchmark in DETECTION.md was not used to set any band. Interfaces do shade a high score more urgently than a low one, and those shades are a display convention — they are not on this page because nothing measured them.
A language absent from the corpus entirely gets no verdict at all, whatever it scores. A language present but too thin to bound its own rate borrows this boundary and carries its own figure beside it, so the reader weighs the rate that was measured for the writing in front of them rather than the pooled one.
@@ -131,8 +131,8 @@ A rule near the top is not automatically wrong. Some tells genuinely appear in h
## What this does not tell you
-- **Nothing about how much AI writing it catches.** That is the other half of the picture and it is not measured here, deliberately. A tool that flags nothing has a perfect false-positive rate.
-- **Nothing about text unlike this corpus.** These are published articles and the essays of adult learners in a university English programme. A first-year essay by a native speaker is a different population again, and the rate on one does not transfer to the other. Calibrating on your own students' pre-2022 work is the fix, and the same tool does it.
+- **Nothing about how much AI writing it catches.** That is the other half of the picture and it is not measured here, deliberately. A tool that flags nothing has a perfect false-positive rate. That half is measured separately, on a public benchmark nobody here assembled, in [DETECTION.md](DETECTION.md).
+- **Nothing about text unlike this corpus.** These are published articles and the essays of adult learners in a university English programme. A first-year essay by a native speaker is a different population again, and the rate on one does not transfer to the other. Calibrating on your own students' pre-2022 work is the fix, and the same tool does it. The first outside check, on long human writing from a public benchmark, is in [DETECTION.md](DETECTION.md): in that one sample the boundary did not hold for the news articles, and why was not measured.
- **The affiliation groups are a proxy, not a fact.** Nobody's first language is recorded in a DOI. The learner group is the exception — its corpus records each writer's first language — which is why it exists. Elsewhere the manifest states the reasoning per text so it can be argued with; where it is wrong, the number moves.
- **Hashes prove what *this* run measured**, not that another person extracting the same articles would get identical text. They would not: PDF and HTML extraction differ. Reproducing this needs the extracted texts, not just the manifest.
diff --git a/Docs/Calibration/README.md b/Docs/Calibration/README.md
index 7ca1325..83fed9a 100644
--- a/Docs/Calibration/README.md
+++ b/Docs/Calibration/README.md
@@ -18,6 +18,12 @@ A **false-positive rate** needs only writing known to be human. It does not go s
the harm this category actually causes: detectors flag 61% of essays by non-native English speakers,
and not one of them publishes that figure about itself.
+The other half, how much machine-written text the score catches, is measured in `Docs/DETECTION.md`
+on RAID, a public benchmark nobody here assembled. It answers one of the two objections above: nobody
+here assembled it. It does not answer the other: its models are from 2023, and the number ages. It
+reports a detection rate beside this false-positive rate rather than an accuracy figure that blends
+the two.
+
## What makes a text admissible
**It was written before 2022.** That is the whole test. A paper with a 2019 DOI predates the models
diff --git a/Docs/DETECTION.md b/Docs/DETECTION.md
new file mode 100644
index 0000000..c44a1be
--- /dev/null
+++ b/Docs/DETECTION.md
@@ -0,0 +1,219 @@
+# How well the score tells machine-written text from human writing
+
+
+
+[CALIBRATION.md](CALIBRATION.md) measures how often this tool is wrong about a person, and says plainly what it leaves out: nothing about how much AI writing it catches, because a tool that flags nothing has a perfect false-positive rate. This page measures that other half.
+
+The calibration page also says why it avoids an accuracy figure: any collection of machine-written text ages, and flatters whoever assembled it. So this page uses a collection nobody here assembled, **RAID** (Dugan et al., ACL 2024, [arXiv 2405.07940](https://arxiv.org/abs/2405.07940)), which its authors describe as the largest benchmark for detectors of machine-generated text, and it states the date. RAID's generators are from 2023. The tells this tool looks for shift with every model, so this half of the picture ages and the human half does not.
+
+## If you are a teacher, read this first
+
+- **A low score tells you nothing.** In this benchmark about three in four machine-written texts (73.4%) scored too low to stand out from human writing.
+- **None of the detection rates further down is what the product does.** They are counted at cutoffs the benchmark chose, on texts mostly under 400 words. The product gives no verdict under 649 words, so on this benchmark it would have given its verdict on no machine-written text at all, and on 14 texts written by people.
+- **For texts of 649 words or more, where the product does give a verdict, how much machine writing it finds has not been measured.** No machine text in this sample is that long. So for a long text that crosses the boundary, this page cannot tell you how likely it is to be machine-written. On this benchmark every long text that crossed was written by a person.
+- **The boundary was checked on long human writing it had never been measured on, and it did not hold for news articles in this sample**: 9 of 108 crossed it, about one in twelve, where the promise is at most one in twenty. The calibration corpus holds research articles, encyclopedia entries and essays by adult learners of English at one university. Anything else is unmeasured, and that includes ordinary secondary-school essays. Until a kind of writing is measured, read a high score on it as saying nothing.
+- **A score above a cutoff is not a catch.** At the benchmark's cutoffs (not the product's), in a class of 30 where 3 texts were machine-written, you would expect fewer than one of the 3 above the cutoff, and one or two of the 27 honest ones. That is arithmetic from 26.6% and 5%, not a classroom measurement.
+- **The machine text here is from 2023.** "ChatGPT" on this page is the 2023 model. Nothing here describes what a student can use today.
+- **What the tool does show, at any length, is which tells a text carries and where.** That is a starting point for a conversation with the writer, not a finding about them.
+
+Words used below. *Chat-tuned* means trained to answer in conversation, like ChatGPT; a *base* model only continues a text. *AUROC* runs from 0.5 (the score cannot tell the two apart) to 1 (it always can). *Above the cutoff* means only that a text scored above it; it does not mean anyone was identified.
+
+## What was measured
+
+- **Benchmark** RAID-train without attacks (`train_none.csv`, SHA-256 `c5467bca6fc7f5c7`), evaluated with raid-bench 0.2.0
+- **Sample** 30,971 texts: all 13,371 human texts, and 200 machine texts for each of 11 generators in each of 8 domains (17,600); seed 20261117, id fingerprint `136c179ba9288e53`
+- **Engine** SignsOfAI.Core 0.8.1 at `7043e66`, rule packs unchanged, language set to English
+- **Run** 2026-11-17
+- **Counted by** RAID's own `run_evaluation`, at a 5% and a 1% false-positive rate
+
+**How RAID counts.** It takes the raw score and finds, separately for each domain, the cutoff that flags 5% of its human texts. Then it counts how much machine text lands above that cutoff. The score is this tool's 0–100 divided by 100, because RAID's search assumes a 0–1 range; dividing by a constant changes no ranking. So the product's own boundary and its length floor play no part here, and the cutoffs RAID finds are not this tool's.
+
+## The headline
+
+- **26.6%** of the machine-written texts score above RAID's cutoffs at a 5% false-positive rate (12.9% at 1%; AUROC 0.63). Weighted to RAID's own mix of generators instead of this sample's equal one: 25.3%.
+- **Chat-tuned generators 37.5%, base models 12.6%.** ChatGPT alone 59.6%. The four base models sit between 10.1% and 15.8%, two to three times the 5% a score that knew nothing would reach, while their AUROC is near 0.5: more of their text than of human text scores high, and across the rest of the range that advantage is cancelled. Where was not measured.
+- **As shipped, the product would have given its verdict on none of the machine-written texts in this sample, and on 14 texts written by people.** No machine-written text in the sample reaches 649 words, the shortest text the published false-positive rate was measured on. Below that length the product shows the score and the tells and withholds its verdict (#59).
+- **Found, not predicted:** on long human writing from registers the calibration never held, 14 of 705 texts cross the product's boundary, 9 of them news articles. Details under *Long human writing* below.
+
+This is below the entries submitted by others at the top of RAID's leaderboard and below each of the open detectors in the scale table further down, with the caveats given there. On this benchmark the tells are mostly fingerprints of chat-tuning. They are evidence about how a text reads, not a way to tell who wrote it.
+
+## What we predicted, and what happened
+
+Written and committed before any RAID text was scored ([prediction.md](Detection/prediction.md), commit `e007adc`). Each line is judged by the refutation criterion it was written with.
+
+| # | Prediction | Measured | Result |
+|---|---|---|---|
+| P1 | Chat-tuned rate at least twice the base-model rate | chat 37.5%, base 12.6%, ratio 2.97 | held |
+| P2 | Base-model rate under 20% | 12.6% | held |
+| P3 | text-davinci-003 between the two groups | 16.6% | held |
+| P4 | Poetry and recipes not among the three best domains | best three: film reviews 50.9%, poetry 36.7%, book summaries 33.7% | **failed** |
+| P5 | Scanner reports at least 99% of homoglyph and zero-width attacks, with false alarms on at most 1% of human texts | homoglyph 99.7%, zero-width 100.0%; any artifact on 0.84% of unattacked human texts (0.70–1.01%) | held on the point estimate; the interval touches 1% |
+| P6 | The homoglyph attack lowers the score | mean 20.6 → 20.1; above the cutoff 24.6% → 23.7% | held, weakly (−0.53 points) |
+| — | Between 15% and 40% overall, without attacks | 26.6% | held |
+
+**P4 failed.** One observation that may explain it, found afterwards: `stat.burstiness` fires on 52% of machine poems and 25% of human ones. Recipes were as weak as predicted (12.1%).
+
+**P6 held, weakly.** The score dropped by 0.53 points on average across the machine texts, through the mechanism P6 named (see *Attacks*). The note added the same day predicted no drop; it was right about the zero-width attack and wrong about homoglyphs. Between them the prediction and the note covered both outcomes, so this line was not a test that could fail.
+
+At the single cutoff every line holds or fails as above (P4's best three become arXiv abstracts, poetry and news articles): chat-tuned 34.3%, base 10.2%, text-davinci-003 10.9%, so P3 holds there by 0.7 points.
+
+## By generator
+
+| Generator | Kind | Above the cutoff at 5% | At 1% | AUROC |
+|---|---|---|---|---|
+| ChatGPT | chat-tuned | 59.6% | 37.4% | 0.82 |
+| GPT-4 | chat-tuned | 43.5% | 24.4% | 0.78 |
+| Llama 2 70B Chat | chat-tuned | 36.2% | 16.7% | 0.73 |
+| Mistral 7B Chat | chat-tuned | 35.6% | 14.9% | 0.73 |
+| MPT-30B Chat | chat-tuned | 28.6% | 12.1% | 0.66 |
+| Cohere Chat | chat-tuned | 21.5% | 7.7% | 0.63 |
+| GPT-3 (text-davinci-003) | instruction-tuned | 16.6% | 4.8% | 0.57 |
+| GPT-2 XL | base | 15.8% | 8.9% | 0.50 |
+| Mistral 7B | base | 12.4% | 5.6% | 0.50 |
+| Cohere | base | 12.2% | 3.7% | 0.56 |
+| MPT-30B | base | 10.1% | 5.7% | 0.44 |
+
+An AUROC below 0.5 (MPT-30B) means a text from that generator scores lower than a human text more often than it scores higher.
+
+## By domain
+
+| Domain | Above the cutoff at 5% | AUROC | Lowest score above RAID's cutoff (not the product's 30) |
+|---|---|---|---|
+| film reviews | 50.9% | 0.78 | 23.8 |
+| poetry | 36.7% | 0.67 | 34.6 |
+| book summaries | 33.7% | 0.71 | 34.7 |
+| news articles | 22.2% | 0.61 | 40.6 |
+| arXiv abstracts | 22.0% | 0.64 | 44.7 |
+| Wikipedia | 19.9% | 0.67 | 34.5 |
+| Reddit posts | 14.8% | 0.55 | 31.0 |
+| recipes | 12.1% | 0.53 | 8.9 |
+
+The engine scores to one decimal and RAID's cutoffs fall between those steps, so the last column gives the lowest score each cutoff counts rather than the cutoff rounded: rounding can carry a whole tie across it. The exact cutoffs are in `Docs/Detection/results.json`.
+
+Greedy decoding scores above the cutoff more often than sampling (30.4% against 22.7%, across all 11 generators). Only 6 generators were run both with and without a repetition penalty; across those, texts generated with it are above the cutoff 18.1% of the time and texts without it 28.4%. These are observed differences between settings, not controlled experiments, and what causes them was not measured.
+
+With one cutoff for every domain, closer to how the product works, the rate is 23.4% (lowest score counted 37.0).
+
+## Found along the way
+
+None of what follows was predicted. It is reported because it is true, and it should be read as observation rather than as a test that could have failed.
+
+### The product's boundary on short text
+
+At the product's boundary of 30/100, ignoring the length floor, 11.3% of RAID's human texts score above it, against 33.2% of the machine ones. The floor was not set from this number: it is the shortest text the calibration measured (#59). This is the first outside look at what lies under it. The human rate at the boundary falls as texts get longer, as the table shows, but this sample does not separate length from register.
+
+| Length | Human texts | Machine texts | Human above 30 (no verdict under 649 words) | Machine above 30 (same) | Human above RAID's cutoff | Machine above RAID's cutoff |
+|---|---|---|---|---|---|---|
+| under 200 words | 4,990 | 7,800 | 14.5% | 29.7% | 5.8% | 23.0% |
+| 200-399 words | 6,463 | 8,875 | 10.7% | 35.5% | 5.6% | 29.2% |
+| 400-648 words | 1,210 | 925 | 6.7% | 40.2% | 1.0% | 31.7% |
+| 649 or more words | 708 | 0 | 2.0% | n/a | 0.1% | n/a |
+
+Only 5.3% of the human texts and none of the machine texts reach 649 words.
+
+The only rates that describe the product as shipped are the published 0.7% (2 of 296) and, on this benchmark's long human texts, 2.0% (14 of 705). Every other percentage on this page is counted at a cutoff the product does not use, or on texts it gives no verdict on.
+
+### Long human writing the calibration does not contain
+
+705 distinct human texts in the sample reach the floor (708 rows: RAID repeats a few texts verbatim, and here each is counted once). They are an outside sample of long human writing, but of a different population from the calibration corpus, which holds research articles, encyclopedia revisions and learner essays. These are news articles, book summaries, film reviews and verse. At the product's boundary of 30/100:
+
+| Domain | Texts | Crossing the boundary | 95% interval | `stat.burstiness` fires on |
+|---|---|---|---|---|
+| news articles | 108 | 9 | 4.4–15.1% | 74% |
+| book summaries | 322 | 3 | 0.3–2.7% | 43% |
+| poetry | 108 | 2 | 0.5–6.5% | 13% |
+| recipes | 16 | 0 | 0.0–19.4% | 0% |
+| film reviews | 151 | 0 | 0.0–2.5% | 3% |
+| **all** | **705** | **14** | **1.2–3.3%** | **34%** |
+
+Overall that is 2.0% (14 of 705), against the published 0.7% (2 of 296, 0.2–2.4%): about three times the rate, a difference this sample is too small to establish. Without the news articles it is 5 of 597 (0.8%).
+
+**9 of 108 news articles cross the boundary** (8.3%, interval 4.4–15.1%). That interval lies wholly above the published one and reaches three times the 5% target: by the rule that set the boundary, 30/100 is not supported for news. News was singled out after seeing the table, and a domain chosen that way overstates itself, so this needs a second sample before it is quoted as a rate.
+
+All 14 texts that cross carry `stat.burstiness`. The rule fires on 74% of the long news articles, crossing or not, against 34% of all the long human texts here and 32% of the calibration corpus. News articles trip the sentence-rhythm rule far more often than the writing the boundary was measured on; why was not measured.
+
+The published rate was measured with engine 0.7.1 (2026-09-07) and this sample with 0.8.1. Re-running the calibration with 0.8.1 changed only its engine and date fields ([calibration-rerun.json](Detection/calibration-rerun.json)).
+
+This does not move the published number, which describes the corpus it was measured on. Any change to it goes through the calibration corpus and the review committee, not through RAID. What it does say: until a kind of writing has been measured, a score above 30 on it should be read as saying nothing, and that includes every kind absent from the calibration corpus, not only news. The fix is the one the calibration page gives: run the calibration on your own students' pre-2022 work (`Docs/Calibration/README.md`). If you have no such work, you have no boundary for your classroom.
+
+### Which rules carry the signal
+
+Share of texts each rule fires on, for the rules that fire most often on chat-tuned text relative to human text (at least 50 texts):
+
+| Rule | Human | Chat-tuned | Base | Chat ÷ human |
+|---|---|---|---|---|
+| `syn.testament-to` | 0.05% | 3.35% | 0.11% | 64× |
+| `lex.delve` | 0.13% | 4.96% | 0.09% | 39× |
+| `lex.tapestry` | 0.04% | 1.31% | 0.03% | 29× |
+| `lex.transformative` | 0.02% | 0.64% | 0.05% | 28× |
+| `lex.underscore` | 0.02% | 0.57% | 0.03% | 26× |
+| `lex.beacon` | 0.07% | 1.57% | 0.11% | 21× |
+| `lex.testament` | 0.19% | 3.97% | 0.17% | 20× |
+| `rhet.in-conclusion` | 0.14% | 2.82% | 0.23% | 20× |
+| `lex.nuanced` | 0.07% | 1.20% | 0.11% | 16× |
+| `lex.vibrant` | 0.19% | 2.84% | 0.14% | 15× |
+| `lex.bustling` | 0.05% | 0.74% | 0.02% | 14× |
+| `syn.superficial-ing` | 0.32% | 4.52% | 0.14% | 14× |
+
+Each of them is rare: strong evidence where it appears, absent from most texts. The rule with the largest absolute gap is `stat.burstiness`, which fires on 46.6% of human texts, 73.0% of chat-tuned ones and 42.7% of base-model ones. No rule was removed and re-scored, so how much of the separation each rule carries was not measured.
+
+Some rules fire more often on human text than on RAID's chat-tuned output: `lex.actually` (4.7% of human texts, 1.4% of chat-tuned), `lex.literally` (1.3% of human texts, 0.2% of chat-tuned), `rhet.em-dash` (1.6% of human texts, 0.1% of chat-tuned). On RAID these rules point the wrong way. Whether they separate later models was not measured here, and no rule was changed in response.
+
+### Attacks
+
+Eight of RAID's eleven attacks need no language model. They were run, at the strengths RAID's own code uses, on 1,000 human and 1,000 machine texts drawn from the sample. Paraphrase, synonym swap and perplexity misspelling need a model and were left out, and paraphrase is the attack most likely to hurt a detector built on word choice. What follows is robustness to cheap attacks, not to rewriting.
+
+| Attack | Strength | Above the cutoff before | After | Mean score | Scanner reports a strong artifact |
+|---|---|---|---|---|---|
+| alternative spelling | 1.0 | 24.6% | 24.1% | 20.6 → 20.4 | 0.2% |
+| article deletion | 0.5 | 24.6% | 24.7% | 20.6 → 20.6 | 0.2% |
+| homoglyph | 1.0 | 24.6% | 23.7% | 20.6 → 20.1 | 99.7% |
+| insert paragraphs | 0.5 | 24.6% | 24.6% | 20.6 → 20.6 | 0.2% |
+| number | 0.5 | 24.6% | 24.6% | 20.6 → 20.6 | 0.2% |
+| upper lower | 0.05 | 24.6% | 24.6% | 20.6 → 20.6 | 0.2% |
+| whitespace | 0.2 | 24.6% | 23.8% | 20.6 → 20.1 | 0.2% |
+| zero width space | 1.0 | 24.6% | 24.6% | 20.6 → 20.6 | 100.0% |
+
+Counting only characters that typing does not produce, the scanner reports 99.7% of the homoglyph-attacked machine texts and 99.8% of the human ones, and 100.0% and 100.0% under the zero-width attack: it describes the file, not who wrote it. On unattacked human texts it reports some artifact in 0.84%, a strong one in 0.03%, and a systematic pattern in 0.00%. The 5 homoglyph texts it did not report (3 machine, 2 human) are listed in `attacks.json` with a reason computed from the text: 3 are written in another script, where those letters are the alphabet, and 2 have no Latin letter left to disguise.
+
+**What the homoglyph attack still moves.** The scanner counts a disguised word only when Latin letters are at least half of it, so that a Greek or Cyrillic word quoted in English is not called a disguise. RAID's attack replaces every letter it can. In words such as *crucial*, *intricate* or a capitalised *Moreover* the substitutes become the majority. Those words are neither reported nor restored, so the rules that need them miss, while the document as a whole is still reported. Closing this is a change of behaviour, so it was not made here: it needs its own measurement against human text, because a Greek or Cyrillic word is sometimes exactly what the writer meant.
+
+### For scale: RAID's leaderboard
+
+Measured on RAID-test, read from the leaderboard's files at commit `84c2ea4`. The files were chosen by hand: the open detectors RAID's authors ran, and the highest entries on 2026-11-17; the list is in `leaderboard-scale.json`, and `fastdetectgpt.json` (no score for the all-domains, no-attack split) was skipped. These are different texts and a different mix: this sample weights the 11 generators equally, while RAID weights generators run with and without a repetition penalty twice (re-weighted to RAID's mix, the rate above is 25.3%). So they set a scale and are not rows in the same table as the numbers above. Nothing from this project has been submitted.
+
+| Detector, run by RAID's authors | No attack | Homoglyph | Zero-width | Whitespace |
+|---|---|---|---|---|
+| Binoculars | 79.0% | 36.1% | 98.4% | 68.7% |
+| RADAR | 65.6% | 44.8% | 78.4% | 61.1% |
+| GLTR | 59.7% | 20.3% | 97.9% | 43.1% |
+| RoBERTa (ChatGPT) | 42.5% | 0.0% | 0.0% | 0.2% |
+| LLMDet | 33.4% | 39.1% | 22.2% | 4.0% |
+
+Entries submitted by others sit between 92.4% and 99.9%. No result on this benchmark, theirs or the 26.6% above, says how a detector does on other text.
+
+Under the zero-width attack Binoculars and GLTR rise to about 98%. RAID sets its cutoff on unattacked human text only (`find_threshold` in RAID's `raid/evaluate.py`), so a detector whose score rises on any text carrying these characters gains detection and is charged nothing for it. Whether that is what happens with these two detectors was not tested here. This tool's scanner reports the attacked human files exactly as often as the machine ones, and keeps the finding out of the score.
+
+## Where this departs from the plan
+
+- **All 13,371 human texts** were used instead of 500 per domain. More human text makes the cutoff steadier, and it cannot be tuned in anyone's favour.
+- **Language fixed to English**, as the calibration does. The web app detects the language instead and would have misread 0.3% of the human texts and 0.1% of the machine ones.
+- **Scores divided by 100** for RAID's threshold search, which assumes a 0–1 range.
+- **Both cutoffs reported**: RAID's per-domain cutoff first, then one cutoff for all domains.
+- **Repeated texts.** RAID-train's human set holds 13,272 distinct texts in 13,371 rows. The headline counts rows, as RAID does; the long-text check counts each distinct text once.
+- **Generator mix.** The sample weights the 11 generators equally; RAID does not. Re-weighted to RAID's proportions the headline is 25.3% instead of 26.6%.
+
+## What this does not tell you
+
+- **What the product does on long machine text.** No machine text in the sample reaches 649 words, where the product starts giving verdicts.
+- **Anything about current models.** RAID's generators are from 2023. Newer models write differently and the tells move with them. The right response to "a newer model writes differently" is to measure it.
+- **A leaderboard result.** This is a sample of RAID-train, not RAID-test, and nothing has been submitted.
+- **Anything about students.** RAID's human texts are published writing. The effect on second-language learners (#75) is invisible here, which is what the calibration corpus is for.
+- **A cutoff tuned on other text.** RAID chooses each cutoff on the same human texts it then counts against. That is its method, and it favours every detector equally.
+- **Robustness to rewriting.** Paraphrase, synonym swap and misspelling were not run.
+- **More than one run.** One seed, one sample. Intervals are given only where shown.
+- **An independent author.** The same model that wrote the rule packs wrote these scripts and this page. The defence is that the counting is RAID's code, not ours, and that the predictions were committed first. The page was reviewed adversarially before merging; the reviews are not published.
+
+## Re-run it yourself
+
+See [tools/raid/README.md](../tools/raid/README.md). In short: download `train_none.csv`, draw the sample with `tools/raid/sample.py`, score it with `signsofai-calibrate raid`, count it with `tools/raid/evaluate.py`, attack it with `tools/raid/attack.py`, and write this page with `tools/raid/report.py`. The id fingerprint above says whether you drew the same sample.
diff --git a/Docs/Detection/attacks.json b/Docs/Detection/attacks.json
new file mode 100644
index 0000000..b511730
--- /dev/null
+++ b/Docs/Detection/attacks.json
@@ -0,0 +1,208 @@
+{
+ "setup": {
+ "seed": 20261117,
+ "perClass": 1000,
+ "attacks": {
+ "homoglyph": 1.0,
+ "zero_width_space": 1.0,
+ "whitespace": 0.2,
+ "upper_lower": 0.05,
+ "number": 0.5,
+ "article_deletion": 0.5,
+ "insert_paragraphs": 0.5,
+ "alternative_spelling": 1.0
+ },
+ "leftOut": [
+ "paraphrase",
+ "synonym",
+ "perplexity_misspelling"
+ ]
+ },
+ "attacks": {
+ "alternative_spelling": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.241,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.398300000000003,
+ "scanMachine": {
+ "any": 0.006,
+ "strong": 0.002,
+ "systematic": 0.0
+ },
+ "scanHuman": {
+ "any": 0.003,
+ "strong": 0.0,
+ "systematic": 0.0
+ }
+ },
+ "article_deletion": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.247,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.583,
+ "scanMachine": {
+ "any": 0.006,
+ "strong": 0.002,
+ "systematic": 0.0
+ },
+ "scanHuman": {
+ "any": 0.003,
+ "strong": 0.0,
+ "systematic": 0.0
+ }
+ },
+ "homoglyph": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.237,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.081599999999998,
+ "scanMachine": {
+ "any": 0.997,
+ "strong": 0.997,
+ "systematic": 0.996
+ },
+ "scanHuman": {
+ "any": 0.998,
+ "strong": 0.998,
+ "systematic": 0.997
+ }
+ },
+ "insert_paragraphs": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.246,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.578800000000005,
+ "scanMachine": {
+ "any": 0.005,
+ "strong": 0.002,
+ "systematic": 0.0
+ },
+ "scanHuman": {
+ "any": 0.003,
+ "strong": 0.0,
+ "systematic": 0.0
+ }
+ },
+ "number": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.246,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.6116,
+ "scanMachine": {
+ "any": 0.006,
+ "strong": 0.002,
+ "systematic": 0.0
+ },
+ "scanHuman": {
+ "any": 0.003,
+ "strong": 0.0,
+ "systematic": 0.0
+ }
+ },
+ "upper_lower": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.246,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.6116,
+ "scanMachine": {
+ "any": 0.006,
+ "strong": 0.002,
+ "systematic": 0.0
+ },
+ "scanHuman": {
+ "any": 0.003,
+ "strong": 0.0,
+ "systematic": 0.0
+ }
+ },
+ "whitespace": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.238,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.140099999999997,
+ "scanMachine": {
+ "any": 0.006,
+ "strong": 0.002,
+ "systematic": 0.0
+ },
+ "scanHuman": {
+ "any": 0.003,
+ "strong": 0.0,
+ "systematic": 0.0
+ }
+ },
+ "zero_width_space": {
+ "tprBefore": 0.246,
+ "tprAfter": 0.246,
+ "meanScoreBefore": 20.6116,
+ "meanScoreAfter": 20.6116,
+ "scanMachine": {
+ "any": 1.0,
+ "strong": 1.0,
+ "systematic": 1.0
+ },
+ "scanHuman": {
+ "any": 1.0,
+ "strong": 1.0,
+ "systematic": 1.0
+ }
+ }
+ },
+ "unattackedHumanScan": {
+ "any": 0.00837633684840326,
+ "strong": 0.0002991548874429736,
+ "systematic": 0.0
+ },
+ "unattackedMachineScan": {
+ "any": 0.004431818181818182,
+ "strong": 0.0021022727272727274,
+ "systematic": 0.00011363636363636364
+ },
+ "predictions": {
+ "P5": {
+ "homoglyphStrong": 0.997,
+ "zeroWidthStrong": 1.0,
+ "humanAlarms": 112,
+ "humanTexts": 13371,
+ "humanAlarmInterval": [
+ 0.006966539636313223,
+ 0.0100685377200619
+ ],
+ "held": true
+ },
+ "P6": {
+ "scoreDrop": 0.5300000000000011,
+ "held": true
+ }
+ },
+ "scannerMisses": {
+ "homoglyph": [
+ {
+ "id": "1be69639-7772-42e3-a9a4-b7ba487f043a::homoglyph",
+ "class": "machine",
+ "reason": "written in another script"
+ },
+ {
+ "id": "253f18bf-bad3-45b0-bab7-05f178d47b1b::homoglyph",
+ "class": "human",
+ "reason": "no Latin letter left"
+ },
+ {
+ "id": "28726675-3f10-4220-a2f7-09061175a1c3::homoglyph",
+ "class": "machine",
+ "reason": "no Latin letter left"
+ },
+ {
+ "id": "7974922a-02e8-4db9-8483-862dbe870103::homoglyph",
+ "class": "human",
+ "reason": "written in another script"
+ },
+ {
+ "id": "ef0a1b4f-66a0-4d6a-872b-7e415d9cc788::homoglyph",
+ "class": "machine",
+ "reason": "written in another script"
+ }
+ ],
+ "zero_width_space": []
+ }
+}
diff --git a/Docs/Detection/calibration-rerun.json b/Docs/Detection/calibration-rerun.json
new file mode 100644
index 0000000..4c60826
--- /dev/null
+++ b/Docs/Detection/calibration-rerun.json
@@ -0,0 +1 @@
+{"corpusId":"signsofai-human-baseline","texts":296,"measuredOn":"2026-11-18","engine":"0.8.1","recommendedThreshold":30,"flaggedAtThreshold":2,"rateLow":0.0018549062642001807,"rateHigh":0.024297110573472523,"shortestWords":649,"longestWords":9328,"noisiestRules":[{"ruleId":"lex.just","textShare":0.34797297297297297},{"ruleId":"stat.burstiness","textShare":0.32094594594594594},{"ruleId":"rhet.in-conclusion","textShare":0.30067567567567566},{"ruleId":"rhet.rule-of-three","textShare":0.26013513513513514},{"ruleId":"lex.moreover","textShare":0.22297297297297297},{"ruleId":"rhet.not-only-but","textShare":0.21283783783783783},{"ruleId":"lex.furthermore","textShare":0.1858108108108108},{"ruleId":"lex.crucial","textShare":0.16891891891891891}],"languages":[{"language":"en","texts":271,"recommendedThreshold":30,"bestBound":0.01397699931055571,"rateHighAtThreshold":0.026504623306870336},{"language":"es","texts":25,"recommendedThreshold":null,"bestBound":0.13319225276039096,"rateHighAtThreshold":null}]}
diff --git a/Docs/Detection/leaderboard-scale.json b/Docs/Detection/leaderboard-scale.json
new file mode 100644
index 0000000..f0c49c8
--- /dev/null
+++ b/Docs/Detection/leaderboard-scale.json
@@ -0,0 +1,127 @@
+{
+ "source": "github.com/liamdugan/raid leaderboard/submissions/*/results.json",
+ "commit": "84c2ea413752a39def012394a0ce905c8a692b05",
+ "split": "RAID-test",
+ "selection": "chosen by hand: the open detectors RAID's authors ran, and the highest entries on 2026-11-17",
+ "filesRead": [
+ "binoculars.json",
+ "chatgpt-roberta.json",
+ "desklib.json",
+ "fastdetectgpt.json",
+ "gltr.json",
+ "gptzero.json",
+ "grammarly.json",
+ "llmdet.json",
+ "QuillBot.json",
+ "radar.json",
+ "Slop-Or-Not.json"
+ ],
+ "skipped": [
+ {
+ "file": "fastdetectgpt.json",
+ "reason": "no score for the all-domains, no-attack split"
+ }
+ ],
+ "detectors": [
+ {
+ "name": "Grammarly",
+ "submittedBy": "others",
+ "tpr05": {
+ "none": 0.9991360294117647,
+ "homoglyph": 0.9920588235294118,
+ "zero_width_space": 0.9668382352941176,
+ "whitespace": 0.9989889705882353
+ }
+ },
+ {
+ "name": "QuillBot",
+ "submittedBy": "others",
+ "tpr05": {
+ "none": 0.9972058823529412,
+ "homoglyph": 0.9960845588235294,
+ "zero_width_space": 0.9925551470588235,
+ "whitespace": 0.9982904411764706
+ }
+ },
+ {
+ "name": "Slop Or Not",
+ "submittedBy": "others",
+ "tpr05": {
+ "none": 0.9964338235294118,
+ "homoglyph": 0.9950367647058823,
+ "zero_width_space": 0.9929227941176471,
+ "whitespace": 0.9983455882352941
+ }
+ },
+ {
+ "name": "GPTZero",
+ "submittedBy": "others",
+ "tpr05": {
+ "none": 0.9836764705882353,
+ "homoglyph": 0.9836397058823529,
+ "zero_width_space": 0.9836580882352941,
+ "whitespace": 0.9836397058823529
+ }
+ },
+ {
+ "name": "Desklib",
+ "submittedBy": "others",
+ "tpr05": {
+ "none": 0.9238235294117647,
+ "homoglyph": 0.6915441176470588,
+ "zero_width_space": 0.4871507352941176,
+ "whitespace": 0.9238235294117647
+ }
+ },
+ {
+ "name": "Binoculars",
+ "submittedBy": "RAID authors",
+ "tpr05": {
+ "none": 0.7898161764705882,
+ "homoglyph": 0.36099264705882356,
+ "zero_width_space": 0.9840808823529412,
+ "whitespace": 0.6868566176470589
+ }
+ },
+ {
+ "name": "RADAR",
+ "submittedBy": "RAID authors",
+ "tpr05": {
+ "none": 0.6561397058823529,
+ "homoglyph": 0.44830882352941176,
+ "zero_width_space": 0.7840073529411765,
+ "whitespace": 0.6113786764705882
+ }
+ },
+ {
+ "name": "GLTR",
+ "submittedBy": "RAID authors",
+ "tpr05": {
+ "none": 0.5969485294117647,
+ "homoglyph": 0.2031985294117647,
+ "zero_width_space": 0.97875,
+ "whitespace": 0.4305514705882353
+ }
+ },
+ {
+ "name": "RoBERTa (ChatGPT)",
+ "submittedBy": "RAID authors",
+ "tpr05": {
+ "none": 0.42518382352941175,
+ "homoglyph": 0.00014705882352941175,
+ "zero_width_space": 0.00016544117647058823,
+ "whitespace": 0.0015625
+ }
+ },
+ {
+ "name": "LLMDet",
+ "submittedBy": "RAID authors",
+ "tpr05": {
+ "none": 0.3339889705882353,
+ "homoglyph": 0.39088235294117646,
+ "zero_width_space": 0.22240808823529412,
+ "whitespace": 0.04029411764705882
+ }
+ }
+ ]
+}
diff --git a/Docs/Detection/prediction.md b/Docs/Detection/prediction.md
new file mode 100644
index 0000000..3ba8981
--- /dev/null
+++ b/Docs/Detection/prediction.md
@@ -0,0 +1,67 @@
+# What we expected before measuring detection on RAID
+
+Written on **17 November 2026**, before a single RAID text had been scored by this engine, and
+committed before the measurement so the order can be checked in the history. The results page that
+follows it must report each line below as it stands, including the ones that turn out wrong.
+
+## Why this exists
+
+`Docs/CALIBRATION.md` measures how often the tool is wrong about a human, and says plainly what it
+leaves out: *nothing about how much AI writing it catches*, because a tool that flags nothing has a
+perfect false-positive rate. RAID (Dugan et al., ACL 2024, arXiv 2405.07940) measures that other
+half: the share of machine-generated text a detector catches at a fixed false-positive rate on RAID's
+own human texts.
+
+Two things about RAID shape what can be predicted:
+
+- **RAID picks its own threshold.** It takes the raw score and, by default separately for each
+ domain, finds the cutoff that flags 5% of its human texts. Our 30/100 boundary and the length floor
+ from #59 play no part, and the cutoff RAID finds is not ours.
+- **RAID's humans are not students.** News, Wikipedia, arXiv abstracts, Reddit, reviews, recipes,
+ poetry and book summaries. Whatever this measures says nothing about second-language learners (#75),
+ which is what the calibration is for.
+
+## The predictions
+
+RAID's generators fall into three groups. **Chat-tuned:** ChatGPT, GPT-4, Llama 2 70B Chat, Cohere
+Chat, MPT-30B Chat, Mistral 7B Chat. **Base models:** GPT-2 XL, MPT-30B, Mistral 7B, Cohere.
+**In between:** GPT-3 text-davinci-003, which is instruction-tuned.
+
+The underlying claim: **the tells this engine looks for are fingerprints of chat-tuning, not of
+machine text.**
+
+| # | Prediction | What would refute it |
+|---|---|---|
+| P1 | The detection rate on chat-tuned generators is **at least twice** the rate on base models | A ratio below two |
+| P2 | On base models the rate stays **under 20%**. A score that knew nothing would catch 5% | 20% or more |
+| P3 | text-davinci-003 lands **between** the two groups | Outside that range |
+| P4 | **Poetry and recipes**, short and formulaic, are worse than news, Reddit, reviews and books | Poetry or recipes among the three best domains |
+| P5 | The character scanner catches **at least 99%** of texts under RAID's homoglyph and zero-width-space attacks, and flags **almost none** of the unattacked human texts | Under 99%, or false alarms on more than 1% of human texts |
+| P6 | The homoglyph attack **lowers the score** (disguised words stop matching the rules), and it is exactly the attack the scanner exposes | The score does not drop |
+
+**Overall guess, held with low confidence:** between **15% and 40%** detection at a 5% false-positive
+rate, without attacks. Far below the commercial detectors on the leaderboard, which is consistent with
+what this project has said from the start: it cannot tell who wrote a text, and it does not try.
+
+## Note added later the same day, still before measuring
+
+Reading `AiWritingAnalyzer.Analyze` while writing the scoring code: the analyzer **normalizes
+character artifacts before any rule reads the text**, precisely so that a Cyrillic letter inside
+"delve" cannot switch the catalog off. If that works as written, **P6 is wrong**: RAID's homoglyph and
+zero-width attacks should leave the score where it was, while the scanner reports them. P6 stays above
+exactly as it was written, and the results will say which of the two happened.
+
+## What is measured, and how
+
+- A stratified sample of **RAID-train without attacks** (labelled and public): 500 human texts per
+ domain and 200 machine texts per generator and domain, fixed by a seed and recorded in a manifest.
+- Scored by the shipped engine and packs, unchanged, through `tools/SignsOfAI.Calibration`, with the
+ language set to English as the calibration does. The score is the 0–100 the reader sees, before the
+ length floor, divided by 100 because RAID's threshold search assumes a 0–1 range.
+- Evaluated by **RAID's own `run_evaluation`**, not a reimplementation of it, both per domain (RAID's
+ default) and with a single threshold across domains (closer to how the product works).
+- For attacks, the lightweight ones from RAID's own code are applied to a subsample. The scanner's
+ catch rate is measured **separately** and never added to the score.
+
+No rule or threshold will be changed in response to these numbers inside this measurement. A change
+suggested by RAID goes through the usual path: measured against the human corpus before it is written.
diff --git a/Docs/Detection/results.json b/Docs/Detection/results.json
new file mode 100644
index 0000000..6d538a6
--- /dev/null
+++ b/Docs/Detection/results.json
@@ -0,0 +1,1370 @@
+{
+ "engine": "0.8.1",
+ "raidBench": "0.2.0",
+ "sample": {
+ "dataset": "RAID-train without attacks (train_none.csv)",
+ "datasetUrl": "https://dataset.raid-bench.xyz/train_none.csv",
+ "datasetSha256": "c5467bca6fc7f5c728c676450c7f84ce401df6c6ccc6d82c47e3b5f3c6d6fce4",
+ "seed": 20261117,
+ "machinePerGeneratorAndDomain": 200,
+ "human": 13371,
+ "machine": 17600,
+ "total": 30971,
+ "humanByDomain": {
+ "abstracts": 1766,
+ "books": 1781,
+ "news": 1780,
+ "poetry": 1771,
+ "recipes": 1772,
+ "reddit": 1779,
+ "reviews": 943,
+ "wiki": 1779
+ },
+ "machineByModel": {
+ "chatgpt": 1600,
+ "cohere": 1600,
+ "cohere-chat": 1600,
+ "gpt2": 1600,
+ "gpt3": 1600,
+ "gpt4": 1600,
+ "llama-chat": 1600,
+ "mistral": 1600,
+ "mistral-chat": 1600,
+ "mpt": 1600,
+ "mpt-chat": 1600
+ },
+ "datasetMachineByModel": {
+ "chatgpt": 26742,
+ "cohere": 26742,
+ "cohere-chat": 26742,
+ "gpt2": 53484,
+ "gpt3": 26742,
+ "gpt4": 26742,
+ "llama-chat": 53484,
+ "mistral": 53484,
+ "mistral-chat": 53484,
+ "mpt": 53484,
+ "mpt-chat": 53484
+ },
+ "idFingerprintSha256": "136c179ba9288e53bab98e5086cde93923965863350c0544df1f4f028903db5b",
+ "pandas": "2.2.3"
+ },
+ "language": "en",
+ "detectedNotEnglish": {
+ "human": 0.003141126318151223,
+ "machine": 0.0011931818181818183
+ },
+ "perDomainThreshold": {
+ "overall": {
+ "tpr05": 0.26551136363636363,
+ "tpr01": 0.12886363636363637,
+ "auroc": 0.6290205694481271
+ },
+ "thresholds": {
+ "0.05": {
+ "news": 0.40500000014628257,
+ "reddit": 0.3090000000002513,
+ "reviews": 0.23750112672322377,
+ "books": 0.34672011422655813,
+ "recipes": 0.08807745485327312,
+ "wiki": 0.3442784394322653,
+ "abstracts": 0.44698202151755373,
+ "poetry": 0.3454438170525127
+ },
+ "0.01": {
+ "news": 0.48229480337078656,
+ "reddit": 0.4050079749859472,
+ "reviews": 0.33320425172322377,
+ "books": 0.44144667672655813,
+ "recipes": 0.13885870485327312,
+ "wiki": 0.4380284394322653,
+ "abstracts": 0.5368257715175537,
+ "poetry": 0.5876313170525127
+ }
+ },
+ "lowestCounted": {
+ "news": 40.6,
+ "reddit": 31.0,
+ "reviews": 23.8,
+ "books": 34.7,
+ "recipes": 8.9,
+ "wiki": 34.5,
+ "abstracts": 44.7,
+ "poetry": 34.6
+ },
+ "trueFpr": {
+ "0.05": {
+ "news": 0.047752808988763995,
+ "reddit": 0.04946599213041036,
+ "reviews": 0.049840933191940606,
+ "books": 0.049971925884334634,
+ "recipes": 0.04966139954853277,
+ "wiki": 0.050028105677346835,
+ "abstracts": 0.050396375990939934,
+ "poetry": 0.0496894409937888
+ },
+ "0.01": {
+ "news": 0.009550561797752821,
+ "reddit": 0.009555930297920234,
+ "reviews": 0.009544008483563071,
+ "books": 0.010106681639528325,
+ "recipes": 0.010158013544018019,
+ "wiki": 0.009555930297920234,
+ "abstracts": 0.00962627406568517,
+ "poetry": 0.009599096555618303
+ }
+ },
+ "byModel": {
+ "chatgpt": {
+ "tpr05": 0.595625,
+ "tpr01": 0.37375,
+ "auroc": 0.8241476656570188,
+ "n": 1600
+ },
+ "cohere": {
+ "tpr05": 0.121875,
+ "tpr01": 0.036875,
+ "auroc": 0.5583628748784683,
+ "n": 1600
+ },
+ "cohere-chat": {
+ "tpr05": 0.215,
+ "tpr01": 0.076875,
+ "auroc": 0.6281445852965374,
+ "n": 1600
+ },
+ "gpt2": {
+ "tpr05": 0.1575,
+ "tpr01": 0.089375,
+ "auroc": 0.501044214157505,
+ "n": 1600
+ },
+ "gpt3": {
+ "tpr05": 0.16625,
+ "tpr01": 0.0475,
+ "auroc": 0.5676876495774438,
+ "n": 1600
+ },
+ "gpt4": {
+ "tpr05": 0.435,
+ "tpr01": 0.24375,
+ "auroc": 0.779918410178745,
+ "n": 1600
+ },
+ "llama-chat": {
+ "tpr05": 0.361875,
+ "tpr01": 0.166875,
+ "auroc": 0.7287554221823348,
+ "n": 1600
+ },
+ "mistral": {
+ "tpr05": 0.124375,
+ "tpr01": 0.05625,
+ "auroc": 0.5021589400568394,
+ "n": 1600
+ },
+ "mistral-chat": {
+ "tpr05": 0.35625,
+ "tpr01": 0.14875,
+ "auroc": 0.7305786543639219,
+ "n": 1600
+ },
+ "mpt": {
+ "tpr05": 0.100625,
+ "tpr01": 0.056875,
+ "auroc": 0.4384858088400269,
+ "n": 1600
+ },
+ "mpt-chat": {
+ "tpr05": 0.28625,
+ "tpr01": 0.120625,
+ "auroc": 0.6599420387405579,
+ "n": 1600
+ }
+ },
+ "byDomain": {
+ "abstracts": {
+ "tpr05": 0.22,
+ "tpr01": 0.06409090909090909,
+ "auroc": 0.6359171985998147
+ },
+ "books": {
+ "tpr05": 0.3372727272727273,
+ "tpr01": 0.18045454545454545,
+ "auroc": 0.710377979684549
+ },
+ "news": {
+ "tpr05": 0.22181818181818183,
+ "tpr01": 0.09227272727272727,
+ "auroc": 0.6079744637385086
+ },
+ "poetry": {
+ "tpr05": 0.36727272727272725,
+ "tpr01": 0.15818181818181817,
+ "auroc": 0.6745738155125506
+ },
+ "recipes": {
+ "tpr05": 0.12136363636363637,
+ "tpr01": 0.046363636363636364,
+ "auroc": 0.5292014672686229
+ },
+ "reddit": {
+ "tpr05": 0.1481818181818182,
+ "tpr01": 0.06954545454545455,
+ "auroc": 0.5505960958659104
+ },
+ "reviews": {
+ "tpr05": 0.509090909090909,
+ "tpr01": 0.33545454545454545,
+ "auroc": 0.7830432372505545
+ },
+ "wiki": {
+ "tpr05": 0.1990909090909091,
+ "tpr01": 0.08454545454545455,
+ "auroc": 0.6683374214318565
+ }
+ },
+ "byDecoding": {
+ "greedy": {
+ "tpr05": 0.30430316173952743,
+ "auroc": 0.6565801727116672
+ },
+ "sampling": {
+ "tpr05": 0.2270618848286005,
+ "auroc": 0.6017041666659263
+ }
+ },
+ "byRepetitionPenalty": {
+ "no": {
+ "tpr05": 0.2981752398930313,
+ "auroc": 0.6553431929974791
+ },
+ "yes": {
+ "tpr05": 0.18051575931232092,
+ "auroc": 0.5605257197128714
+ }
+ },
+ "byGroup": {
+ "chat": {
+ "tpr05": 0.375,
+ "models": [
+ "chatgpt",
+ "gpt4",
+ "llama-chat",
+ "cohere-chat",
+ "mpt-chat",
+ "mistral-chat"
+ ],
+ "n": 9600
+ },
+ "base": {
+ "tpr05": 0.12609375,
+ "models": [
+ "gpt2",
+ "mpt",
+ "mistral",
+ "cohere"
+ ],
+ "n": 6400
+ },
+ "between": {
+ "tpr05": 0.16625,
+ "models": [
+ "gpt3"
+ ],
+ "n": 1600
+ }
+ }
+ },
+ "singleThreshold": {
+ "overall": {
+ "tpr05": 0.23375,
+ "tpr01": 0.10744318181818181,
+ "auroc": 0.6290205694481271
+ },
+ "thresholds": {
+ "0.05": 0.3695574225469299,
+ "0.01": 0.4769792975469299
+ },
+ "lowestCounted": 37.0,
+ "trueFpr": {
+ "0.05": 0.049884077481115896,
+ "0.01": 0.009946900007478887
+ },
+ "byModel": {
+ "chatgpt": {
+ "tpr05": 0.55375,
+ "tpr01": 0.3125,
+ "auroc": 0.8241476656570188,
+ "n": 1600
+ },
+ "cohere": {
+ "tpr05": 0.094375,
+ "tpr01": 0.026875,
+ "auroc": 0.5583628748784683,
+ "n": 1600
+ },
+ "cohere-chat": {
+ "tpr05": 0.17375,
+ "tpr01": 0.06125,
+ "auroc": 0.6281445852965374,
+ "n": 1600
+ },
+ "gpt2": {
+ "tpr05": 0.123125,
+ "tpr01": 0.07625,
+ "auroc": 0.501044214157505,
+ "n": 1600
+ },
+ "gpt3": {
+ "tpr05": 0.10875,
+ "tpr01": 0.02,
+ "auroc": 0.5676876495774438,
+ "n": 1600
+ },
+ "gpt4": {
+ "tpr05": 0.4225,
+ "tpr01": 0.218125,
+ "auroc": 0.779918410178745,
+ "n": 1600
+ },
+ "llama-chat": {
+ "tpr05": 0.335625,
+ "tpr01": 0.16125,
+ "auroc": 0.7287554221823348,
+ "n": 1600
+ },
+ "mistral": {
+ "tpr05": 0.105625,
+ "tpr01": 0.0425,
+ "auroc": 0.5021589400568394,
+ "n": 1600
+ },
+ "mistral-chat": {
+ "tpr05": 0.325625,
+ "tpr01": 0.128125,
+ "auroc": 0.7305786543639219,
+ "n": 1600
+ },
+ "mpt": {
+ "tpr05": 0.08375,
+ "tpr01": 0.04875,
+ "auroc": 0.4384858088400269,
+ "n": 1600
+ },
+ "mpt-chat": {
+ "tpr05": 0.244375,
+ "tpr01": 0.08625,
+ "auroc": 0.6599420387405579,
+ "n": 1600
+ }
+ },
+ "byDomain": {
+ "abstracts": {
+ "tpr05": 0.37636363636363634,
+ "tpr01": 0.16863636363636364,
+ "auroc": 0.6359171985998147
+ },
+ "books": {
+ "tpr05": 0.30636363636363634,
+ "tpr01": 0.12545454545454546,
+ "auroc": 0.710377979684549
+ },
+ "news": {
+ "tpr05": 0.30727272727272725,
+ "tpr01": 0.09863636363636363,
+ "auroc": 0.6079744637385086
+ },
+ "poetry": {
+ "tpr05": 0.3481818181818182,
+ "tpr01": 0.25681818181818183,
+ "auroc": 0.6745738155125506
+ },
+ "recipes": {
+ "tpr05": 0.004545454545454545,
+ "tpr01": 0.0027272727272727275,
+ "auroc": 0.5292014672686229
+ },
+ "reddit": {
+ "tpr05": 0.09454545454545454,
+ "tpr01": 0.03954545454545454,
+ "auroc": 0.5505960958659104
+ },
+ "reviews": {
+ "tpr05": 0.27136363636363636,
+ "tpr01": 0.12090909090909091,
+ "auroc": 0.7830432372505545
+ },
+ "wiki": {
+ "tpr05": 0.16136363636363638,
+ "tpr01": 0.04681818181818182,
+ "auroc": 0.6683374214318565
+ }
+ },
+ "byDecoding": {
+ "greedy": {
+ "tpr05": 0.2690332153863714,
+ "auroc": 0.6565801727116672
+ },
+ "sampling": {
+ "tpr05": 0.1987781423237923,
+ "auroc": 0.6017041666659263
+ }
+ },
+ "byRepetitionPenalty": {
+ "no": {
+ "tpr05": 0.26521944313355356,
+ "auroc": 0.6553431929974791
+ },
+ "yes": {
+ "tpr05": 0.1518624641833811,
+ "auroc": 0.5605257197128714
+ }
+ },
+ "byGroup": {
+ "chat": {
+ "tpr05": 0.34260416666666665,
+ "models": [
+ "chatgpt",
+ "gpt4",
+ "llama-chat",
+ "cohere-chat",
+ "mpt-chat",
+ "mistral-chat"
+ ],
+ "n": 9600
+ },
+ "base": {
+ "tpr05": 0.10171875,
+ "models": [
+ "gpt2",
+ "mpt",
+ "mistral",
+ "cohere"
+ ],
+ "n": 6400
+ },
+ "between": {
+ "tpr05": 0.10875,
+ "models": [
+ "gpt3"
+ ],
+ "n": 1600
+ }
+ }
+ },
+ "byLength": [
+ {
+ "bucket": "under 200",
+ "machine": 7800,
+ "human": 4990,
+ "tpr05": 0.2296153846153846,
+ "fpr05": 0.05791583166332665,
+ "humanAtBoundary": 0.14509018036072144,
+ "machineAtBoundary": 0.2969230769230769
+ },
+ {
+ "bucket": "200-399",
+ "machine": 8875,
+ "human": 6463,
+ "tpr05": 0.2917183098591549,
+ "fpr05": 0.05585641343029553,
+ "humanAtBoundary": 0.10676156583629894,
+ "machineAtBoundary": 0.35538028169014085
+ },
+ {
+ "bucket": "400-648",
+ "machine": 925,
+ "human": 1210,
+ "tpr05": 0.31675675675675674,
+ "fpr05": 0.009917355371900827,
+ "humanAtBoundary": 0.06694214876033058,
+ "machineAtBoundary": 0.40216216216216216
+ },
+ {
+ "bucket": "649 or more",
+ "machine": 0,
+ "human": 708,
+ "tpr05": null,
+ "fpr05": 0.0014124293785310734,
+ "humanAtBoundary": 0.01977401129943503,
+ "machineAtBoundary": null
+ }
+ ],
+ "matchedRepetitionPenalty": {
+ "models": [
+ "gpt2",
+ "llama-chat",
+ "mistral",
+ "mistral-chat",
+ "mpt",
+ "mpt-chat"
+ ],
+ "no": {
+ "tpr05": 0.28362324989393295,
+ "n": 4714
+ },
+ "yes": {
+ "tpr05": 0.18051575931232092,
+ "n": 4886
+ }
+ },
+ "matchedDecoding": {
+ "models": [
+ "chatgpt",
+ "cohere",
+ "cohere-chat",
+ "gpt2",
+ "gpt3",
+ "gpt4",
+ "llama-chat",
+ "mistral",
+ "mistral-chat",
+ "mpt",
+ "mpt-chat"
+ ],
+ "greedy": {
+ "tpr05": 0.30430316173952743,
+ "n": 8761
+ },
+ "sampling": {
+ "tpr05": 0.2270618848286005,
+ "n": 8839
+ }
+ },
+ "humanTexts": {
+ "rows": 13371,
+ "distinct": 13272
+ },
+ "product": {
+ "boundary": 30,
+ "lengthFloorWords": 649,
+ "ignoringFloor": {
+ "fpr": 0.11285618128786179,
+ "tpr": 0.33193181818181816
+ },
+ "shareAtOrAboveFloor": {
+ "human": 0.052950415077406325,
+ "machine": 0.0
+ },
+ "withFloor": {
+ "humanFlagged": 0.0010470421060504076,
+ "machineFlagged": 0.0
+ },
+ "medianWords": {
+ "human": 222.0,
+ "machine": 219.0
+ }
+ },
+ "calibration": {
+ "engine": "0.7.1",
+ "measuredOn": "2026-09-07",
+ "burstinessShare": 0.32094594594594594,
+ "rerun": {
+ "engine": "0.8.1",
+ "measuredOn": "2026-11-18",
+ "fieldsChanged": [
+ "engine",
+ "measuredOn"
+ ],
+ "file": "Docs/Detection/calibration-rerun.json"
+ }
+ },
+ "predictions": [
+ {
+ "id": "P1",
+ "claim": "chat-tuned rate at least twice the base-model rate",
+ "measured": "chat 0.375, base 0.126, ratio 2.97",
+ "held": true
+ },
+ {
+ "id": "P2",
+ "claim": "base-model rate under 20%",
+ "measured": "0.126",
+ "held": true
+ },
+ {
+ "id": "P3",
+ "claim": "text-davinci-003 between the two groups",
+ "measured": "0.166 (range 0.126-0.375)",
+ "held": true
+ },
+ {
+ "id": "P4",
+ "claim": "poetry and recipes not among the three best domains",
+ "measured": "best three: reviews 0.509, poetry 0.367, books 0.337",
+ "held": false
+ },
+ {
+ "id": "overall",
+ "claim": "between 15% and 40% overall, without attacks",
+ "measured": "0.266",
+ "held": true
+ }
+ ],
+ "predictionsSingleThreshold": [
+ {
+ "id": "P1",
+ "claim": "chat-tuned rate at least twice the base-model rate",
+ "measured": "chat 0.343, base 0.102, ratio 3.37",
+ "held": true
+ },
+ {
+ "id": "P2",
+ "claim": "base-model rate under 20%",
+ "measured": "0.102",
+ "held": true
+ },
+ {
+ "id": "P3",
+ "claim": "text-davinci-003 between the two groups",
+ "measured": "0.109 (range 0.102-0.343)",
+ "held": true
+ },
+ {
+ "id": "P4",
+ "claim": "poetry and recipes not among the three best domains",
+ "measured": "best three: abstracts 0.376, poetry 0.348, news 0.307",
+ "held": false
+ },
+ {
+ "id": "overall",
+ "claim": "between 15% and 40% overall, without attacks",
+ "measured": "0.234",
+ "held": true
+ }
+ ],
+ "raidMixTpr05": 0.25338235294117645,
+ "longHumanCheck": {
+ "rows": 708,
+ "texts": 705,
+ "flagged": 14,
+ "rate": 0.019858156028368795,
+ "interval": [
+ 0.011865314049517352,
+ 0.03305510994987548
+ ],
+ "byDomain": {
+ "books": {
+ "texts": 322,
+ "flagged": 3,
+ "interval": [
+ 0.0031735030165784012,
+ 0.027029709659563105
+ ],
+ "burstiness": 0.42857142857142855
+ },
+ "news": {
+ "texts": 108,
+ "flagged": 9,
+ "interval": [
+ 0.044460404358563896,
+ 0.15082906253293688
+ ],
+ "burstiness": 0.7407407407407407
+ },
+ "poetry": {
+ "texts": 108,
+ "flagged": 2,
+ "interval": [
+ 0.005093224298949841,
+ 0.06501904855345103
+ ],
+ "burstiness": 0.12962962962962962
+ },
+ "recipes": {
+ "texts": 16,
+ "flagged": 0,
+ "interval": [
+ 0.0,
+ 0.1936076829974049
+ ],
+ "burstiness": 0.0
+ },
+ "reviews": {
+ "texts": 151,
+ "flagged": 0,
+ "interval": [
+ 0.0,
+ 0.024808981451414282
+ ],
+ "burstiness": 0.033112582781456956
+ }
+ },
+ "rulesInFlagged": {
+ "stat.burstiness": 14,
+ "lex.just": 7,
+ "lex.actually": 4,
+ "lex.truly": 2,
+ "lex.simply": 2,
+ "rhet.weasel-attribution": 2,
+ "rhet.with-regard-to": 1,
+ "rhet.rule-of-three": 1,
+ "rhet.in-order-to": 1,
+ "lex.notably": 1,
+ "lex.embark": 1,
+ "lex.literally": 1,
+ "lex.testament": 1,
+ "lex.crucial": 1,
+ "rhet.in-conclusion": 1,
+ "lex.profound": 1,
+ "rhet.not-only-but": 1,
+ "syn.serves-as": 1
+ },
+ "burstinessAll": 0.33617021276595743,
+ "withoutNews": {
+ "texts": 597,
+ "flagged": 5
+ },
+ "published": {
+ "texts": 296,
+ "flagged": 2,
+ "interval": [
+ 0.0018549062642001807,
+ 0.024297110573472523
+ ]
+ }
+ },
+ "rules": [
+ {
+ "rule": "syn.testament-to",
+ "texts": 340,
+ "human": 0.0005235210530252038,
+ "chat": 0.033541666666666664,
+ "base": 0.00109375,
+ "chatLift": 64.069375
+ },
+ {
+ "rule": "lex.delve",
+ "texts": 500,
+ "human": 0.0012714082716326377,
+ "chat": 0.04958333333333333,
+ "base": 0.0009375,
+ "chatLift": 38.99875
+ },
+ {
+ "rule": "lex.tapestry",
+ "texts": 134,
+ "human": 0.0004487323311644604,
+ "chat": 0.013125,
+ "base": 0.0003125,
+ "chatLift": 29.249062499999997
+ },
+ {
+ "rule": "lex.transformative",
+ "texts": 67,
+ "human": 0.0002243661655822302,
+ "chat": 0.006354166666666667,
+ "base": 0.00046875,
+ "chatLift": 28.320520833333333
+ },
+ {
+ "rule": "lex.underscore",
+ "texts": 60,
+ "human": 0.0002243661655822302,
+ "chat": 0.005729166666666666,
+ "base": 0.0003125,
+ "chatLift": 25.53489583333333
+ },
+ {
+ "rule": "lex.beacon",
+ "texts": 169,
+ "human": 0.000747887218607434,
+ "chat": 0.015729166666666666,
+ "base": 0.00109375,
+ "chatLift": 21.03146875
+ },
+ {
+ "rule": "lex.testament",
+ "texts": 422,
+ "human": 0.0019445067683793284,
+ "chat": 0.0396875,
+ "base": 0.00171875,
+ "chatLift": 20.410060096153845
+ },
+ {
+ "rule": "rhet.in-conclusion",
+ "texts": 306,
+ "human": 0.0014209857153541245,
+ "chat": 0.028229166666666666,
+ "base": 0.00234375,
+ "chatLift": 19.865904605263157
+ },
+ {
+ "rule": "lex.nuanced",
+ "texts": 132,
+ "human": 0.000747887218607434,
+ "chat": 0.011979166666666667,
+ "base": 0.00109375,
+ "chatLift": 16.01734375
+ },
+ {
+ "rule": "lex.vibrant",
+ "texts": 310,
+ "human": 0.001869718046518585,
+ "chat": 0.0284375,
+ "base": 0.00140625,
+ "chatLift": 15.2095125
+ },
+ {
+ "rule": "lex.bustling",
+ "texts": 80,
+ "human": 0.0005235210530252038,
+ "chat": 0.007395833333333333,
+ "base": 0.00015625,
+ "chatLift": 14.127098214285715
+ },
+ {
+ "rule": "syn.superficial-ing",
+ "texts": 488,
+ "human": 0.003215915040011966,
+ "chat": 0.045208333333333336,
+ "base": 0.00140625,
+ "chatLift": 14.057688953488373
+ },
+ {
+ "rule": "lex.showcase",
+ "texts": 446,
+ "human": 0.0032907037618727097,
+ "chat": 0.0396875,
+ "base": 0.003125,
+ "chatLift": 12.060490056818182
+ },
+ {
+ "rule": "lex.intricate",
+ "texts": 275,
+ "human": 0.0024680278214045323,
+ "chat": 0.023854166666666666,
+ "base": 0.00171875,
+ "chatLift": 9.66527462121212
+ },
+ {
+ "rule": "rhet.in-realm",
+ "texts": 178,
+ "human": 0.0017949293246578416,
+ "chat": 0.0140625,
+ "base": 0.0021875,
+ "chatLift": 7.8345703125
+ },
+ {
+ "rule": "lex.meticulous",
+ "texts": 141,
+ "human": 0.0016453518809363549,
+ "chat": 0.012291666666666666,
+ "base": 0.00015625,
+ "chatLift": 7.470539772727272
+ },
+ {
+ "rule": "lex.embark",
+ "texts": 432,
+ "human": 0.005309999252112781,
+ "chat": 0.035416666666666666,
+ "base": 0.00265625,
+ "chatLift": 6.669806338028169
+ },
+ {
+ "rule": "lex.seamless",
+ "texts": 176,
+ "human": 0.002243661655822302,
+ "chat": 0.014479166666666666,
+ "base": 0.00109375,
+ "chatLift": 6.453364583333333
+ },
+ {
+ "rule": "lex.comprehensive",
+ "texts": 307,
+ "human": 0.00418816842420163,
+ "chat": 0.0225,
+ "base": 0.00453125,
+ "chatLift": 5.372276785714286
+ },
+ {
+ "rule": "lex.profound",
+ "texts": 299,
+ "human": 0.0043377458679231175,
+ "chat": 0.023020833333333334,
+ "base": 0.00265625,
+ "chatLift": 5.307095905172414
+ },
+ {
+ "rule": "lex.unlock",
+ "texts": 88,
+ "human": 0.0013461969934933811,
+ "chat": 0.006354166666666667,
+ "base": 0.00109375,
+ "chatLift": 4.720086805555556
+ },
+ {
+ "rule": "rhet.not-only-but",
+ "texts": 425,
+ "human": 0.006357041358163189,
+ "chat": 0.0275,
+ "base": 0.010625,
+ "chatLift": 4.325911764705882
+ },
+ {
+ "rule": "lex.harness",
+ "texts": 84,
+ "human": 0.001495774437214868,
+ "chat": 0.00625,
+ "base": 0.000625,
+ "chatLift": 4.1784375
+ },
+ {
+ "rule": "lex.realm",
+ "texts": 229,
+ "human": 0.003889013536758657,
+ "chat": 0.01625,
+ "base": 0.00296875,
+ "chatLift": 4.1784375
+ },
+ {
+ "rule": "syn.plays-role",
+ "texts": 51,
+ "human": 0.0009722533841896642,
+ "chat": 0.0036458333333333334,
+ "base": 0.00046875,
+ "chatLift": 3.7498798076923077
+ },
+ {
+ "rule": "lex.leverage",
+ "texts": 401,
+ "human": 0.007329294742352853,
+ "chat": 0.0271875,
+ "base": 0.00578125,
+ "chatLift": 3.7094292091836736
+ },
+ {
+ "rule": "lex.empower",
+ "texts": 100,
+ "human": 0.001869718046518585,
+ "chat": 0.006875,
+ "base": 0.00125,
+ "chatLift": 3.677025
+ },
+ {
+ "rule": "syn.serves-as",
+ "texts": 291,
+ "human": 0.005758731583277242,
+ "chat": 0.02020833333333333,
+ "base": 0.00296875,
+ "chatLift": 3.5091639610389604
+ },
+ {
+ "rule": "lex.crucial",
+ "texts": 511,
+ "human": 0.010395632338643332,
+ "chat": 0.035,
+ "base": 0.0053125,
+ "chatLift": 3.3667985611510796
+ },
+ {
+ "rule": "lex.utilize",
+ "texts": 467,
+ "human": 0.009647745120035899,
+ "chat": 0.029166666666666667,
+ "base": 0.0084375,
+ "chatLift": 3.023158914728682
+ },
+ {
+ "rule": "lex.pivotal",
+ "texts": 84,
+ "human": 0.002094084212100815,
+ "chat": 0.005520833333333333,
+ "base": 0.00046875,
+ "chatLift": 2.636395089285714
+ },
+ {
+ "rule": "lex.truly",
+ "texts": 1024,
+ "human": 0.02355844738613417,
+ "chat": 0.060520833333333336,
+ "base": 0.0175,
+ "chatLift": 2.568965277777778
+ },
+ {
+ "rule": "rhet.in-terms-of",
+ "texts": 627,
+ "human": 0.013237603769351581,
+ "chat": 0.033541666666666664,
+ "base": 0.01703125,
+ "chatLift": 2.53381709039548
+ },
+ {
+ "rule": "lex.foster",
+ "texts": 198,
+ "human": 0.004636900755366091,
+ "chat": 0.011458333333333333,
+ "base": 0.0034375,
+ "chatLift": 2.471118951612903
+ },
+ {
+ "rule": "rhet.weasel-attribution",
+ "texts": 395,
+ "human": 0.010246054894921846,
+ "chat": 0.020729166666666667,
+ "base": 0.00828125,
+ "chatLift": 2.023136405109489
+ },
+ {
+ "rule": "lex.robust",
+ "texts": 351,
+ "human": 0.008750280457706977,
+ "chat": 0.017291666666666667,
+ "base": 0.00859375,
+ "chatLift": 1.976127136752137
+ },
+ {
+ "rule": "lex.furthermore",
+ "texts": 397,
+ "human": 0.010395632338643332,
+ "chat": 0.019479166666666665,
+ "base": 0.01046875,
+ "chatLift": 1.8737837230215826
+ },
+ {
+ "rule": "rhet.rule-of-three",
+ "texts": 3909,
+ "human": 0.107546182035749,
+ "chat": 0.17927083333333332,
+ "base": 0.08703125,
+ "chatLift": 1.6669195497218359
+ },
+ {
+ "rule": "rhet.on-one-hand",
+ "texts": 53,
+ "human": 0.0015705631590756115,
+ "chat": 0.0026041666666666665,
+ "base": 0.0009375,
+ "chatLift": 1.6581101190476188
+ },
+ {
+ "rule": "rhet.when-it-comes",
+ "texts": 155,
+ "human": 0.0043377458679231175,
+ "chat": 0.0069791666666666665,
+ "base": 0.00453125,
+ "chatLift": 1.6089385775862066
+ },
+ {
+ "rule": "stat.burstiness",
+ "texts": 16923,
+ "human": 0.46555979358312766,
+ "chat": 0.7296875,
+ "base": 0.42671875,
+ "chatLift": 1.5673335843373495
+ },
+ {
+ "rule": "lex.facilitate",
+ "texts": 118,
+ "human": 0.0042629571460623735,
+ "chat": 0.005104166666666667,
+ "base": 0.001875,
+ "chatLift": 1.1973300438596493
+ },
+ {
+ "rule": "lex.notably",
+ "texts": 125,
+ "human": 0.004711689477226834,
+ "chat": 0.004375,
+ "base": 0.0025,
+ "chatLift": 0.9285416666666667
+ },
+ {
+ "rule": "lex.importantly",
+ "texts": 120,
+ "human": 0.004038590980480144,
+ "chat": 0.0035416666666666665,
+ "base": 0.00484375,
+ "chatLift": 0.8769560185185183
+ },
+ {
+ "rule": "rhet.the-reality-is",
+ "texts": 67,
+ "human": 0.002617605265126019,
+ "chat": 0.0021875,
+ "base": 0.0015625,
+ "chatLift": 0.8356875
+ },
+ {
+ "rule": "rhet.rhetorical-setup",
+ "texts": 52,
+ "human": 0.0017949293246578416,
+ "chat": 0.0014583333333333334,
+ "base": 0.00203125,
+ "chatLift": 0.8124739583333334
+ },
+ {
+ "rule": "rhet.throat-clearing",
+ "texts": 109,
+ "human": 0.00418816842420163,
+ "chat": 0.0033333333333333335,
+ "base": 0.0028125,
+ "chatLift": 0.7958928571428572
+ },
+ {
+ "rule": "lex.moreover",
+ "texts": 286,
+ "human": 0.011966195497718945,
+ "chat": 0.009270833333333334,
+ "base": 0.00578125,
+ "chatLift": 0.774751953125
+ },
+ {
+ "rule": "lex.just",
+ "texts": 7251,
+ "human": 0.26191010395632336,
+ "chat": 0.17708333333333334,
+ "base": 0.291875,
+ "chatLift": 0.6761225728155341
+ },
+ {
+ "rule": "lex.simply",
+ "texts": 878,
+ "human": 0.0317104180689552,
+ "chat": 0.02,
+ "base": 0.03875,
+ "chatLift": 0.6307075471698114
+ },
+ {
+ "rule": "rhet.in-order-to",
+ "texts": 841,
+ "human": 0.03230872784384115,
+ "chat": 0.018958333333333334,
+ "base": 0.03125,
+ "chatLift": 0.5867867476851851
+ },
+ {
+ "rule": "rhet.in-this-article",
+ "texts": 56,
+ "human": 0.0015705631590756115,
+ "chat": 0.0008333333333333334,
+ "base": 0.00421875,
+ "chatLift": 0.5305952380952381
+ },
+ {
+ "rule": "lex.honestly",
+ "texts": 173,
+ "human": 0.00747887218607434,
+ "chat": 0.0035416666666666665,
+ "base": 0.00578125,
+ "chatLift": 0.47355624999999996
+ },
+ {
+ "rule": "lex.actually",
+ "texts": 1144,
+ "human": 0.04719168349412908,
+ "chat": 0.014479166666666666,
+ "base": 0.056875,
+ "chatLift": 0.30681606576862125
+ },
+ {
+ "rule": "lex.literally",
+ "texts": 230,
+ "human": 0.01278887143818712,
+ "chat": 0.0016666666666666668,
+ "base": 0.0065625,
+ "chatLift": 0.1303216374269006
+ },
+ {
+ "rule": "lex.inevitably",
+ "texts": 57,
+ "human": 0.0032907037618727097,
+ "chat": 0.0004166666666666667,
+ "base": 0.00140625,
+ "chatLift": 0.12661931818181818
+ },
+ {
+ "rule": "rhet.em-dash",
+ "texts": 298,
+ "human": 0.015780420312616856,
+ "chat": 0.0010416666666666667,
+ "base": 0.01171875,
+ "chatLift": 0.0660100710900474
+ }
+ ],
+ "domainRules": {
+ "poetry": [
+ {
+ "rule": "stat.burstiness",
+ "human": 0.2518351214003388,
+ "machine": 0.5186363636363637
+ },
+ {
+ "rule": "lex.beacon",
+ "human": 0.002258610954263128,
+ "machine": 0.06363636363636363
+ },
+ {
+ "rule": "lex.testament",
+ "human": 0.0016939582156973462,
+ "machine": 0.05090909090909091
+ },
+ {
+ "rule": "rhet.rule-of-three",
+ "human": 0.12309429700734048,
+ "machine": 0.16545454545454547
+ },
+ {
+ "rule": "lex.tapestry",
+ "human": 0.000564652738565782,
+ "machine": 0.04090909090909091
+ },
+ {
+ "rule": "lex.vibrant",
+ "human": 0.00564652738565782,
+ "machine": 0.04318181818181818
+ },
+ {
+ "rule": "lex.realm",
+ "human": 0.00564652738565782,
+ "machine": 0.04
+ },
+ {
+ "rule": "lex.truly",
+ "human": 0.033314511575381144,
+ "machine": 0.06727272727272728
+ }
+ ],
+ "reviews": [
+ {
+ "rule": "stat.burstiness",
+ "human": 0.18451749734888653,
+ "machine": 0.7304545454545455
+ },
+ {
+ "rule": "lex.showcase",
+ "human": 0.009544008483563097,
+ "machine": 0.08545454545454545
+ },
+ {
+ "rule": "lex.testament",
+ "human": 0.006362672322375398,
+ "machine": 0.06272727272727273
+ },
+ {
+ "rule": "syn.superficial-ing",
+ "human": 0.0021208907741251328,
+ "machine": 0.057727272727272724
+ },
+ {
+ "rule": "rhet.in-conclusion",
+ "human": 0.006362672322375398,
+ "machine": 0.06136363636363636
+ },
+ {
+ "rule": "syn.testament-to",
+ "human": 0.0042417815482502655,
+ "machine": 0.05818181818181818
+ },
+ {
+ "rule": "lex.delve",
+ "human": 0.006362672322375398,
+ "machine": 0.05818181818181818
+ },
+ {
+ "rule": "lex.nuanced",
+ "human": 0.006362672322375398,
+ "machine": 0.045
+ }
+ ],
+ "recipes": [
+ {
+ "rule": "stat.burstiness",
+ "human": 0.035553047404063204,
+ "machine": 0.07818181818181819
+ },
+ {
+ "rule": "rhet.rule-of-three",
+ "human": 0.22742663656884876,
+ "machine": 0.25363636363636366
+ },
+ {
+ "rule": "lex.simply",
+ "human": 0.01072234762979684,
+ "machine": 0.019545454545454546
+ },
+ {
+ "rule": "rhet.em-dash",
+ "human": 0.000564334085778781,
+ "machine": 0.004545454545454545
+ },
+ {
+ "rule": "lex.showcase",
+ "human": 0.0,
+ "machine": 0.0027272727272727275
+ },
+ {
+ "rule": "fmt.midsentence-bold",
+ "human": 0.0,
+ "machine": 0.0013636363636363637
+ },
+ {
+ "rule": "chat.signoff",
+ "human": 0.000564334085778781,
+ "machine": 0.0018181818181818182
+ },
+ {
+ "rule": "lex.holistic",
+ "human": 0.0,
+ "machine": 0.0009090909090909091
+ }
+ ],
+ "reddit": [
+ {
+ "rule": "stat.burstiness",
+ "human": 0.3468240584598089,
+ "machine": 0.4781818181818182
+ },
+ {
+ "rule": "rhet.rule-of-three",
+ "human": 0.022484541877459248,
+ "machine": 0.04318181818181818
+ },
+ {
+ "rule": "lex.simply",
+ "human": 0.01854974704890388,
+ "machine": 0.038181818181818185
+ },
+ {
+ "rule": "lex.truly",
+ "human": 0.023608768971332208,
+ "machine": 0.03772727272727273
+ },
+ {
+ "rule": "rhet.not-only-but",
+ "human": 0.0011242270938729624,
+ "machine": 0.012272727272727272
+ },
+ {
+ "rule": "lex.crucial",
+ "human": 0.0011242270938729624,
+ "machine": 0.011818181818181818
+ },
+ {
+ "rule": "rhet.in-conclusion",
+ "human": 0.0011242270938729624,
+ "machine": 0.01090909090909091
+ },
+ {
+ "rule": "rhet.when-it-comes",
+ "human": 0.010680157391793142,
+ "machine": 0.019545454545454546
+ }
+ ]
+ },
+ "scoreQuantiles": {
+ "human": {
+ "0.5": 8.6,
+ "0.9": 31.1,
+ "0.95": 36.9,
+ "0.99": 47.52999999999993
+ },
+ "machine": {
+ "0.5": 17.8,
+ "0.9": 48.5,
+ "0.95": 55.0,
+ "0.99": 66.7
+ }
+ }
+}
diff --git a/Docs/Detection/sample.json b/Docs/Detection/sample.json
new file mode 100644
index 0000000..cd19044
--- /dev/null
+++ b/Docs/Detection/sample.json
@@ -0,0 +1,48 @@
+{
+ "dataset": "RAID-train without attacks (train_none.csv)",
+ "datasetUrl": "https://dataset.raid-bench.xyz/train_none.csv",
+ "datasetSha256": "c5467bca6fc7f5c728c676450c7f84ce401df6c6ccc6d82c47e3b5f3c6d6fce4",
+ "seed": 20261117,
+ "machinePerGeneratorAndDomain": 200,
+ "human": 13371,
+ "machine": 17600,
+ "total": 30971,
+ "humanByDomain": {
+ "abstracts": 1766,
+ "books": 1781,
+ "news": 1780,
+ "poetry": 1771,
+ "recipes": 1772,
+ "reddit": 1779,
+ "reviews": 943,
+ "wiki": 1779
+ },
+ "machineByModel": {
+ "chatgpt": 1600,
+ "cohere": 1600,
+ "cohere-chat": 1600,
+ "gpt2": 1600,
+ "gpt3": 1600,
+ "gpt4": 1600,
+ "llama-chat": 1600,
+ "mistral": 1600,
+ "mistral-chat": 1600,
+ "mpt": 1600,
+ "mpt-chat": 1600
+ },
+ "datasetMachineByModel": {
+ "chatgpt": 26742,
+ "cohere": 26742,
+ "cohere-chat": 26742,
+ "gpt2": 53484,
+ "gpt3": 26742,
+ "gpt4": 26742,
+ "llama-chat": 53484,
+ "mistral": 53484,
+ "mistral-chat": 53484,
+ "mpt": 53484,
+ "mpt-chat": 53484
+ },
+ "idFingerprintSha256": "136c179ba9288e53bab98e5086cde93923965863350c0544df1f4f028903db5b",
+ "pandas": "2.2.3"
+}
diff --git a/Docs/PARAPHRASE.md b/Docs/PARAPHRASE.md
index 1c9b9b7..37383f3 100644
--- a/Docs/PARAPHRASE.md
+++ b/Docs/PARAPHRASE.md
@@ -185,7 +185,7 @@ What argues against it is also on this page: the rewrite *introduced* tells as w
## What this does not tell you
- **Nothing about detecting a watermark.** The key is Anthropic's and no third party holds it. This measures what happens to *prose*, which is a different question that happens to be answerable without anybody's permission.
-- **Nothing about how much machine writing this tool catches.** That would need a corpus of machine writing, which `Docs/Calibration/README.md` argues at length against assembling, and the argument has not changed.
+- **Nothing about how much machine writing this tool catches.** This study needs no collection of machine writing, because each passage is its own baseline. That half is measured separately, on a public benchmark nobody here assembled, in `Docs/DETECTION.md`.
- **Nothing about whether rewriting helps or hurts a student.** The study failed to establish a direction. Anyone quoting it in either direction is quoting something that is not here.
- **One model, one instruction, one day**, and the treatment was only partly delivered. All of that is above.
- **These are published articles and encyclopedia entries**, not student essays, and the windows are cut from them rather than composed at that length. A four-hundred-word paragraph somebody wrote as a paragraph has a whole distribution of sentence lengths; a truncation does not.
diff --git a/README.md b/README.md
index fb91452..bfda77e 100644
--- a/README.md
+++ b/README.md
@@ -98,6 +98,15 @@ which is a sample of whichever models were around that month; a false-positive r
writing, and it measures the harm this category actually causes — studies report that detectors flag 61% of essays by
non-native English speakers, and none of them publish that about themselves.
+The other half, how much machine-written text the score catches, is measured separately in
+[`Docs/DETECTION.md`](Docs/DETECTION.md), on RAID, a public benchmark of 2023 models that nobody here
+assembled, with the predictions committed before the run. Most machine-written text there scores too
+low to stand out from human writing, and the score crosses the cutoffs far more often for 2023
+chat-tuned models than for base models: on that benchmark the tells are mostly fingerprints of
+chat-tuning, not a way to tell who wrote a text. The same page found that, in that one sample, the
+boundary did not hold for long news articles, a kind of writing the calibration corpus does not
+contain.
+
The report also names **which rules misfire**, ranked. That list is uncomfortable and it is the most
useful thing the exercise produces.
diff --git a/SKILL.md b/SKILL.md
index 592f15b..618b3a9 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -35,8 +35,8 @@ a measurement into an accusation.
1. **A finding is a fact about the tool, not about the writer.** Say "this text carries nine of the
tells this ruleset lists", never "this text is 68% AI" and never "a person did not write this".
2. **Finding nothing is not evidence a human wrote it.** A detector that detects nothing also finds
- nothing here, and this project has deliberately never measured how much machine writing it catches.
- Report what you found and stop.
+ nothing here, and on a public benchmark most machine-written text also scores too low with this
+ engine to stand out from human writing (`Docs/DETECTION.md`). Report what you found and stop.
3. **If you quote the engine's score, quote its error rate too.** The boundary is the lowest score
whose 95% interval stays under the project's 5% target for writing known to be human: at 30/100
the published build flagged 2 of 296 pre-2022 texts, an observed 0.7% with the interval reaching
diff --git a/src/SignsOfAI.Cli/Program.cs b/src/SignsOfAI.Cli/Program.cs
index 75d6dd7..dcf612e 100644
--- a/src/SignsOfAI.Cli/Program.cs
+++ b/src/SignsOfAI.Cli/Program.cs
@@ -424,8 +424,8 @@ static void PrintReport(string path, AnalysisResult r, int top, bool useColor)
Console.WriteLine(Col($" … and {r.Findings.Count - shown.Count} more (use --top {r.Findings.Count}).", 90));
// Not "reads mostly human", and not a green tick. Both were the same claim in two forms, and it
// is the one #32 removed from the report and the interface because it was never ours to make: a
- // detector that detects nothing also finds nothing, and this project has deliberately never
- // measured how much machine writing it catches. It survived here for months because the test
+ // detector that detects nothing also finds nothing, and on the public benchmark in
+ // Docs/DETECTION.md most machine-written text scores low too. It survived here for months because the test
// that guards the wording reads AnalysisResult.Verdict, and this line is a literal.
if (r.Findings.Count == 0)
Console.WriteLine(Col(
diff --git a/src/SignsOfAI.Core/Reporting/EvidenceReport.cs b/src/SignsOfAI.Core/Reporting/EvidenceReport.cs
index e9426c8..3e5e82e 100644
--- a/src/SignsOfAI.Core/Reporting/EvidenceReport.cs
+++ b/src/SignsOfAI.Core/Reporting/EvidenceReport.cs
@@ -70,8 +70,8 @@ public static string ToMarkdown(AnalysisResult result, ReportOptions? options =
// The verdict is withheld below the threshold this build can support, because printing
// "Reads mostly human" and then, four lines down, "treat the score as saying nothing" is a
// page arguing with itself — and the reader will keep whichever half suits them. A low score
- // is not evidence of a human: a detector that detects nothing also returns zero, and this
- // project deliberately never measured how much machine writing it catches.
+ // is not evidence of a human: a detector that detects nothing also returns zero, and on the
+ // public benchmark in Docs/DETECTION.md most machine-written text scores low too.
if (VerdictHolds(result))
AppendBlock(sb, text, ReportMessages.AnalysisScoreWithVerdict,
Num(result.OverallScore), Verdict(text, result.OverallScore));
diff --git a/src/SignsOfAI.Core/Reporting/ReportMessages.cs b/src/SignsOfAI.Core/Reporting/ReportMessages.cs
index a5f5730..9f65f5b 100644
--- a/src/SignsOfAI.Core/Reporting/ReportMessages.cs
+++ b/src/SignsOfAI.Core/Reporting/ReportMessages.cs
@@ -46,8 +46,8 @@ public static class ReportMessages
///
/// The report used to say "Reads mostly human" and the interface "Minimal signs of AI writing" —
/// the same state, two claims, and the first one is not ours to make. A detector that detects
- /// nothing also returns a low score, and this project has deliberately never measured how much
- /// machine writing it catches, so a low score is evidence about the boundary and nothing else.
+ /// nothing also returns a low score, and on the public benchmark in Docs/DETECTION.md most
+ /// machine-written text does too, so a low score is evidence about the boundary and nothing else.
///
public const string VerdictNone = "verdict.none";
public const string AnalysisScoreWithVerdict = "analysis.score.with-verdict";
@@ -301,7 +301,7 @@ public static class ReportMessages
[HowReadInterval] = "Read the interval, not the observed rate. {0} out of {1} is not a false-positive rate you can round down.",
[HowNoisyIntro] = "The rules seen most often on that human writing, worst first — if the evidence above leans on one of these, weigh it accordingly:",
[HowNoisyRule] = "- `{0}` — {1} of human texts",
- [HowLimitation] = "What this does **not** tell you: how much machine-written text it catches. That is the other half of the picture and it is deliberately not measured here, because any collection of machine-written text samples whichever models were convenient that month. A tool that flags nothing has a perfect false-positive rate.",
+ [HowLimitation] = "What this does **not** tell you: how much machine-written text it catches. The corpus behind this report is human writing only. That half was measured separately, on a public benchmark of 2023 models (https://github.com/peopleworks/SignsofAI/blob/main/Docs/DETECTION.md), and most machine-written text there scored too low to stand out from human writing, so a low score is not evidence that a person wrote this. A tool that flags nothing has a perfect false-positive rate.",
};
///
- Cuánta escritura de máquina se le escapa. No está medido, y es a propósito:
- para medirlo habría que reunir textos generados, y esa colección solo representa los modelos
- que estaban de moda ese mes. Una herramienta que no marca nada tiene una tasa de error perfecta.
+ Cuánta escritura de máquina se le escapa. La calibración no lo mide: para eso
+ hace falta una colección de textos generados, y cualquiera envejece con los modelos de su momento.
+ Se midió aparte, en un banco de pruebas público con modelos de 2023 que nadie de aquí reunió, y
+ la mayor parte del texto de máquina también puntúa demasiado bajo para distinguirse de la
+ escritura humana
+ (DETECTION.md).
+ Por eso una puntuación baja no demuestra que lo escribiera una persona. Una herramienta que no
+ marca nada tiene una tasa de error perfecta.
Sí incluye trabajos de estudiantes, y esa es la parte que más le importa a usted.
@@ -284,6 +289,14 @@ Y no es el mismo número para todos
Lo que no le dice
Lo que no le dice
textos en español son revisiones de Wikipedia de una sola fuente. Ahí esta herramienta sabe
menos, y lo dice en vez de prestarle a un idioma una cifra que midió en otro.
+ Tampoco está medido ningún tipo de escritura que el corpus no contenga, y eso incluye el ensayo + escolar corriente. La primera comprobación, con escritura humana larga del mismo banco de pruebas + público, encontró que la frontera no se sostuvo en los artículos de prensa de esa + muestra. Es una sola muestra, y no se midió por qué ocurre. Hasta que un tipo de escritura + esté medido, lea una puntuación alta en él como si no dijera nada + (DETECTION.md). +
- How much machine writing it misses. That is not measured, deliberately: measuring - it would mean collecting generated text, and any such collection represents whichever models were - fashionable that month. A tool that flags nothing has a perfect error rate. + How much machine writing it misses. The calibration does not measure it: that + needs a collection of generated text, and any such collection ages with the models of its month. + It was measured separately, on a public benchmark of 2023 models that nobody here assembled, and + most machine-written text there also scores too low to stand out from human + writing + (DETECTION.md). + That is why a low score is not evidence that a person wrote something. A tool that flags nothing + has a perfect error rate.
It does include student work, and that is the part that matters most to you. @@ -490,6 +508,14 @@
+ Nor is any kind of writing the corpus does not contain, and that includes the ordinary school + essay. The first check of that, on long human writing from the same public benchmark, found that + the boundary did not hold for the news articles in that sample. It is one sample, + and why it happens was not measured. Until a kind of writing has been measured, read a high score + on it as saying nothing + (DETECTION.md). +