diff --git a/Docs/CALIBRATION.md b/Docs/CALIBRATION.md index 868297a..77bc445 100644 --- a/Docs/CALIBRATION.md +++ b/Docs/CALIBRATION.md @@ -89,9 +89,9 @@ Across these groups the median score runs from 9.5 (**en-second-language-learner ## What the product does with this number -The tool speaks at **30/100** and nowhere else, taking the boundary from the table above rather than from anybody's judgement. Below it a document gets its score and the reason it gets nothing more: a low score is not evidence that a person wrote something, since a detector that detects nothing also returns a low score, and this project has deliberately never measured how much machine writing it catches. The boundary moves when this page moves — including upward if a larger corpus turns out to be less flattering. +The tool speaks at **30/100** and nowhere else, taking the boundary from the table above rather than from anybody's judgement. Below it a document gets its score and the reason it gets nothing more: a low score is not evidence that a person wrote something, since a detector that detects nothing also returns a low score, and on the public benchmark in DETECTION.md most machine-written text scores low as well. The boundary moves when this page moves — including upward if a larger corpus turns out to be less flattering. -Above it there is **one** verdict, not a scale of them. This corpus can place a boundary and can say nothing whatever about how much further past it a score has travelled: no text known to be human came close to the upper reaches, and grading "moderate" against "strong" would need machine-written text, which the opening of this page argues against collecting. Interfaces do shade a high score more urgently than a low one, and those shades are a display convention — they are not on this page because nothing measured them. +Above it there is **one** verdict, not a scale of them. This corpus can place a boundary and can say nothing whatever about how much further past it a score has travelled: no text known to be human came close to the upper reaches, and grading "moderate" against "strong" would need machine-written text, which the opening of this page argues against collecting; the benchmark in DETECTION.md was not used to set any band. Interfaces do shade a high score more urgently than a low one, and those shades are a display convention — they are not on this page because nothing measured them. A language absent from the corpus entirely gets no verdict at all, whatever it scores. A language present but too thin to bound its own rate borrows this boundary and carries its own figure beside it, so the reader weighs the rate that was measured for the writing in front of them rather than the pooled one. @@ -131,8 +131,8 @@ A rule near the top is not automatically wrong. Some tells genuinely appear in h ## What this does not tell you -- **Nothing about how much AI writing it catches.** That is the other half of the picture and it is not measured here, deliberately. A tool that flags nothing has a perfect false-positive rate. -- **Nothing about text unlike this corpus.** These are published articles and the essays of adult learners in a university English programme. A first-year essay by a native speaker is a different population again, and the rate on one does not transfer to the other. Calibrating on your own students' pre-2022 work is the fix, and the same tool does it. +- **Nothing about how much AI writing it catches.** That is the other half of the picture and it is not measured here, deliberately. A tool that flags nothing has a perfect false-positive rate. That half is measured separately, on a public benchmark nobody here assembled, in [DETECTION.md](DETECTION.md). +- **Nothing about text unlike this corpus.** These are published articles and the essays of adult learners in a university English programme. A first-year essay by a native speaker is a different population again, and the rate on one does not transfer to the other. Calibrating on your own students' pre-2022 work is the fix, and the same tool does it. The first outside check, on long human writing from a public benchmark, is in [DETECTION.md](DETECTION.md): in that one sample the boundary did not hold for the news articles, and why was not measured. - **The affiliation groups are a proxy, not a fact.** Nobody's first language is recorded in a DOI. The learner group is the exception — its corpus records each writer's first language — which is why it exists. Elsewhere the manifest states the reasoning per text so it can be argued with; where it is wrong, the number moves. - **Hashes prove what *this* run measured**, not that another person extracting the same articles would get identical text. They would not: PDF and HTML extraction differ. Reproducing this needs the extracted texts, not just the manifest. diff --git a/Docs/Calibration/README.md b/Docs/Calibration/README.md index 7ca1325..83fed9a 100644 --- a/Docs/Calibration/README.md +++ b/Docs/Calibration/README.md @@ -18,6 +18,12 @@ A **false-positive rate** needs only writing known to be human. It does not go s the harm this category actually causes: detectors flag 61% of essays by non-native English speakers, and not one of them publishes that figure about itself. +The other half, how much machine-written text the score catches, is measured in `Docs/DETECTION.md` +on RAID, a public benchmark nobody here assembled. It answers one of the two objections above: nobody +here assembled it. It does not answer the other: its models are from 2023, and the number ages. It +reports a detection rate beside this false-positive rate rather than an accuracy figure that blends +the two. + ## What makes a text admissible **It was written before 2022.** That is the whole test. A paper with a 2019 DOI predates the models diff --git a/Docs/DETECTION.md b/Docs/DETECTION.md new file mode 100644 index 0000000..c44a1be --- /dev/null +++ b/Docs/DETECTION.md @@ -0,0 +1,219 @@ +# How well the score tells machine-written text from human writing + + + +[CALIBRATION.md](CALIBRATION.md) measures how often this tool is wrong about a person, and says plainly what it leaves out: nothing about how much AI writing it catches, because a tool that flags nothing has a perfect false-positive rate. This page measures that other half. + +The calibration page also says why it avoids an accuracy figure: any collection of machine-written text ages, and flatters whoever assembled it. So this page uses a collection nobody here assembled, **RAID** (Dugan et al., ACL 2024, [arXiv 2405.07940](https://arxiv.org/abs/2405.07940)), which its authors describe as the largest benchmark for detectors of machine-generated text, and it states the date. RAID's generators are from 2023. The tells this tool looks for shift with every model, so this half of the picture ages and the human half does not. + +## If you are a teacher, read this first + +- **A low score tells you nothing.** In this benchmark about three in four machine-written texts (73.4%) scored too low to stand out from human writing. +- **None of the detection rates further down is what the product does.** They are counted at cutoffs the benchmark chose, on texts mostly under 400 words. The product gives no verdict under 649 words, so on this benchmark it would have given its verdict on no machine-written text at all, and on 14 texts written by people. +- **For texts of 649 words or more, where the product does give a verdict, how much machine writing it finds has not been measured.** No machine text in this sample is that long. So for a long text that crosses the boundary, this page cannot tell you how likely it is to be machine-written. On this benchmark every long text that crossed was written by a person. +- **The boundary was checked on long human writing it had never been measured on, and it did not hold for news articles in this sample**: 9 of 108 crossed it, about one in twelve, where the promise is at most one in twenty. The calibration corpus holds research articles, encyclopedia entries and essays by adult learners of English at one university. Anything else is unmeasured, and that includes ordinary secondary-school essays. Until a kind of writing is measured, read a high score on it as saying nothing. +- **A score above a cutoff is not a catch.** At the benchmark's cutoffs (not the product's), in a class of 30 where 3 texts were machine-written, you would expect fewer than one of the 3 above the cutoff, and one or two of the 27 honest ones. That is arithmetic from 26.6% and 5%, not a classroom measurement. +- **The machine text here is from 2023.** "ChatGPT" on this page is the 2023 model. Nothing here describes what a student can use today. +- **What the tool does show, at any length, is which tells a text carries and where.** That is a starting point for a conversation with the writer, not a finding about them. + +Words used below. *Chat-tuned* means trained to answer in conversation, like ChatGPT; a *base* model only continues a text. *AUROC* runs from 0.5 (the score cannot tell the two apart) to 1 (it always can). *Above the cutoff* means only that a text scored above it; it does not mean anyone was identified. + +## What was measured + +- **Benchmark** RAID-train without attacks (`train_none.csv`, SHA-256 `c5467bca6fc7f5c7`), evaluated with raid-bench 0.2.0 +- **Sample** 30,971 texts: all 13,371 human texts, and 200 machine texts for each of 11 generators in each of 8 domains (17,600); seed 20261117, id fingerprint `136c179ba9288e53` +- **Engine** SignsOfAI.Core 0.8.1 at `7043e66`, rule packs unchanged, language set to English +- **Run** 2026-11-17 +- **Counted by** RAID's own `run_evaluation`, at a 5% and a 1% false-positive rate + +**How RAID counts.** It takes the raw score and finds, separately for each domain, the cutoff that flags 5% of its human texts. Then it counts how much machine text lands above that cutoff. The score is this tool's 0–100 divided by 100, because RAID's search assumes a 0–1 range; dividing by a constant changes no ranking. So the product's own boundary and its length floor play no part here, and the cutoffs RAID finds are not this tool's. + +## The headline + +- **26.6%** of the machine-written texts score above RAID's cutoffs at a 5% false-positive rate (12.9% at 1%; AUROC 0.63). Weighted to RAID's own mix of generators instead of this sample's equal one: 25.3%. +- **Chat-tuned generators 37.5%, base models 12.6%.** ChatGPT alone 59.6%. The four base models sit between 10.1% and 15.8%, two to three times the 5% a score that knew nothing would reach, while their AUROC is near 0.5: more of their text than of human text scores high, and across the rest of the range that advantage is cancelled. Where was not measured. +- **As shipped, the product would have given its verdict on none of the machine-written texts in this sample, and on 14 texts written by people.** No machine-written text in the sample reaches 649 words, the shortest text the published false-positive rate was measured on. Below that length the product shows the score and the tells and withholds its verdict (#59). +- **Found, not predicted:** on long human writing from registers the calibration never held, 14 of 705 texts cross the product's boundary, 9 of them news articles. Details under *Long human writing* below. + +This is below the entries submitted by others at the top of RAID's leaderboard and below each of the open detectors in the scale table further down, with the caveats given there. On this benchmark the tells are mostly fingerprints of chat-tuning. They are evidence about how a text reads, not a way to tell who wrote it. + +## What we predicted, and what happened + +Written and committed before any RAID text was scored ([prediction.md](Detection/prediction.md), commit `e007adc`). Each line is judged by the refutation criterion it was written with. + +| # | Prediction | Measured | Result | +|---|---|---|---| +| P1 | Chat-tuned rate at least twice the base-model rate | chat 37.5%, base 12.6%, ratio 2.97 | held | +| P2 | Base-model rate under 20% | 12.6% | held | +| P3 | text-davinci-003 between the two groups | 16.6% | held | +| P4 | Poetry and recipes not among the three best domains | best three: film reviews 50.9%, poetry 36.7%, book summaries 33.7% | **failed** | +| P5 | Scanner reports at least 99% of homoglyph and zero-width attacks, with false alarms on at most 1% of human texts | homoglyph 99.7%, zero-width 100.0%; any artifact on 0.84% of unattacked human texts (0.70–1.01%) | held on the point estimate; the interval touches 1% | +| P6 | The homoglyph attack lowers the score | mean 20.6 → 20.1; above the cutoff 24.6% → 23.7% | held, weakly (−0.53 points) | +| — | Between 15% and 40% overall, without attacks | 26.6% | held | + +**P4 failed.** One observation that may explain it, found afterwards: `stat.burstiness` fires on 52% of machine poems and 25% of human ones. Recipes were as weak as predicted (12.1%). + +**P6 held, weakly.** The score dropped by 0.53 points on average across the machine texts, through the mechanism P6 named (see *Attacks*). The note added the same day predicted no drop; it was right about the zero-width attack and wrong about homoglyphs. Between them the prediction and the note covered both outcomes, so this line was not a test that could fail. + +At the single cutoff every line holds or fails as above (P4's best three become arXiv abstracts, poetry and news articles): chat-tuned 34.3%, base 10.2%, text-davinci-003 10.9%, so P3 holds there by 0.7 points. + +## By generator + +| Generator | Kind | Above the cutoff at 5% | At 1% | AUROC | +|---|---|---|---|---| +| ChatGPT | chat-tuned | 59.6% | 37.4% | 0.82 | +| GPT-4 | chat-tuned | 43.5% | 24.4% | 0.78 | +| Llama 2 70B Chat | chat-tuned | 36.2% | 16.7% | 0.73 | +| Mistral 7B Chat | chat-tuned | 35.6% | 14.9% | 0.73 | +| MPT-30B Chat | chat-tuned | 28.6% | 12.1% | 0.66 | +| Cohere Chat | chat-tuned | 21.5% | 7.7% | 0.63 | +| GPT-3 (text-davinci-003) | instruction-tuned | 16.6% | 4.8% | 0.57 | +| GPT-2 XL | base | 15.8% | 8.9% | 0.50 | +| Mistral 7B | base | 12.4% | 5.6% | 0.50 | +| Cohere | base | 12.2% | 3.7% | 0.56 | +| MPT-30B | base | 10.1% | 5.7% | 0.44 | + +An AUROC below 0.5 (MPT-30B) means a text from that generator scores lower than a human text more often than it scores higher. + +## By domain + +| Domain | Above the cutoff at 5% | AUROC | Lowest score above RAID's cutoff (not the product's 30) | +|---|---|---|---| +| film reviews | 50.9% | 0.78 | 23.8 | +| poetry | 36.7% | 0.67 | 34.6 | +| book summaries | 33.7% | 0.71 | 34.7 | +| news articles | 22.2% | 0.61 | 40.6 | +| arXiv abstracts | 22.0% | 0.64 | 44.7 | +| Wikipedia | 19.9% | 0.67 | 34.5 | +| Reddit posts | 14.8% | 0.55 | 31.0 | +| recipes | 12.1% | 0.53 | 8.9 | + +The engine scores to one decimal and RAID's cutoffs fall between those steps, so the last column gives the lowest score each cutoff counts rather than the cutoff rounded: rounding can carry a whole tie across it. The exact cutoffs are in `Docs/Detection/results.json`. + +Greedy decoding scores above the cutoff more often than sampling (30.4% against 22.7%, across all 11 generators). Only 6 generators were run both with and without a repetition penalty; across those, texts generated with it are above the cutoff 18.1% of the time and texts without it 28.4%. These are observed differences between settings, not controlled experiments, and what causes them was not measured. + +With one cutoff for every domain, closer to how the product works, the rate is 23.4% (lowest score counted 37.0). + +## Found along the way + +None of what follows was predicted. It is reported because it is true, and it should be read as observation rather than as a test that could have failed. + +### The product's boundary on short text + +At the product's boundary of 30/100, ignoring the length floor, 11.3% of RAID's human texts score above it, against 33.2% of the machine ones. The floor was not set from this number: it is the shortest text the calibration measured (#59). This is the first outside look at what lies under it. The human rate at the boundary falls as texts get longer, as the table shows, but this sample does not separate length from register. + +| Length | Human texts | Machine texts | Human above 30 (no verdict under 649 words) | Machine above 30 (same) | Human above RAID's cutoff | Machine above RAID's cutoff | +|---|---|---|---|---|---|---| +| under 200 words | 4,990 | 7,800 | 14.5% | 29.7% | 5.8% | 23.0% | +| 200-399 words | 6,463 | 8,875 | 10.7% | 35.5% | 5.6% | 29.2% | +| 400-648 words | 1,210 | 925 | 6.7% | 40.2% | 1.0% | 31.7% | +| 649 or more words | 708 | 0 | 2.0% | n/a | 0.1% | n/a | + +Only 5.3% of the human texts and none of the machine texts reach 649 words. + +The only rates that describe the product as shipped are the published 0.7% (2 of 296) and, on this benchmark's long human texts, 2.0% (14 of 705). Every other percentage on this page is counted at a cutoff the product does not use, or on texts it gives no verdict on. + +### Long human writing the calibration does not contain + +705 distinct human texts in the sample reach the floor (708 rows: RAID repeats a few texts verbatim, and here each is counted once). They are an outside sample of long human writing, but of a different population from the calibration corpus, which holds research articles, encyclopedia revisions and learner essays. These are news articles, book summaries, film reviews and verse. At the product's boundary of 30/100: + +| Domain | Texts | Crossing the boundary | 95% interval | `stat.burstiness` fires on | +|---|---|---|---|---| +| news articles | 108 | 9 | 4.4–15.1% | 74% | +| book summaries | 322 | 3 | 0.3–2.7% | 43% | +| poetry | 108 | 2 | 0.5–6.5% | 13% | +| recipes | 16 | 0 | 0.0–19.4% | 0% | +| film reviews | 151 | 0 | 0.0–2.5% | 3% | +| **all** | **705** | **14** | **1.2–3.3%** | **34%** | + +Overall that is 2.0% (14 of 705), against the published 0.7% (2 of 296, 0.2–2.4%): about three times the rate, a difference this sample is too small to establish. Without the news articles it is 5 of 597 (0.8%). + +**9 of 108 news articles cross the boundary** (8.3%, interval 4.4–15.1%). That interval lies wholly above the published one and reaches three times the 5% target: by the rule that set the boundary, 30/100 is not supported for news. News was singled out after seeing the table, and a domain chosen that way overstates itself, so this needs a second sample before it is quoted as a rate. + +All 14 texts that cross carry `stat.burstiness`. The rule fires on 74% of the long news articles, crossing or not, against 34% of all the long human texts here and 32% of the calibration corpus. News articles trip the sentence-rhythm rule far more often than the writing the boundary was measured on; why was not measured. + +The published rate was measured with engine 0.7.1 (2026-09-07) and this sample with 0.8.1. Re-running the calibration with 0.8.1 changed only its engine and date fields ([calibration-rerun.json](Detection/calibration-rerun.json)). + +This does not move the published number, which describes the corpus it was measured on. Any change to it goes through the calibration corpus and the review committee, not through RAID. What it does say: until a kind of writing has been measured, a score above 30 on it should be read as saying nothing, and that includes every kind absent from the calibration corpus, not only news. The fix is the one the calibration page gives: run the calibration on your own students' pre-2022 work (`Docs/Calibration/README.md`). If you have no such work, you have no boundary for your classroom. + +### Which rules carry the signal + +Share of texts each rule fires on, for the rules that fire most often on chat-tuned text relative to human text (at least 50 texts): + +| Rule | Human | Chat-tuned | Base | Chat ÷ human | +|---|---|---|---|---| +| `syn.testament-to` | 0.05% | 3.35% | 0.11% | 64× | +| `lex.delve` | 0.13% | 4.96% | 0.09% | 39× | +| `lex.tapestry` | 0.04% | 1.31% | 0.03% | 29× | +| `lex.transformative` | 0.02% | 0.64% | 0.05% | 28× | +| `lex.underscore` | 0.02% | 0.57% | 0.03% | 26× | +| `lex.beacon` | 0.07% | 1.57% | 0.11% | 21× | +| `lex.testament` | 0.19% | 3.97% | 0.17% | 20× | +| `rhet.in-conclusion` | 0.14% | 2.82% | 0.23% | 20× | +| `lex.nuanced` | 0.07% | 1.20% | 0.11% | 16× | +| `lex.vibrant` | 0.19% | 2.84% | 0.14% | 15× | +| `lex.bustling` | 0.05% | 0.74% | 0.02% | 14× | +| `syn.superficial-ing` | 0.32% | 4.52% | 0.14% | 14× | + +Each of them is rare: strong evidence where it appears, absent from most texts. The rule with the largest absolute gap is `stat.burstiness`, which fires on 46.6% of human texts, 73.0% of chat-tuned ones and 42.7% of base-model ones. No rule was removed and re-scored, so how much of the separation each rule carries was not measured. + +Some rules fire more often on human text than on RAID's chat-tuned output: `lex.actually` (4.7% of human texts, 1.4% of chat-tuned), `lex.literally` (1.3% of human texts, 0.2% of chat-tuned), `rhet.em-dash` (1.6% of human texts, 0.1% of chat-tuned). On RAID these rules point the wrong way. Whether they separate later models was not measured here, and no rule was changed in response. + +### Attacks + +Eight of RAID's eleven attacks need no language model. They were run, at the strengths RAID's own code uses, on 1,000 human and 1,000 machine texts drawn from the sample. Paraphrase, synonym swap and perplexity misspelling need a model and were left out, and paraphrase is the attack most likely to hurt a detector built on word choice. What follows is robustness to cheap attacks, not to rewriting. + +| Attack | Strength | Above the cutoff before | After | Mean score | Scanner reports a strong artifact | +|---|---|---|---|---|---| +| alternative spelling | 1.0 | 24.6% | 24.1% | 20.6 → 20.4 | 0.2% | +| article deletion | 0.5 | 24.6% | 24.7% | 20.6 → 20.6 | 0.2% | +| homoglyph | 1.0 | 24.6% | 23.7% | 20.6 → 20.1 | 99.7% | +| insert paragraphs | 0.5 | 24.6% | 24.6% | 20.6 → 20.6 | 0.2% | +| number | 0.5 | 24.6% | 24.6% | 20.6 → 20.6 | 0.2% | +| upper lower | 0.05 | 24.6% | 24.6% | 20.6 → 20.6 | 0.2% | +| whitespace | 0.2 | 24.6% | 23.8% | 20.6 → 20.1 | 0.2% | +| zero width space | 1.0 | 24.6% | 24.6% | 20.6 → 20.6 | 100.0% | + +Counting only characters that typing does not produce, the scanner reports 99.7% of the homoglyph-attacked machine texts and 99.8% of the human ones, and 100.0% and 100.0% under the zero-width attack: it describes the file, not who wrote it. On unattacked human texts it reports some artifact in 0.84%, a strong one in 0.03%, and a systematic pattern in 0.00%. The 5 homoglyph texts it did not report (3 machine, 2 human) are listed in `attacks.json` with a reason computed from the text: 3 are written in another script, where those letters are the alphabet, and 2 have no Latin letter left to disguise. + +**What the homoglyph attack still moves.** The scanner counts a disguised word only when Latin letters are at least half of it, so that a Greek or Cyrillic word quoted in English is not called a disguise. RAID's attack replaces every letter it can. In words such as *crucial*, *intricate* or a capitalised *Moreover* the substitutes become the majority. Those words are neither reported nor restored, so the rules that need them miss, while the document as a whole is still reported. Closing this is a change of behaviour, so it was not made here: it needs its own measurement against human text, because a Greek or Cyrillic word is sometimes exactly what the writer meant. + +### For scale: RAID's leaderboard + +Measured on RAID-test, read from the leaderboard's files at commit `84c2ea4`. The files were chosen by hand: the open detectors RAID's authors ran, and the highest entries on 2026-11-17; the list is in `leaderboard-scale.json`, and `fastdetectgpt.json` (no score for the all-domains, no-attack split) was skipped. These are different texts and a different mix: this sample weights the 11 generators equally, while RAID weights generators run with and without a repetition penalty twice (re-weighted to RAID's mix, the rate above is 25.3%). So they set a scale and are not rows in the same table as the numbers above. Nothing from this project has been submitted. + +| Detector, run by RAID's authors | No attack | Homoglyph | Zero-width | Whitespace | +|---|---|---|---|---| +| Binoculars | 79.0% | 36.1% | 98.4% | 68.7% | +| RADAR | 65.6% | 44.8% | 78.4% | 61.1% | +| GLTR | 59.7% | 20.3% | 97.9% | 43.1% | +| RoBERTa (ChatGPT) | 42.5% | 0.0% | 0.0% | 0.2% | +| LLMDet | 33.4% | 39.1% | 22.2% | 4.0% | + +Entries submitted by others sit between 92.4% and 99.9%. No result on this benchmark, theirs or the 26.6% above, says how a detector does on other text. + +Under the zero-width attack Binoculars and GLTR rise to about 98%. RAID sets its cutoff on unattacked human text only (`find_threshold` in RAID's `raid/evaluate.py`), so a detector whose score rises on any text carrying these characters gains detection and is charged nothing for it. Whether that is what happens with these two detectors was not tested here. This tool's scanner reports the attacked human files exactly as often as the machine ones, and keeps the finding out of the score. + +## Where this departs from the plan + +- **All 13,371 human texts** were used instead of 500 per domain. More human text makes the cutoff steadier, and it cannot be tuned in anyone's favour. +- **Language fixed to English**, as the calibration does. The web app detects the language instead and would have misread 0.3% of the human texts and 0.1% of the machine ones. +- **Scores divided by 100** for RAID's threshold search, which assumes a 0–1 range. +- **Both cutoffs reported**: RAID's per-domain cutoff first, then one cutoff for all domains. +- **Repeated texts.** RAID-train's human set holds 13,272 distinct texts in 13,371 rows. The headline counts rows, as RAID does; the long-text check counts each distinct text once. +- **Generator mix.** The sample weights the 11 generators equally; RAID does not. Re-weighted to RAID's proportions the headline is 25.3% instead of 26.6%. + +## What this does not tell you + +- **What the product does on long machine text.** No machine text in the sample reaches 649 words, where the product starts giving verdicts. +- **Anything about current models.** RAID's generators are from 2023. Newer models write differently and the tells move with them. The right response to "a newer model writes differently" is to measure it. +- **A leaderboard result.** This is a sample of RAID-train, not RAID-test, and nothing has been submitted. +- **Anything about students.** RAID's human texts are published writing. The effect on second-language learners (#75) is invisible here, which is what the calibration corpus is for. +- **A cutoff tuned on other text.** RAID chooses each cutoff on the same human texts it then counts against. That is its method, and it favours every detector equally. +- **Robustness to rewriting.** Paraphrase, synonym swap and misspelling were not run. +- **More than one run.** One seed, one sample. Intervals are given only where shown. +- **An independent author.** The same model that wrote the rule packs wrote these scripts and this page. The defence is that the counting is RAID's code, not ours, and that the predictions were committed first. The page was reviewed adversarially before merging; the reviews are not published. + +## Re-run it yourself + +See [tools/raid/README.md](../tools/raid/README.md). In short: download `train_none.csv`, draw the sample with `tools/raid/sample.py`, score it with `signsofai-calibrate raid`, count it with `tools/raid/evaluate.py`, attack it with `tools/raid/attack.py`, and write this page with `tools/raid/report.py`. The id fingerprint above says whether you drew the same sample. diff --git a/Docs/Detection/attacks.json b/Docs/Detection/attacks.json new file mode 100644 index 0000000..b511730 --- /dev/null +++ b/Docs/Detection/attacks.json @@ -0,0 +1,208 @@ +{ + "setup": { + "seed": 20261117, + "perClass": 1000, + "attacks": { + "homoglyph": 1.0, + "zero_width_space": 1.0, + "whitespace": 0.2, + "upper_lower": 0.05, + "number": 0.5, + "article_deletion": 0.5, + "insert_paragraphs": 0.5, + "alternative_spelling": 1.0 + }, + "leftOut": [ + "paraphrase", + "synonym", + "perplexity_misspelling" + ] + }, + "attacks": { + "alternative_spelling": { + "tprBefore": 0.246, + "tprAfter": 0.241, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.398300000000003, + "scanMachine": { + "any": 0.006, + "strong": 0.002, + "systematic": 0.0 + }, + "scanHuman": { + "any": 0.003, + "strong": 0.0, + "systematic": 0.0 + } + }, + "article_deletion": { + "tprBefore": 0.246, + "tprAfter": 0.247, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.583, + "scanMachine": { + "any": 0.006, + "strong": 0.002, + "systematic": 0.0 + }, + "scanHuman": { + "any": 0.003, + "strong": 0.0, + "systematic": 0.0 + } + }, + "homoglyph": { + "tprBefore": 0.246, + "tprAfter": 0.237, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.081599999999998, + "scanMachine": { + "any": 0.997, + "strong": 0.997, + "systematic": 0.996 + }, + "scanHuman": { + "any": 0.998, + "strong": 0.998, + "systematic": 0.997 + } + }, + "insert_paragraphs": { + "tprBefore": 0.246, + "tprAfter": 0.246, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.578800000000005, + "scanMachine": { + "any": 0.005, + "strong": 0.002, + "systematic": 0.0 + }, + "scanHuman": { + "any": 0.003, + "strong": 0.0, + "systematic": 0.0 + } + }, + "number": { + "tprBefore": 0.246, + "tprAfter": 0.246, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.6116, + "scanMachine": { + "any": 0.006, + "strong": 0.002, + "systematic": 0.0 + }, + "scanHuman": { + "any": 0.003, + "strong": 0.0, + "systematic": 0.0 + } + }, + "upper_lower": { + "tprBefore": 0.246, + "tprAfter": 0.246, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.6116, + "scanMachine": { + "any": 0.006, + "strong": 0.002, + "systematic": 0.0 + }, + "scanHuman": { + "any": 0.003, + "strong": 0.0, + "systematic": 0.0 + } + }, + "whitespace": { + "tprBefore": 0.246, + "tprAfter": 0.238, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.140099999999997, + "scanMachine": { + "any": 0.006, + "strong": 0.002, + "systematic": 0.0 + }, + "scanHuman": { + "any": 0.003, + "strong": 0.0, + "systematic": 0.0 + } + }, + "zero_width_space": { + "tprBefore": 0.246, + "tprAfter": 0.246, + "meanScoreBefore": 20.6116, + "meanScoreAfter": 20.6116, + "scanMachine": { + "any": 1.0, + "strong": 1.0, + "systematic": 1.0 + }, + "scanHuman": { + "any": 1.0, + "strong": 1.0, + "systematic": 1.0 + } + } + }, + "unattackedHumanScan": { + "any": 0.00837633684840326, + "strong": 0.0002991548874429736, + "systematic": 0.0 + }, + "unattackedMachineScan": { + "any": 0.004431818181818182, + "strong": 0.0021022727272727274, + "systematic": 0.00011363636363636364 + }, + "predictions": { + "P5": { + "homoglyphStrong": 0.997, + "zeroWidthStrong": 1.0, + "humanAlarms": 112, + "humanTexts": 13371, + "humanAlarmInterval": [ + 0.006966539636313223, + 0.0100685377200619 + ], + "held": true + }, + "P6": { + "scoreDrop": 0.5300000000000011, + "held": true + } + }, + "scannerMisses": { + "homoglyph": [ + { + "id": "1be69639-7772-42e3-a9a4-b7ba487f043a::homoglyph", + "class": "machine", + "reason": "written in another script" + }, + { + "id": "253f18bf-bad3-45b0-bab7-05f178d47b1b::homoglyph", + "class": "human", + "reason": "no Latin letter left" + }, + { + "id": "28726675-3f10-4220-a2f7-09061175a1c3::homoglyph", + "class": "machine", + "reason": "no Latin letter left" + }, + { + "id": "7974922a-02e8-4db9-8483-862dbe870103::homoglyph", + "class": "human", + "reason": "written in another script" + }, + { + "id": "ef0a1b4f-66a0-4d6a-872b-7e415d9cc788::homoglyph", + "class": "machine", + "reason": "written in another script" + } + ], + "zero_width_space": [] + } +} diff --git a/Docs/Detection/calibration-rerun.json b/Docs/Detection/calibration-rerun.json new file mode 100644 index 0000000..4c60826 --- /dev/null +++ b/Docs/Detection/calibration-rerun.json @@ -0,0 +1 @@ +{"corpusId":"signsofai-human-baseline","texts":296,"measuredOn":"2026-11-18","engine":"0.8.1","recommendedThreshold":30,"flaggedAtThreshold":2,"rateLow":0.0018549062642001807,"rateHigh":0.024297110573472523,"shortestWords":649,"longestWords":9328,"noisiestRules":[{"ruleId":"lex.just","textShare":0.34797297297297297},{"ruleId":"stat.burstiness","textShare":0.32094594594594594},{"ruleId":"rhet.in-conclusion","textShare":0.30067567567567566},{"ruleId":"rhet.rule-of-three","textShare":0.26013513513513514},{"ruleId":"lex.moreover","textShare":0.22297297297297297},{"ruleId":"rhet.not-only-but","textShare":0.21283783783783783},{"ruleId":"lex.furthermore","textShare":0.1858108108108108},{"ruleId":"lex.crucial","textShare":0.16891891891891891}],"languages":[{"language":"en","texts":271,"recommendedThreshold":30,"bestBound":0.01397699931055571,"rateHighAtThreshold":0.026504623306870336},{"language":"es","texts":25,"recommendedThreshold":null,"bestBound":0.13319225276039096,"rateHighAtThreshold":null}]} diff --git a/Docs/Detection/leaderboard-scale.json b/Docs/Detection/leaderboard-scale.json new file mode 100644 index 0000000..f0c49c8 --- /dev/null +++ b/Docs/Detection/leaderboard-scale.json @@ -0,0 +1,127 @@ +{ + "source": "github.com/liamdugan/raid leaderboard/submissions/*/results.json", + "commit": "84c2ea413752a39def012394a0ce905c8a692b05", + "split": "RAID-test", + "selection": "chosen by hand: the open detectors RAID's authors ran, and the highest entries on 2026-11-17", + "filesRead": [ + "binoculars.json", + "chatgpt-roberta.json", + "desklib.json", + "fastdetectgpt.json", + "gltr.json", + "gptzero.json", + "grammarly.json", + "llmdet.json", + "QuillBot.json", + "radar.json", + "Slop-Or-Not.json" + ], + "skipped": [ + { + "file": "fastdetectgpt.json", + "reason": "no score for the all-domains, no-attack split" + } + ], + "detectors": [ + { + "name": "Grammarly", + "submittedBy": "others", + "tpr05": { + "none": 0.9991360294117647, + "homoglyph": 0.9920588235294118, + "zero_width_space": 0.9668382352941176, + "whitespace": 0.9989889705882353 + } + }, + { + "name": "QuillBot", + "submittedBy": "others", + "tpr05": { + "none": 0.9972058823529412, + "homoglyph": 0.9960845588235294, + "zero_width_space": 0.9925551470588235, + "whitespace": 0.9982904411764706 + } + }, + { + "name": "Slop Or Not", + "submittedBy": "others", + "tpr05": { + "none": 0.9964338235294118, + "homoglyph": 0.9950367647058823, + "zero_width_space": 0.9929227941176471, + "whitespace": 0.9983455882352941 + } + }, + { + "name": "GPTZero", + "submittedBy": "others", + "tpr05": { + "none": 0.9836764705882353, + "homoglyph": 0.9836397058823529, + "zero_width_space": 0.9836580882352941, + "whitespace": 0.9836397058823529 + } + }, + { + "name": "Desklib", + "submittedBy": "others", + "tpr05": { + "none": 0.9238235294117647, + "homoglyph": 0.6915441176470588, + "zero_width_space": 0.4871507352941176, + "whitespace": 0.9238235294117647 + } + }, + { + "name": "Binoculars", + "submittedBy": "RAID authors", + "tpr05": { + "none": 0.7898161764705882, + "homoglyph": 0.36099264705882356, + "zero_width_space": 0.9840808823529412, + "whitespace": 0.6868566176470589 + } + }, + { + "name": "RADAR", + "submittedBy": "RAID authors", + "tpr05": { + "none": 0.6561397058823529, + "homoglyph": 0.44830882352941176, + "zero_width_space": 0.7840073529411765, + "whitespace": 0.6113786764705882 + } + }, + { + "name": "GLTR", + "submittedBy": "RAID authors", + "tpr05": { + "none": 0.5969485294117647, + "homoglyph": 0.2031985294117647, + "zero_width_space": 0.97875, + "whitespace": 0.4305514705882353 + } + }, + { + "name": "RoBERTa (ChatGPT)", + "submittedBy": "RAID authors", + "tpr05": { + "none": 0.42518382352941175, + "homoglyph": 0.00014705882352941175, + "zero_width_space": 0.00016544117647058823, + "whitespace": 0.0015625 + } + }, + { + "name": "LLMDet", + "submittedBy": "RAID authors", + "tpr05": { + "none": 0.3339889705882353, + "homoglyph": 0.39088235294117646, + "zero_width_space": 0.22240808823529412, + "whitespace": 0.04029411764705882 + } + } + ] +} diff --git a/Docs/Detection/prediction.md b/Docs/Detection/prediction.md new file mode 100644 index 0000000..3ba8981 --- /dev/null +++ b/Docs/Detection/prediction.md @@ -0,0 +1,67 @@ +# What we expected before measuring detection on RAID + +Written on **17 November 2026**, before a single RAID text had been scored by this engine, and +committed before the measurement so the order can be checked in the history. The results page that +follows it must report each line below as it stands, including the ones that turn out wrong. + +## Why this exists + +`Docs/CALIBRATION.md` measures how often the tool is wrong about a human, and says plainly what it +leaves out: *nothing about how much AI writing it catches*, because a tool that flags nothing has a +perfect false-positive rate. RAID (Dugan et al., ACL 2024, arXiv 2405.07940) measures that other +half: the share of machine-generated text a detector catches at a fixed false-positive rate on RAID's +own human texts. + +Two things about RAID shape what can be predicted: + +- **RAID picks its own threshold.** It takes the raw score and, by default separately for each + domain, finds the cutoff that flags 5% of its human texts. Our 30/100 boundary and the length floor + from #59 play no part, and the cutoff RAID finds is not ours. +- **RAID's humans are not students.** News, Wikipedia, arXiv abstracts, Reddit, reviews, recipes, + poetry and book summaries. Whatever this measures says nothing about second-language learners (#75), + which is what the calibration is for. + +## The predictions + +RAID's generators fall into three groups. **Chat-tuned:** ChatGPT, GPT-4, Llama 2 70B Chat, Cohere +Chat, MPT-30B Chat, Mistral 7B Chat. **Base models:** GPT-2 XL, MPT-30B, Mistral 7B, Cohere. +**In between:** GPT-3 text-davinci-003, which is instruction-tuned. + +The underlying claim: **the tells this engine looks for are fingerprints of chat-tuning, not of +machine text.** + +| # | Prediction | What would refute it | +|---|---|---| +| P1 | The detection rate on chat-tuned generators is **at least twice** the rate on base models | A ratio below two | +| P2 | On base models the rate stays **under 20%**. A score that knew nothing would catch 5% | 20% or more | +| P3 | text-davinci-003 lands **between** the two groups | Outside that range | +| P4 | **Poetry and recipes**, short and formulaic, are worse than news, Reddit, reviews and books | Poetry or recipes among the three best domains | +| P5 | The character scanner catches **at least 99%** of texts under RAID's homoglyph and zero-width-space attacks, and flags **almost none** of the unattacked human texts | Under 99%, or false alarms on more than 1% of human texts | +| P6 | The homoglyph attack **lowers the score** (disguised words stop matching the rules), and it is exactly the attack the scanner exposes | The score does not drop | + +**Overall guess, held with low confidence:** between **15% and 40%** detection at a 5% false-positive +rate, without attacks. Far below the commercial detectors on the leaderboard, which is consistent with +what this project has said from the start: it cannot tell who wrote a text, and it does not try. + +## Note added later the same day, still before measuring + +Reading `AiWritingAnalyzer.Analyze` while writing the scoring code: the analyzer **normalizes +character artifacts before any rule reads the text**, precisely so that a Cyrillic letter inside +"delve" cannot switch the catalog off. If that works as written, **P6 is wrong**: RAID's homoglyph and +zero-width attacks should leave the score where it was, while the scanner reports them. P6 stays above +exactly as it was written, and the results will say which of the two happened. + +## What is measured, and how + +- A stratified sample of **RAID-train without attacks** (labelled and public): 500 human texts per + domain and 200 machine texts per generator and domain, fixed by a seed and recorded in a manifest. +- Scored by the shipped engine and packs, unchanged, through `tools/SignsOfAI.Calibration`, with the + language set to English as the calibration does. The score is the 0–100 the reader sees, before the + length floor, divided by 100 because RAID's threshold search assumes a 0–1 range. +- Evaluated by **RAID's own `run_evaluation`**, not a reimplementation of it, both per domain (RAID's + default) and with a single threshold across domains (closer to how the product works). +- For attacks, the lightweight ones from RAID's own code are applied to a subsample. The scanner's + catch rate is measured **separately** and never added to the score. + +No rule or threshold will be changed in response to these numbers inside this measurement. A change +suggested by RAID goes through the usual path: measured against the human corpus before it is written. diff --git a/Docs/Detection/results.json b/Docs/Detection/results.json new file mode 100644 index 0000000..6d538a6 --- /dev/null +++ b/Docs/Detection/results.json @@ -0,0 +1,1370 @@ +{ + "engine": "0.8.1", + "raidBench": "0.2.0", + "sample": { + "dataset": "RAID-train without attacks (train_none.csv)", + "datasetUrl": "https://dataset.raid-bench.xyz/train_none.csv", + "datasetSha256": "c5467bca6fc7f5c728c676450c7f84ce401df6c6ccc6d82c47e3b5f3c6d6fce4", + "seed": 20261117, + "machinePerGeneratorAndDomain": 200, + "human": 13371, + "machine": 17600, + "total": 30971, + "humanByDomain": { + "abstracts": 1766, + "books": 1781, + "news": 1780, + "poetry": 1771, + "recipes": 1772, + "reddit": 1779, + "reviews": 943, + "wiki": 1779 + }, + "machineByModel": { + "chatgpt": 1600, + "cohere": 1600, + "cohere-chat": 1600, + "gpt2": 1600, + "gpt3": 1600, + "gpt4": 1600, + "llama-chat": 1600, + "mistral": 1600, + "mistral-chat": 1600, + "mpt": 1600, + "mpt-chat": 1600 + }, + "datasetMachineByModel": { + "chatgpt": 26742, + "cohere": 26742, + "cohere-chat": 26742, + "gpt2": 53484, + "gpt3": 26742, + "gpt4": 26742, + "llama-chat": 53484, + "mistral": 53484, + "mistral-chat": 53484, + "mpt": 53484, + "mpt-chat": 53484 + }, + "idFingerprintSha256": "136c179ba9288e53bab98e5086cde93923965863350c0544df1f4f028903db5b", + "pandas": "2.2.3" + }, + "language": "en", + "detectedNotEnglish": { + "human": 0.003141126318151223, + "machine": 0.0011931818181818183 + }, + "perDomainThreshold": { + "overall": { + "tpr05": 0.26551136363636363, + "tpr01": 0.12886363636363637, + "auroc": 0.6290205694481271 + }, + "thresholds": { + "0.05": { + "news": 0.40500000014628257, + "reddit": 0.3090000000002513, + "reviews": 0.23750112672322377, + "books": 0.34672011422655813, + "recipes": 0.08807745485327312, + "wiki": 0.3442784394322653, + "abstracts": 0.44698202151755373, + "poetry": 0.3454438170525127 + }, + "0.01": { + "news": 0.48229480337078656, + "reddit": 0.4050079749859472, + "reviews": 0.33320425172322377, + "books": 0.44144667672655813, + "recipes": 0.13885870485327312, + "wiki": 0.4380284394322653, + "abstracts": 0.5368257715175537, + "poetry": 0.5876313170525127 + } + }, + "lowestCounted": { + "news": 40.6, + "reddit": 31.0, + "reviews": 23.8, + "books": 34.7, + "recipes": 8.9, + "wiki": 34.5, + "abstracts": 44.7, + "poetry": 34.6 + }, + "trueFpr": { + "0.05": { + "news": 0.047752808988763995, + "reddit": 0.04946599213041036, + "reviews": 0.049840933191940606, + "books": 0.049971925884334634, + "recipes": 0.04966139954853277, + "wiki": 0.050028105677346835, + "abstracts": 0.050396375990939934, + "poetry": 0.0496894409937888 + }, + "0.01": { + "news": 0.009550561797752821, + "reddit": 0.009555930297920234, + "reviews": 0.009544008483563071, + "books": 0.010106681639528325, + "recipes": 0.010158013544018019, + "wiki": 0.009555930297920234, + "abstracts": 0.00962627406568517, + "poetry": 0.009599096555618303 + } + }, + "byModel": { + "chatgpt": { + "tpr05": 0.595625, + "tpr01": 0.37375, + "auroc": 0.8241476656570188, + "n": 1600 + }, + "cohere": { + "tpr05": 0.121875, + "tpr01": 0.036875, + "auroc": 0.5583628748784683, + "n": 1600 + }, + "cohere-chat": { + "tpr05": 0.215, + "tpr01": 0.076875, + "auroc": 0.6281445852965374, + "n": 1600 + }, + "gpt2": { + "tpr05": 0.1575, + "tpr01": 0.089375, + "auroc": 0.501044214157505, + "n": 1600 + }, + "gpt3": { + "tpr05": 0.16625, + "tpr01": 0.0475, + "auroc": 0.5676876495774438, + "n": 1600 + }, + "gpt4": { + "tpr05": 0.435, + "tpr01": 0.24375, + "auroc": 0.779918410178745, + "n": 1600 + }, + "llama-chat": { + "tpr05": 0.361875, + "tpr01": 0.166875, + "auroc": 0.7287554221823348, + "n": 1600 + }, + "mistral": { + "tpr05": 0.124375, + "tpr01": 0.05625, + "auroc": 0.5021589400568394, + "n": 1600 + }, + "mistral-chat": { + "tpr05": 0.35625, + "tpr01": 0.14875, + "auroc": 0.7305786543639219, + "n": 1600 + }, + "mpt": { + "tpr05": 0.100625, + "tpr01": 0.056875, + "auroc": 0.4384858088400269, + "n": 1600 + }, + "mpt-chat": { + "tpr05": 0.28625, + "tpr01": 0.120625, + "auroc": 0.6599420387405579, + "n": 1600 + } + }, + "byDomain": { + "abstracts": { + "tpr05": 0.22, + "tpr01": 0.06409090909090909, + "auroc": 0.6359171985998147 + }, + "books": { + "tpr05": 0.3372727272727273, + "tpr01": 0.18045454545454545, + "auroc": 0.710377979684549 + }, + "news": { + "tpr05": 0.22181818181818183, + "tpr01": 0.09227272727272727, + "auroc": 0.6079744637385086 + }, + "poetry": { + "tpr05": 0.36727272727272725, + "tpr01": 0.15818181818181817, + "auroc": 0.6745738155125506 + }, + "recipes": { + "tpr05": 0.12136363636363637, + "tpr01": 0.046363636363636364, + "auroc": 0.5292014672686229 + }, + "reddit": { + "tpr05": 0.1481818181818182, + "tpr01": 0.06954545454545455, + "auroc": 0.5505960958659104 + }, + "reviews": { + "tpr05": 0.509090909090909, + "tpr01": 0.33545454545454545, + "auroc": 0.7830432372505545 + }, + "wiki": { + "tpr05": 0.1990909090909091, + "tpr01": 0.08454545454545455, + "auroc": 0.6683374214318565 + } + }, + "byDecoding": { + "greedy": { + "tpr05": 0.30430316173952743, + "auroc": 0.6565801727116672 + }, + "sampling": { + "tpr05": 0.2270618848286005, + "auroc": 0.6017041666659263 + } + }, + "byRepetitionPenalty": { + "no": { + "tpr05": 0.2981752398930313, + "auroc": 0.6553431929974791 + }, + "yes": { + "tpr05": 0.18051575931232092, + "auroc": 0.5605257197128714 + } + }, + "byGroup": { + "chat": { + "tpr05": 0.375, + "models": [ + "chatgpt", + "gpt4", + "llama-chat", + "cohere-chat", + "mpt-chat", + "mistral-chat" + ], + "n": 9600 + }, + "base": { + "tpr05": 0.12609375, + "models": [ + "gpt2", + "mpt", + "mistral", + "cohere" + ], + "n": 6400 + }, + "between": { + "tpr05": 0.16625, + "models": [ + "gpt3" + ], + "n": 1600 + } + } + }, + "singleThreshold": { + "overall": { + "tpr05": 0.23375, + "tpr01": 0.10744318181818181, + "auroc": 0.6290205694481271 + }, + "thresholds": { + "0.05": 0.3695574225469299, + "0.01": 0.4769792975469299 + }, + "lowestCounted": 37.0, + "trueFpr": { + "0.05": 0.049884077481115896, + "0.01": 0.009946900007478887 + }, + "byModel": { + "chatgpt": { + "tpr05": 0.55375, + "tpr01": 0.3125, + "auroc": 0.8241476656570188, + "n": 1600 + }, + "cohere": { + "tpr05": 0.094375, + "tpr01": 0.026875, + "auroc": 0.5583628748784683, + "n": 1600 + }, + "cohere-chat": { + "tpr05": 0.17375, + "tpr01": 0.06125, + "auroc": 0.6281445852965374, + "n": 1600 + }, + "gpt2": { + "tpr05": 0.123125, + "tpr01": 0.07625, + "auroc": 0.501044214157505, + "n": 1600 + }, + "gpt3": { + "tpr05": 0.10875, + "tpr01": 0.02, + "auroc": 0.5676876495774438, + "n": 1600 + }, + "gpt4": { + "tpr05": 0.4225, + "tpr01": 0.218125, + "auroc": 0.779918410178745, + "n": 1600 + }, + "llama-chat": { + "tpr05": 0.335625, + "tpr01": 0.16125, + "auroc": 0.7287554221823348, + "n": 1600 + }, + "mistral": { + "tpr05": 0.105625, + "tpr01": 0.0425, + "auroc": 0.5021589400568394, + "n": 1600 + }, + "mistral-chat": { + "tpr05": 0.325625, + "tpr01": 0.128125, + "auroc": 0.7305786543639219, + "n": 1600 + }, + "mpt": { + "tpr05": 0.08375, + "tpr01": 0.04875, + "auroc": 0.4384858088400269, + "n": 1600 + }, + "mpt-chat": { + "tpr05": 0.244375, + "tpr01": 0.08625, + "auroc": 0.6599420387405579, + "n": 1600 + } + }, + "byDomain": { + "abstracts": { + "tpr05": 0.37636363636363634, + "tpr01": 0.16863636363636364, + "auroc": 0.6359171985998147 + }, + "books": { + "tpr05": 0.30636363636363634, + "tpr01": 0.12545454545454546, + "auroc": 0.710377979684549 + }, + "news": { + "tpr05": 0.30727272727272725, + "tpr01": 0.09863636363636363, + "auroc": 0.6079744637385086 + }, + "poetry": { + "tpr05": 0.3481818181818182, + "tpr01": 0.25681818181818183, + "auroc": 0.6745738155125506 + }, + "recipes": { + "tpr05": 0.004545454545454545, + "tpr01": 0.0027272727272727275, + "auroc": 0.5292014672686229 + }, + "reddit": { + "tpr05": 0.09454545454545454, + "tpr01": 0.03954545454545454, + "auroc": 0.5505960958659104 + }, + "reviews": { + "tpr05": 0.27136363636363636, + "tpr01": 0.12090909090909091, + "auroc": 0.7830432372505545 + }, + "wiki": { + "tpr05": 0.16136363636363638, + "tpr01": 0.04681818181818182, + "auroc": 0.6683374214318565 + } + }, + "byDecoding": { + "greedy": { + "tpr05": 0.2690332153863714, + "auroc": 0.6565801727116672 + }, + "sampling": { + "tpr05": 0.1987781423237923, + "auroc": 0.6017041666659263 + } + }, + "byRepetitionPenalty": { + "no": { + "tpr05": 0.26521944313355356, + "auroc": 0.6553431929974791 + }, + "yes": { + "tpr05": 0.1518624641833811, + "auroc": 0.5605257197128714 + } + }, + "byGroup": { + "chat": { + "tpr05": 0.34260416666666665, + "models": [ + "chatgpt", + "gpt4", + "llama-chat", + "cohere-chat", + "mpt-chat", + "mistral-chat" + ], + "n": 9600 + }, + "base": { + "tpr05": 0.10171875, + "models": [ + "gpt2", + "mpt", + "mistral", + "cohere" + ], + "n": 6400 + }, + "between": { + "tpr05": 0.10875, + "models": [ + "gpt3" + ], + "n": 1600 + } + } + }, + "byLength": [ + { + "bucket": "under 200", + "machine": 7800, + "human": 4990, + "tpr05": 0.2296153846153846, + "fpr05": 0.05791583166332665, + "humanAtBoundary": 0.14509018036072144, + "machineAtBoundary": 0.2969230769230769 + }, + { + "bucket": "200-399", + "machine": 8875, + "human": 6463, + "tpr05": 0.2917183098591549, + "fpr05": 0.05585641343029553, + "humanAtBoundary": 0.10676156583629894, + "machineAtBoundary": 0.35538028169014085 + }, + { + "bucket": "400-648", + "machine": 925, + "human": 1210, + "tpr05": 0.31675675675675674, + "fpr05": 0.009917355371900827, + "humanAtBoundary": 0.06694214876033058, + "machineAtBoundary": 0.40216216216216216 + }, + { + "bucket": "649 or more", + "machine": 0, + "human": 708, + "tpr05": null, + "fpr05": 0.0014124293785310734, + "humanAtBoundary": 0.01977401129943503, + "machineAtBoundary": null + } + ], + "matchedRepetitionPenalty": { + "models": [ + "gpt2", + "llama-chat", + "mistral", + "mistral-chat", + "mpt", + "mpt-chat" + ], + "no": { + "tpr05": 0.28362324989393295, + "n": 4714 + }, + "yes": { + "tpr05": 0.18051575931232092, + "n": 4886 + } + }, + "matchedDecoding": { + "models": [ + "chatgpt", + "cohere", + "cohere-chat", + "gpt2", + "gpt3", + "gpt4", + "llama-chat", + "mistral", + "mistral-chat", + "mpt", + "mpt-chat" + ], + "greedy": { + "tpr05": 0.30430316173952743, + "n": 8761 + }, + "sampling": { + "tpr05": 0.2270618848286005, + "n": 8839 + } + }, + "humanTexts": { + "rows": 13371, + "distinct": 13272 + }, + "product": { + "boundary": 30, + "lengthFloorWords": 649, + "ignoringFloor": { + "fpr": 0.11285618128786179, + "tpr": 0.33193181818181816 + }, + "shareAtOrAboveFloor": { + "human": 0.052950415077406325, + "machine": 0.0 + }, + "withFloor": { + "humanFlagged": 0.0010470421060504076, + "machineFlagged": 0.0 + }, + "medianWords": { + "human": 222.0, + "machine": 219.0 + } + }, + "calibration": { + "engine": "0.7.1", + "measuredOn": "2026-09-07", + "burstinessShare": 0.32094594594594594, + "rerun": { + "engine": "0.8.1", + "measuredOn": "2026-11-18", + "fieldsChanged": [ + "engine", + "measuredOn" + ], + "file": "Docs/Detection/calibration-rerun.json" + } + }, + "predictions": [ + { + "id": "P1", + "claim": "chat-tuned rate at least twice the base-model rate", + "measured": "chat 0.375, base 0.126, ratio 2.97", + "held": true + }, + { + "id": "P2", + "claim": "base-model rate under 20%", + "measured": "0.126", + "held": true + }, + { + "id": "P3", + "claim": "text-davinci-003 between the two groups", + "measured": "0.166 (range 0.126-0.375)", + "held": true + }, + { + "id": "P4", + "claim": "poetry and recipes not among the three best domains", + "measured": "best three: reviews 0.509, poetry 0.367, books 0.337", + "held": false + }, + { + "id": "overall", + "claim": "between 15% and 40% overall, without attacks", + "measured": "0.266", + "held": true + } + ], + "predictionsSingleThreshold": [ + { + "id": "P1", + "claim": "chat-tuned rate at least twice the base-model rate", + "measured": "chat 0.343, base 0.102, ratio 3.37", + "held": true + }, + { + "id": "P2", + "claim": "base-model rate under 20%", + "measured": "0.102", + "held": true + }, + { + "id": "P3", + "claim": "text-davinci-003 between the two groups", + "measured": "0.109 (range 0.102-0.343)", + "held": true + }, + { + "id": "P4", + "claim": "poetry and recipes not among the three best domains", + "measured": "best three: abstracts 0.376, poetry 0.348, news 0.307", + "held": false + }, + { + "id": "overall", + "claim": "between 15% and 40% overall, without attacks", + "measured": "0.234", + "held": true + } + ], + "raidMixTpr05": 0.25338235294117645, + "longHumanCheck": { + "rows": 708, + "texts": 705, + "flagged": 14, + "rate": 0.019858156028368795, + "interval": [ + 0.011865314049517352, + 0.03305510994987548 + ], + "byDomain": { + "books": { + "texts": 322, + "flagged": 3, + "interval": [ + 0.0031735030165784012, + 0.027029709659563105 + ], + "burstiness": 0.42857142857142855 + }, + "news": { + "texts": 108, + "flagged": 9, + "interval": [ + 0.044460404358563896, + 0.15082906253293688 + ], + "burstiness": 0.7407407407407407 + }, + "poetry": { + "texts": 108, + "flagged": 2, + "interval": [ + 0.005093224298949841, + 0.06501904855345103 + ], + "burstiness": 0.12962962962962962 + }, + "recipes": { + "texts": 16, + "flagged": 0, + "interval": [ + 0.0, + 0.1936076829974049 + ], + "burstiness": 0.0 + }, + "reviews": { + "texts": 151, + "flagged": 0, + "interval": [ + 0.0, + 0.024808981451414282 + ], + "burstiness": 0.033112582781456956 + } + }, + "rulesInFlagged": { + "stat.burstiness": 14, + "lex.just": 7, + "lex.actually": 4, + "lex.truly": 2, + "lex.simply": 2, + "rhet.weasel-attribution": 2, + "rhet.with-regard-to": 1, + "rhet.rule-of-three": 1, + "rhet.in-order-to": 1, + "lex.notably": 1, + "lex.embark": 1, + "lex.literally": 1, + "lex.testament": 1, + "lex.crucial": 1, + "rhet.in-conclusion": 1, + "lex.profound": 1, + "rhet.not-only-but": 1, + "syn.serves-as": 1 + }, + "burstinessAll": 0.33617021276595743, + "withoutNews": { + "texts": 597, + "flagged": 5 + }, + "published": { + "texts": 296, + "flagged": 2, + "interval": [ + 0.0018549062642001807, + 0.024297110573472523 + ] + } + }, + "rules": [ + { + "rule": "syn.testament-to", + "texts": 340, + "human": 0.0005235210530252038, + "chat": 0.033541666666666664, + "base": 0.00109375, + "chatLift": 64.069375 + }, + { + "rule": "lex.delve", + "texts": 500, + "human": 0.0012714082716326377, + "chat": 0.04958333333333333, + "base": 0.0009375, + "chatLift": 38.99875 + }, + { + "rule": "lex.tapestry", + "texts": 134, + "human": 0.0004487323311644604, + "chat": 0.013125, + "base": 0.0003125, + "chatLift": 29.249062499999997 + }, + { + "rule": "lex.transformative", + "texts": 67, + "human": 0.0002243661655822302, + "chat": 0.006354166666666667, + "base": 0.00046875, + "chatLift": 28.320520833333333 + }, + { + "rule": "lex.underscore", + "texts": 60, + "human": 0.0002243661655822302, + "chat": 0.005729166666666666, + "base": 0.0003125, + "chatLift": 25.53489583333333 + }, + { + "rule": "lex.beacon", + "texts": 169, + "human": 0.000747887218607434, + "chat": 0.015729166666666666, + "base": 0.00109375, + "chatLift": 21.03146875 + }, + { + "rule": "lex.testament", + "texts": 422, + "human": 0.0019445067683793284, + "chat": 0.0396875, + "base": 0.00171875, + "chatLift": 20.410060096153845 + }, + { + "rule": "rhet.in-conclusion", + "texts": 306, + "human": 0.0014209857153541245, + "chat": 0.028229166666666666, + "base": 0.00234375, + "chatLift": 19.865904605263157 + }, + { + "rule": "lex.nuanced", + "texts": 132, + "human": 0.000747887218607434, + "chat": 0.011979166666666667, + "base": 0.00109375, + "chatLift": 16.01734375 + }, + { + "rule": "lex.vibrant", + "texts": 310, + "human": 0.001869718046518585, + "chat": 0.0284375, + "base": 0.00140625, + "chatLift": 15.2095125 + }, + { + "rule": "lex.bustling", + "texts": 80, + "human": 0.0005235210530252038, + "chat": 0.007395833333333333, + "base": 0.00015625, + "chatLift": 14.127098214285715 + }, + { + "rule": "syn.superficial-ing", + "texts": 488, + "human": 0.003215915040011966, + "chat": 0.045208333333333336, + "base": 0.00140625, + "chatLift": 14.057688953488373 + }, + { + "rule": "lex.showcase", + "texts": 446, + "human": 0.0032907037618727097, + "chat": 0.0396875, + "base": 0.003125, + "chatLift": 12.060490056818182 + }, + { + "rule": "lex.intricate", + "texts": 275, + "human": 0.0024680278214045323, + "chat": 0.023854166666666666, + "base": 0.00171875, + "chatLift": 9.66527462121212 + }, + { + "rule": "rhet.in-realm", + "texts": 178, + "human": 0.0017949293246578416, + "chat": 0.0140625, + "base": 0.0021875, + "chatLift": 7.8345703125 + }, + { + "rule": "lex.meticulous", + "texts": 141, + "human": 0.0016453518809363549, + "chat": 0.012291666666666666, + "base": 0.00015625, + "chatLift": 7.470539772727272 + }, + { + "rule": "lex.embark", + "texts": 432, + "human": 0.005309999252112781, + "chat": 0.035416666666666666, + "base": 0.00265625, + "chatLift": 6.669806338028169 + }, + { + "rule": "lex.seamless", + "texts": 176, + "human": 0.002243661655822302, + "chat": 0.014479166666666666, + "base": 0.00109375, + "chatLift": 6.453364583333333 + }, + { + "rule": "lex.comprehensive", + "texts": 307, + "human": 0.00418816842420163, + "chat": 0.0225, + "base": 0.00453125, + "chatLift": 5.372276785714286 + }, + { + "rule": "lex.profound", + "texts": 299, + "human": 0.0043377458679231175, + "chat": 0.023020833333333334, + "base": 0.00265625, + "chatLift": 5.307095905172414 + }, + { + "rule": "lex.unlock", + "texts": 88, + "human": 0.0013461969934933811, + "chat": 0.006354166666666667, + "base": 0.00109375, + "chatLift": 4.720086805555556 + }, + { + "rule": "rhet.not-only-but", + "texts": 425, + "human": 0.006357041358163189, + "chat": 0.0275, + "base": 0.010625, + "chatLift": 4.325911764705882 + }, + { + "rule": "lex.harness", + "texts": 84, + "human": 0.001495774437214868, + "chat": 0.00625, + "base": 0.000625, + "chatLift": 4.1784375 + }, + { + "rule": "lex.realm", + "texts": 229, + "human": 0.003889013536758657, + "chat": 0.01625, + "base": 0.00296875, + "chatLift": 4.1784375 + }, + { + "rule": "syn.plays-role", + "texts": 51, + "human": 0.0009722533841896642, + "chat": 0.0036458333333333334, + "base": 0.00046875, + "chatLift": 3.7498798076923077 + }, + { + "rule": "lex.leverage", + "texts": 401, + "human": 0.007329294742352853, + "chat": 0.0271875, + "base": 0.00578125, + "chatLift": 3.7094292091836736 + }, + { + "rule": "lex.empower", + "texts": 100, + "human": 0.001869718046518585, + "chat": 0.006875, + "base": 0.00125, + "chatLift": 3.677025 + }, + { + "rule": "syn.serves-as", + "texts": 291, + "human": 0.005758731583277242, + "chat": 0.02020833333333333, + "base": 0.00296875, + "chatLift": 3.5091639610389604 + }, + { + "rule": "lex.crucial", + "texts": 511, + "human": 0.010395632338643332, + "chat": 0.035, + "base": 0.0053125, + "chatLift": 3.3667985611510796 + }, + { + "rule": "lex.utilize", + "texts": 467, + "human": 0.009647745120035899, + "chat": 0.029166666666666667, + "base": 0.0084375, + "chatLift": 3.023158914728682 + }, + { + "rule": "lex.pivotal", + "texts": 84, + "human": 0.002094084212100815, + "chat": 0.005520833333333333, + "base": 0.00046875, + "chatLift": 2.636395089285714 + }, + { + "rule": "lex.truly", + "texts": 1024, + "human": 0.02355844738613417, + "chat": 0.060520833333333336, + "base": 0.0175, + "chatLift": 2.568965277777778 + }, + { + "rule": "rhet.in-terms-of", + "texts": 627, + "human": 0.013237603769351581, + "chat": 0.033541666666666664, + "base": 0.01703125, + "chatLift": 2.53381709039548 + }, + { + "rule": "lex.foster", + "texts": 198, + "human": 0.004636900755366091, + "chat": 0.011458333333333333, + "base": 0.0034375, + "chatLift": 2.471118951612903 + }, + { + "rule": "rhet.weasel-attribution", + "texts": 395, + "human": 0.010246054894921846, + "chat": 0.020729166666666667, + "base": 0.00828125, + "chatLift": 2.023136405109489 + }, + { + "rule": "lex.robust", + "texts": 351, + "human": 0.008750280457706977, + "chat": 0.017291666666666667, + "base": 0.00859375, + "chatLift": 1.976127136752137 + }, + { + "rule": "lex.furthermore", + "texts": 397, + "human": 0.010395632338643332, + "chat": 0.019479166666666665, + "base": 0.01046875, + "chatLift": 1.8737837230215826 + }, + { + "rule": "rhet.rule-of-three", + "texts": 3909, + "human": 0.107546182035749, + "chat": 0.17927083333333332, + "base": 0.08703125, + "chatLift": 1.6669195497218359 + }, + { + "rule": "rhet.on-one-hand", + "texts": 53, + "human": 0.0015705631590756115, + "chat": 0.0026041666666666665, + "base": 0.0009375, + "chatLift": 1.6581101190476188 + }, + { + "rule": "rhet.when-it-comes", + "texts": 155, + "human": 0.0043377458679231175, + "chat": 0.0069791666666666665, + "base": 0.00453125, + "chatLift": 1.6089385775862066 + }, + { + "rule": "stat.burstiness", + "texts": 16923, + "human": 0.46555979358312766, + "chat": 0.7296875, + "base": 0.42671875, + "chatLift": 1.5673335843373495 + }, + { + "rule": "lex.facilitate", + "texts": 118, + "human": 0.0042629571460623735, + "chat": 0.005104166666666667, + "base": 0.001875, + "chatLift": 1.1973300438596493 + }, + { + "rule": "lex.notably", + "texts": 125, + "human": 0.004711689477226834, + "chat": 0.004375, + "base": 0.0025, + "chatLift": 0.9285416666666667 + }, + { + "rule": "lex.importantly", + "texts": 120, + "human": 0.004038590980480144, + "chat": 0.0035416666666666665, + "base": 0.00484375, + "chatLift": 0.8769560185185183 + }, + { + "rule": "rhet.the-reality-is", + "texts": 67, + "human": 0.002617605265126019, + "chat": 0.0021875, + "base": 0.0015625, + "chatLift": 0.8356875 + }, + { + "rule": "rhet.rhetorical-setup", + "texts": 52, + "human": 0.0017949293246578416, + "chat": 0.0014583333333333334, + "base": 0.00203125, + "chatLift": 0.8124739583333334 + }, + { + "rule": "rhet.throat-clearing", + "texts": 109, + "human": 0.00418816842420163, + "chat": 0.0033333333333333335, + "base": 0.0028125, + "chatLift": 0.7958928571428572 + }, + { + "rule": "lex.moreover", + "texts": 286, + "human": 0.011966195497718945, + "chat": 0.009270833333333334, + "base": 0.00578125, + "chatLift": 0.774751953125 + }, + { + "rule": "lex.just", + "texts": 7251, + "human": 0.26191010395632336, + "chat": 0.17708333333333334, + "base": 0.291875, + "chatLift": 0.6761225728155341 + }, + { + "rule": "lex.simply", + "texts": 878, + "human": 0.0317104180689552, + "chat": 0.02, + "base": 0.03875, + "chatLift": 0.6307075471698114 + }, + { + "rule": "rhet.in-order-to", + "texts": 841, + "human": 0.03230872784384115, + "chat": 0.018958333333333334, + "base": 0.03125, + "chatLift": 0.5867867476851851 + }, + { + "rule": "rhet.in-this-article", + "texts": 56, + "human": 0.0015705631590756115, + "chat": 0.0008333333333333334, + "base": 0.00421875, + "chatLift": 0.5305952380952381 + }, + { + "rule": "lex.honestly", + "texts": 173, + "human": 0.00747887218607434, + "chat": 0.0035416666666666665, + "base": 0.00578125, + "chatLift": 0.47355624999999996 + }, + { + "rule": "lex.actually", + "texts": 1144, + "human": 0.04719168349412908, + "chat": 0.014479166666666666, + "base": 0.056875, + "chatLift": 0.30681606576862125 + }, + { + "rule": "lex.literally", + "texts": 230, + "human": 0.01278887143818712, + "chat": 0.0016666666666666668, + "base": 0.0065625, + "chatLift": 0.1303216374269006 + }, + { + "rule": "lex.inevitably", + "texts": 57, + "human": 0.0032907037618727097, + "chat": 0.0004166666666666667, + "base": 0.00140625, + "chatLift": 0.12661931818181818 + }, + { + "rule": "rhet.em-dash", + "texts": 298, + "human": 0.015780420312616856, + "chat": 0.0010416666666666667, + "base": 0.01171875, + "chatLift": 0.0660100710900474 + } + ], + "domainRules": { + "poetry": [ + { + "rule": "stat.burstiness", + "human": 0.2518351214003388, + "machine": 0.5186363636363637 + }, + { + "rule": "lex.beacon", + "human": 0.002258610954263128, + "machine": 0.06363636363636363 + }, + { + "rule": "lex.testament", + "human": 0.0016939582156973462, + "machine": 0.05090909090909091 + }, + { + "rule": "rhet.rule-of-three", + "human": 0.12309429700734048, + "machine": 0.16545454545454547 + }, + { + "rule": "lex.tapestry", + "human": 0.000564652738565782, + "machine": 0.04090909090909091 + }, + { + "rule": "lex.vibrant", + "human": 0.00564652738565782, + "machine": 0.04318181818181818 + }, + { + "rule": "lex.realm", + "human": 0.00564652738565782, + "machine": 0.04 + }, + { + "rule": "lex.truly", + "human": 0.033314511575381144, + "machine": 0.06727272727272728 + } + ], + "reviews": [ + { + "rule": "stat.burstiness", + "human": 0.18451749734888653, + "machine": 0.7304545454545455 + }, + { + "rule": "lex.showcase", + "human": 0.009544008483563097, + "machine": 0.08545454545454545 + }, + { + "rule": "lex.testament", + "human": 0.006362672322375398, + "machine": 0.06272727272727273 + }, + { + "rule": "syn.superficial-ing", + "human": 0.0021208907741251328, + "machine": 0.057727272727272724 + }, + { + "rule": "rhet.in-conclusion", + "human": 0.006362672322375398, + "machine": 0.06136363636363636 + }, + { + "rule": "syn.testament-to", + "human": 0.0042417815482502655, + "machine": 0.05818181818181818 + }, + { + "rule": "lex.delve", + "human": 0.006362672322375398, + "machine": 0.05818181818181818 + }, + { + "rule": "lex.nuanced", + "human": 0.006362672322375398, + "machine": 0.045 + } + ], + "recipes": [ + { + "rule": "stat.burstiness", + "human": 0.035553047404063204, + "machine": 0.07818181818181819 + }, + { + "rule": "rhet.rule-of-three", + "human": 0.22742663656884876, + "machine": 0.25363636363636366 + }, + { + "rule": "lex.simply", + "human": 0.01072234762979684, + "machine": 0.019545454545454546 + }, + { + "rule": "rhet.em-dash", + "human": 0.000564334085778781, + "machine": 0.004545454545454545 + }, + { + "rule": "lex.showcase", + "human": 0.0, + "machine": 0.0027272727272727275 + }, + { + "rule": "fmt.midsentence-bold", + "human": 0.0, + "machine": 0.0013636363636363637 + }, + { + "rule": "chat.signoff", + "human": 0.000564334085778781, + "machine": 0.0018181818181818182 + }, + { + "rule": "lex.holistic", + "human": 0.0, + "machine": 0.0009090909090909091 + } + ], + "reddit": [ + { + "rule": "stat.burstiness", + "human": 0.3468240584598089, + "machine": 0.4781818181818182 + }, + { + "rule": "rhet.rule-of-three", + "human": 0.022484541877459248, + "machine": 0.04318181818181818 + }, + { + "rule": "lex.simply", + "human": 0.01854974704890388, + "machine": 0.038181818181818185 + }, + { + "rule": "lex.truly", + "human": 0.023608768971332208, + "machine": 0.03772727272727273 + }, + { + "rule": "rhet.not-only-but", + "human": 0.0011242270938729624, + "machine": 0.012272727272727272 + }, + { + "rule": "lex.crucial", + "human": 0.0011242270938729624, + "machine": 0.011818181818181818 + }, + { + "rule": "rhet.in-conclusion", + "human": 0.0011242270938729624, + "machine": 0.01090909090909091 + }, + { + "rule": "rhet.when-it-comes", + "human": 0.010680157391793142, + "machine": 0.019545454545454546 + } + ] + }, + "scoreQuantiles": { + "human": { + "0.5": 8.6, + "0.9": 31.1, + "0.95": 36.9, + "0.99": 47.52999999999993 + }, + "machine": { + "0.5": 17.8, + "0.9": 48.5, + "0.95": 55.0, + "0.99": 66.7 + } + } +} diff --git a/Docs/Detection/sample.json b/Docs/Detection/sample.json new file mode 100644 index 0000000..cd19044 --- /dev/null +++ b/Docs/Detection/sample.json @@ -0,0 +1,48 @@ +{ + "dataset": "RAID-train without attacks (train_none.csv)", + "datasetUrl": "https://dataset.raid-bench.xyz/train_none.csv", + "datasetSha256": "c5467bca6fc7f5c728c676450c7f84ce401df6c6ccc6d82c47e3b5f3c6d6fce4", + "seed": 20261117, + "machinePerGeneratorAndDomain": 200, + "human": 13371, + "machine": 17600, + "total": 30971, + "humanByDomain": { + "abstracts": 1766, + "books": 1781, + "news": 1780, + "poetry": 1771, + "recipes": 1772, + "reddit": 1779, + "reviews": 943, + "wiki": 1779 + }, + "machineByModel": { + "chatgpt": 1600, + "cohere": 1600, + "cohere-chat": 1600, + "gpt2": 1600, + "gpt3": 1600, + "gpt4": 1600, + "llama-chat": 1600, + "mistral": 1600, + "mistral-chat": 1600, + "mpt": 1600, + "mpt-chat": 1600 + }, + "datasetMachineByModel": { + "chatgpt": 26742, + "cohere": 26742, + "cohere-chat": 26742, + "gpt2": 53484, + "gpt3": 26742, + "gpt4": 26742, + "llama-chat": 53484, + "mistral": 53484, + "mistral-chat": 53484, + "mpt": 53484, + "mpt-chat": 53484 + }, + "idFingerprintSha256": "136c179ba9288e53bab98e5086cde93923965863350c0544df1f4f028903db5b", + "pandas": "2.2.3" +} diff --git a/Docs/PARAPHRASE.md b/Docs/PARAPHRASE.md index 1c9b9b7..37383f3 100644 --- a/Docs/PARAPHRASE.md +++ b/Docs/PARAPHRASE.md @@ -185,7 +185,7 @@ What argues against it is also on this page: the rewrite *introduced* tells as w ## What this does not tell you - **Nothing about detecting a watermark.** The key is Anthropic's and no third party holds it. This measures what happens to *prose*, which is a different question that happens to be answerable without anybody's permission. -- **Nothing about how much machine writing this tool catches.** That would need a corpus of machine writing, which `Docs/Calibration/README.md` argues at length against assembling, and the argument has not changed. +- **Nothing about how much machine writing this tool catches.** This study needs no collection of machine writing, because each passage is its own baseline. That half is measured separately, on a public benchmark nobody here assembled, in `Docs/DETECTION.md`. - **Nothing about whether rewriting helps or hurts a student.** The study failed to establish a direction. Anyone quoting it in either direction is quoting something that is not here. - **One model, one instruction, one day**, and the treatment was only partly delivered. All of that is above. - **These are published articles and encyclopedia entries**, not student essays, and the windows are cut from them rather than composed at that length. A four-hundred-word paragraph somebody wrote as a paragraph has a whole distribution of sentence lengths; a truncation does not. diff --git a/README.md b/README.md index fb91452..bfda77e 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,15 @@ which is a sample of whichever models were around that month; a false-positive r writing, and it measures the harm this category actually causes — studies report that detectors flag 61% of essays by non-native English speakers, and none of them publish that about themselves. +The other half, how much machine-written text the score catches, is measured separately in +[`Docs/DETECTION.md`](Docs/DETECTION.md), on RAID, a public benchmark of 2023 models that nobody here +assembled, with the predictions committed before the run. Most machine-written text there scores too +low to stand out from human writing, and the score crosses the cutoffs far more often for 2023 +chat-tuned models than for base models: on that benchmark the tells are mostly fingerprints of +chat-tuning, not a way to tell who wrote a text. The same page found that, in that one sample, the +boundary did not hold for long news articles, a kind of writing the calibration corpus does not +contain. + The report also names **which rules misfire**, ranked. That list is uncomfortable and it is the most useful thing the exercise produces. diff --git a/SKILL.md b/SKILL.md index 592f15b..618b3a9 100644 --- a/SKILL.md +++ b/SKILL.md @@ -35,8 +35,8 @@ a measurement into an accusation. 1. **A finding is a fact about the tool, not about the writer.** Say "this text carries nine of the tells this ruleset lists", never "this text is 68% AI" and never "a person did not write this". 2. **Finding nothing is not evidence a human wrote it.** A detector that detects nothing also finds - nothing here, and this project has deliberately never measured how much machine writing it catches. - Report what you found and stop. + nothing here, and on a public benchmark most machine-written text also scores too low with this + engine to stand out from human writing (`Docs/DETECTION.md`). Report what you found and stop. 3. **If you quote the engine's score, quote its error rate too.** The boundary is the lowest score whose 95% interval stays under the project's 5% target for writing known to be human: at 30/100 the published build flagged 2 of 296 pre-2022 texts, an observed 0.7% with the interval reaching diff --git a/src/SignsOfAI.Cli/Program.cs b/src/SignsOfAI.Cli/Program.cs index 75d6dd7..dcf612e 100644 --- a/src/SignsOfAI.Cli/Program.cs +++ b/src/SignsOfAI.Cli/Program.cs @@ -424,8 +424,8 @@ static void PrintReport(string path, AnalysisResult r, int top, bool useColor) Console.WriteLine(Col($" … and {r.Findings.Count - shown.Count} more (use --top {r.Findings.Count}).", 90)); // Not "reads mostly human", and not a green tick. Both were the same claim in two forms, and it // is the one #32 removed from the report and the interface because it was never ours to make: a - // detector that detects nothing also finds nothing, and this project has deliberately never - // measured how much machine writing it catches. It survived here for months because the test + // detector that detects nothing also finds nothing, and on the public benchmark in + // Docs/DETECTION.md most machine-written text scores low too. It survived here for months because the test // that guards the wording reads AnalysisResult.Verdict, and this line is a literal. if (r.Findings.Count == 0) Console.WriteLine(Col( diff --git a/src/SignsOfAI.Core/Reporting/EvidenceReport.cs b/src/SignsOfAI.Core/Reporting/EvidenceReport.cs index e9426c8..3e5e82e 100644 --- a/src/SignsOfAI.Core/Reporting/EvidenceReport.cs +++ b/src/SignsOfAI.Core/Reporting/EvidenceReport.cs @@ -70,8 +70,8 @@ public static string ToMarkdown(AnalysisResult result, ReportOptions? options = // The verdict is withheld below the threshold this build can support, because printing // "Reads mostly human" and then, four lines down, "treat the score as saying nothing" is a // page arguing with itself — and the reader will keep whichever half suits them. A low score - // is not evidence of a human: a detector that detects nothing also returns zero, and this - // project deliberately never measured how much machine writing it catches. + // is not evidence of a human: a detector that detects nothing also returns zero, and on the + // public benchmark in Docs/DETECTION.md most machine-written text scores low too. if (VerdictHolds(result)) AppendBlock(sb, text, ReportMessages.AnalysisScoreWithVerdict, Num(result.OverallScore), Verdict(text, result.OverallScore)); diff --git a/src/SignsOfAI.Core/Reporting/ReportMessages.cs b/src/SignsOfAI.Core/Reporting/ReportMessages.cs index a5f5730..9f65f5b 100644 --- a/src/SignsOfAI.Core/Reporting/ReportMessages.cs +++ b/src/SignsOfAI.Core/Reporting/ReportMessages.cs @@ -46,8 +46,8 @@ public static class ReportMessages /// /// The report used to say "Reads mostly human" and the interface "Minimal signs of AI writing" — /// the same state, two claims, and the first one is not ours to make. A detector that detects - /// nothing also returns a low score, and this project has deliberately never measured how much - /// machine writing it catches, so a low score is evidence about the boundary and nothing else. + /// nothing also returns a low score, and on the public benchmark in Docs/DETECTION.md most + /// machine-written text does too, so a low score is evidence about the boundary and nothing else. /// public const string VerdictNone = "verdict.none"; public const string AnalysisScoreWithVerdict = "analysis.score.with-verdict"; @@ -301,7 +301,7 @@ public static class ReportMessages [HowReadInterval] = "Read the interval, not the observed rate. {0} out of {1} is not a false-positive rate you can round down.", [HowNoisyIntro] = "The rules seen most often on that human writing, worst first — if the evidence above leans on one of these, weigh it accordingly:", [HowNoisyRule] = "- `{0}` — {1} of human texts", - [HowLimitation] = "What this does **not** tell you: how much machine-written text it catches. That is the other half of the picture and it is deliberately not measured here, because any collection of machine-written text samples whichever models were convenient that month. A tool that flags nothing has a perfect false-positive rate.", + [HowLimitation] = "What this does **not** tell you: how much machine-written text it catches. The corpus behind this report is human writing only. That half was measured separately, on a public benchmark of 2023 models (https://github.com/peopleworks/SignsofAI/blob/main/Docs/DETECTION.md), and most machine-written text there scored too low to stand out from human writing, so a low score is not evidence that a person wrote this. A tool that flags nothing has a perfect false-positive rate.", }; /// diff --git a/src/SignsOfAI.Core/Reporting/report.en.json b/src/SignsOfAI.Core/Reporting/report.en.json index f1d6942..2241f94 100644 --- a/src/SignsOfAI.Core/Reporting/report.en.json +++ b/src/SignsOfAI.Core/Reporting/report.en.json @@ -230,7 +230,7 @@ "text": "- `{0}` — {1} of human texts" }, "how.limitation": { - "text": "What this does **not** tell you: how much machine-written text it catches. That is the other half of the picture and it is deliberately not measured here, because any collection of machine-written text samples whichever models were convenient that month. A tool that flags nothing has a perfect false-positive rate." + "text": "What this does **not** tell you: how much machine-written text it catches. The corpus behind this report is human writing only. That half was measured separately, on a public benchmark of 2023 models (https://github.com/peopleworks/SignsofAI/blob/main/Docs/DETECTION.md), and most machine-written text there scored too low to stand out from human writing, so a low score is not evidence that a person wrote this. A tool that flags nothing has a perfect false-positive rate." } } } diff --git a/src/SignsOfAI.Core/Reporting/report.es.json b/src/SignsOfAI.Core/Reporting/report.es.json index 29161df..70db5f1 100644 --- a/src/SignsOfAI.Core/Reporting/report.es.json +++ b/src/SignsOfAI.Core/Reporting/report.es.json @@ -193,8 +193,8 @@ "sourceHash": "0fd44e4f3d48c7f8a7e2ee3f588ce6dda6e8dd83dd99db3ea2c47531fc1a77a8" }, "how.limitation": { - "text": "Lo que esto **no** indica es cuánto texto escrito por máquinas detecta. Esa es la otra mitad del problema y aquí no se mide deliberadamente, porque cualquier colección de textos escritos por máquinas sólo representa los modelos que resultaron convenientes ese mes. Una herramienta que no marca nada tiene una tasa de falsos positivos perfecta.", - "sourceHash": "28e614ad0734da232d17fbf1e7a826e040184990935189aacd8aadfde88951a2" + "text": "Lo que esto **no** indica es cuánto texto escrito por máquinas detecta. El corpus en que se basa este informe es solo de escritura humana. Esa mitad se midió aparte, en un banco de pruebas público con modelos de 2023 (https://github.com/peopleworks/SignsofAI/blob/main/Docs/DETECTION.md), y allí la mayor parte del texto escrito por máquinas también puntuó demasiado bajo para distinguirse de la escritura humana, así que una puntuación baja no demuestra que esto lo escribiera una persona. Una herramienta que no marca nada tiene una tasa de falsos positivos perfecta.", + "sourceHash": "d9343cbd74be7eab7cf931609e9ebbab3e9abb1ae393e7e316faa0ed10c52cff" }, "how.noisy-intro": { "text": "Las reglas que aparecieron con mayor frecuencia en esos textos humanos, de peor a mejor; si la evidencia anterior depende de alguna, sopésela en consecuencia:", diff --git a/src/SignsOfAI.Web/wwwroot/why.html b/src/SignsOfAI.Web/wwwroot/why.html index 30c23dd..7958947 100644 --- a/src/SignsOfAI.Web/wwwroot/why.html +++ b/src/SignsOfAI.Web/wwwroot/why.html @@ -267,9 +267,14 @@

Y no es el mismo número para todos

Lo que no le dice

- Cuánta escritura de máquina se le escapa. No está medido, y es a propósito: - para medirlo habría que reunir textos generados, y esa colección solo representa los modelos - que estaban de moda ese mes. Una herramienta que no marca nada tiene una tasa de error perfecta. + Cuánta escritura de máquina se le escapa. La calibración no lo mide: para eso + hace falta una colección de textos generados, y cualquiera envejece con los modelos de su momento. + Se midió aparte, en un banco de pruebas público con modelos de 2023 que nadie de aquí reunió, y + la mayor parte del texto de máquina también puntúa demasiado bajo para distinguirse de la + escritura humana + (DETECTION.md). + Por eso una puntuación baja no demuestra que lo escribiera una persona. Una herramienta que no + marca nada tiene una tasa de error perfecta.

Sí incluye trabajos de estudiantes, y esa es la parte que más le importa a usted. @@ -284,6 +289,14 @@

Lo que no le dice

textos en español son revisiones de Wikipedia de una sola fuente. Ahí esta herramienta sabe menos, y lo dice en vez de prestarle a un idioma una cifra que midió en otro.

+

+ Tampoco está medido ningún tipo de escritura que el corpus no contenga, y eso incluye el ensayo + escolar corriente. La primera comprobación, con escritura humana larga del mismo banco de pruebas + público, encontró que la frontera no se sostuvo en los artículos de prensa de esa + muestra. Es una sola muestra, y no se midió por qué ocurre. Hasta que un tipo de escritura + esté medido, lea una puntuación alta en él como si no dijera nada + (DETECTION.md). +

Y una que conviene saber antes de mirar ninguna puntuación: las señales que más saltan en escritura humana casi no dicen nada. El ritmo uniforme de las frases aparece en el @@ -473,9 +486,14 @@

And it is not the same number for everybody

What it does not tell you

- How much machine writing it misses. That is not measured, deliberately: measuring - it would mean collecting generated text, and any such collection represents whichever models were - fashionable that month. A tool that flags nothing has a perfect error rate. + How much machine writing it misses. The calibration does not measure it: that + needs a collection of generated text, and any such collection ages with the models of its month. + It was measured separately, on a public benchmark of 2023 models that nobody here assembled, and + most machine-written text there also scores too low to stand out from human + writing + (DETECTION.md). + That is why a low score is not evidence that a person wrote something. A tool that flags nothing + has a perfect error rate.

It does include student work, and that is the part that matters most to you. @@ -490,6 +508,14 @@

What it does not tell you

texts are Wikipedia revisions from a single source. There this tool knows less, and it says so rather than lending one language a figure it measured in another.

+

+ Nor is any kind of writing the corpus does not contain, and that includes the ordinary school + essay. The first check of that, on long human writing from the same public benchmark, found that + the boundary did not hold for the news articles in that sample. It is one sample, + and why it happens was not measured. Until a kind of writing has been measured, read a high score + on it as saying nothing + (DETECTION.md). +

And one worth knowing before you look at any score: the signals that fire most often on human writing say almost nothing. Uniform sentence rhythm appears in 32.1% of the human diff --git a/tests/SignsOfAI.Core.Tests/LocaleFileTests.cs b/tests/SignsOfAI.Core.Tests/LocaleFileTests.cs index 0c4847f..82c92a1 100644 --- a/tests/SignsOfAI.Core.Tests/LocaleFileTests.cs +++ b/tests/SignsOfAI.Core.Tests/LocaleFileTests.cs @@ -149,8 +149,8 @@ public void No_locale_tells_the_reader_a_person_wrote_the_text() Assert.True(problems.Count == 0, "A low score is a fact about this tool, not about who wrote the text: a detector that " + - "detects nothing also returns a low score, and this project has deliberately never " + - "measured how much machine writing it catches. Say what was measured instead — see " + + "detects nothing also returns a low score, and on the public benchmark in " + + "Docs/DETECTION.md most machine-written text does too. Say what was measured instead — see " + "verdict.none. Found:\n " + string.Join("\n ", problems)); } diff --git a/tests/SignsOfAI.Core.Tests/VerdictAgreementTests.cs b/tests/SignsOfAI.Core.Tests/VerdictAgreementTests.cs index 02fedd9..71080ab 100644 --- a/tests/SignsOfAI.Core.Tests/VerdictAgreementTests.cs +++ b/tests/SignsOfAI.Core.Tests/VerdictAgreementTests.cs @@ -141,8 +141,8 @@ public void A_low_score_says_nothing_about_who_wrote_the_text() { // "Reads mostly human" in the report, "Minimal signs of AI writing" in the interface: the // same state, two claims, and the first was never ours to make. A detector that detects - // nothing also returns zero, and this project has deliberately never measured how much - // machine writing it catches. + // nothing also returns zero, and on the public benchmark in Docs/DETECTION.md most + // machine-written text scores low too. var human = new AiWritingAnalyzer().Analyze( "We measured height, weight and age in three hundred participants recruited at two " + "hospitals during the winter of 2011. At one site the effect vanished entirely.", "en"); diff --git a/tools/SignsOfAI.Calibration/ParaphraseReport.cs b/tools/SignsOfAI.Calibration/ParaphraseReport.cs index d07599c..294fc3b 100644 --- a/tools/SignsOfAI.Calibration/ParaphraseReport.cs +++ b/tools/SignsOfAI.Calibration/ParaphraseReport.cs @@ -360,9 +360,10 @@ public static string Render( sb.AppendLine("- **Nothing about detecting a watermark.** The key is Anthropic's and no third " + "party holds it. This measures what happens to *prose*, which is a different " + "question that happens to be answerable without anybody's permission."); - sb.AppendLine("- **Nothing about how much machine writing this tool catches.** That would need " + - "a corpus of machine writing, which `Docs/Calibration/README.md` argues at length " + - "against assembling, and the argument has not changed."); + sb.AppendLine("- **Nothing about how much machine writing this tool catches.** This study needs " + + "no collection of machine writing, because each passage is its own baseline. That " + + "half is measured separately, on a public benchmark nobody here assembled, in " + + "`Docs/DETECTION.md`."); sb.AppendLine("- **Nothing about whether rewriting helps or hurts a student.** The study " + "failed to establish a direction. Anyone quoting it in either direction is " + "quoting something that is not here."); diff --git a/tools/SignsOfAI.Calibration/Program.cs b/tools/SignsOfAI.Calibration/Program.cs index 1fb40f1..f32c1e5 100644 --- a/tools/SignsOfAI.Calibration/Program.cs +++ b/tools/SignsOfAI.Calibration/Program.cs @@ -38,6 +38,8 @@ string humanDir = "", rewrittenDir = ""; int perStratum = 8, targetWords = 400; string paraphrasedBy = "", instructionPath = ""; +string inPath = ""; +int threads = -1; for (int i = 1; i < argv.Count; i++) { @@ -61,6 +63,8 @@ case "--words": targetWords = int.Parse(Next()); break; case "--paraphrased-by": paraphrasedBy = Next(); break; case "--instruction": instructionPath = Next(); break; + case "--in": inPath = Next(); break; + case "--threads": threads = int.Parse(Next()); break; default: Console.Error.WriteLine($"Unknown option '{argv[i]}'."); return 2; @@ -68,12 +72,25 @@ string Next() => ++i < argv.Count ? argv[i] : throw new ArgumentException($"Missing value for {argv[i - 1]}"); } -if (argv[0] is not ("run" or "fetch" or "thresholds" or "excerpt" or "paraphrase")) +if (argv[0] is not ("run" or "fetch" or "thresholds" or "excerpt" or "paraphrase" or "raid")) { Console.Error.WriteLine($"Unknown command '{argv[0]}'. Run --help."); return 2; } +// ── `raid` ─────────────────────────────────────────────────────────────────── +// Scores texts for an outside benchmark and writes down what the engine said. It evaluates nothing: +// the benchmark's own code picks the threshold, so the scorer cannot shape the result. +if (argv[0] == "raid") +{ + if (string.IsNullOrWhiteSpace(inPath) || outPath == "Docs/CALIBRATION.md") + { + Console.Error.WriteLine("raid needs --in and --out ."); + return 2; + } + return Raid.Run(inPath, outPath, threads: threads); +} + // ── `fetch` ────────────────────────────────────────────────────────────────── // Adds texts that are human by virtue of their date rather than by anybody's judgement, and appends // them to the manifest. Nothing fetched is redistributed: the texts stay on this machine and the @@ -635,6 +652,8 @@ dotnet run --project tools/SignsOfAI.Calibration -- fetch --source pelic dotnet run --project tools/SignsOfAI.Calibration -- excerpt --per-stratum 8 --words 400 dotnet run --project tools/SignsOfAI.Calibration -- paraphrase --paraphrased-by "" + dotnet run --project tools/SignsOfAI.Calibration -- raid --in texts.jsonl --out scores.jsonl + OPTIONS --manifest Corpus manifest (default: Docs/Calibration/corpus.json) --texts Where the texts live (default: /texts) @@ -654,6 +673,8 @@ a number nobody can reproduce. --rewritten The same passages after a model rewrote them (default: /rewritten) --per-stratum excerpt only: passages to take from each group (default: 8) --words excerpt only: length to cut each passage to (default: 400) + --in raid only: JSON lines of {id, text} to score (--out is required too) + --threads raid only: cap the parallelism; 1 runs in order (default: all cores) --paraphrased-by paraphrase only: the model that did the rewriting. Required — a study of what a model does to prose that cannot name the model has produced a number nobody can reproduce or date. @@ -678,4 +699,10 @@ whichever models were around that month. A false-positive rate needs only human language model rewriting a passage do to it. Each unit is one passage measured as its author wrote it and again after a model rewrote it, so no collection of machine-written text is needed — the baseline is the passage itself. See Docs/Paraphrase/README.md. + + `raid` serves the half the calibration leaves out: how much machine-written text the + score separates, measured on RAID (Dugan et al., ACL 2024), a collection nobody here + assembled. It reads JSON lines of {id, text}, scores each one with the shipped engine + as English, and writes {id, score, ...}. It evaluates nothing: RAID's own code picks + the threshold. See tools/raid/README.md and Docs/DETECTION.md. """); diff --git a/tools/SignsOfAI.Calibration/Raid.cs b/tools/SignsOfAI.Calibration/Raid.cs new file mode 100644 index 0000000..1e69ef2 --- /dev/null +++ b/tools/SignsOfAI.Calibration/Raid.cs @@ -0,0 +1,128 @@ +using System.Text.Json; +using System.Text.Json.Serialization; +using SignsOfAI.Core; +using SignsOfAI.Core.Artifacts; +using SignsOfAI.Core.Text; + +namespace SignsOfAI.Calibration; + +/// +/// Scores a batch of texts for an outside benchmark, RAID (Dugan et al., ACL 2024), and nothing else. +/// +/// The rest of this tool measures the half of the question that needs only human writing: how often +/// the engine is wrong about a person. This verb serves the other half, how much machine-written text +/// the score separates, and it deliberately does no evaluating of its own. The benchmark's own code +/// chooses the threshold and counts the hits, so the number cannot be shaped by whoever wrote the +/// scorer. All this does is run the shipped engine, unchanged, and write down what it said. +/// +public static class Raid +{ + private static readonly JsonSerializerOptions Json = new() + { + PropertyNamingPolicy = JsonNamingPolicy.CamelCase, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, + }; + + public sealed record Input(string Id, string Text); + + public sealed record Output + { + public required string Id { get; init; } + + /// The 0–100 the reader sees, before the length floor. RAID gets this divided by 100. + public required double Score { get; init; } + + public required int Words { get; init; } + + /// Findings above the human rate: the ones that move the score. + public required int Signals { get; init; } + + /// Which rules those were, so a result can be explained rule by rule. + public required IReadOnlyList Rules { get; init; } + + /// What the product would have guessed, recorded because the scoring itself is told "en". + public required string DetectedLanguage { get; init; } + + /// + /// The character scan, reported apart from the score and never folded into it. Three levels so + /// the results can be read at each one rather than at whichever flatters the scanner. + /// + public required int Artifacts { get; init; } + public required int StrongArtifacts { get; init; } + public required string ArtifactPattern { get; init; } + } + + /// Reads JSON lines of {id, text} and writes one scored line per input, in order. + public static int Run(string inPath, string outPath, string language = "en", int threads = -1) + { + if (!File.Exists(inPath)) + { + Console.Error.WriteLine($"Input not found: {inPath}"); + return 2; + } + + var inputs = new List(); + foreach (var line in File.ReadLines(inPath)) + { + if (string.IsNullOrWhiteSpace(line)) continue; + var input = JsonSerializer.Deserialize(line, Json) + ?? throw new InvalidDataException($"Unreadable line in {inPath}"); + inputs.Add(input); + } + + var ids = new HashSet(StringComparer.Ordinal); + foreach (var input in inputs) + { + if (!ids.Add(input.Id)) + { + // RAID joins predictions to its rows by id; a duplicate would be scored twice and + // silently count double in whichever cell it falls into. + Console.Error.WriteLine($"Duplicate id in input: {input.Id}"); + return 2; + } + } + + Console.WriteLine($"Scoring {inputs.Count:N0} texts as '{language}'…"); + var started = DateTime.UtcNow; + + var outputs = new Output[inputs.Count]; + using var analyzer = new ThreadLocal(() => new AiWritingAnalyzer()); + // One analyzer per thread. --threads 1 exists to prove the parallel run gives the same answers. + var options = new ParallelOptions { MaxDegreeOfParallelism = threads }; + Parallel.For(0, inputs.Count, options, i => outputs[i] = Score(analyzer.Value!, inputs[i], language)); + + Directory.CreateDirectory(Path.GetDirectoryName(Path.GetFullPath(outPath))!); + using (var writer = new StreamWriter(outPath, append: false, new System.Text.UTF8Encoding(false))) + { + foreach (var output in outputs) + writer.WriteLine(JsonSerializer.Serialize(output, Json)); + } + + var elapsed = DateTime.UtcNow - started; + Console.WriteLine($" {inputs.Count:N0} scored in {elapsed.TotalSeconds:N1} s → {outPath}"); + return 0; + } + + public static Output Score(AiWritingAnalyzer analyzer, Input input, string language) + { + var text = input.Text ?? string.Empty; + var result = analyzer.Analyze(text, language); + + // The scan the product shows is of the original text, not the cleaned copy the rules read, + // so the attack is seen exactly as a reader's file would carry it. + var scan = result.Artifacts; + + return new Output + { + Id = input.Id, + Score = result.OverallScore, + Words = result.Statistics.WordCount, + Signals = result.Signals.Count, + Rules = [.. result.Signals.Select(f => f.RuleId).Distinct().Order(StringComparer.Ordinal)], + DetectedLanguage = LanguageDetector.Detect(text), + Artifacts = scan.Count, + StrongArtifacts = scan.StrongCount, + ArtifactPattern = scan.Pattern.ToString(), + }; + } +} diff --git a/tools/SignsOfAI.Calibration/Report.cs b/tools/SignsOfAI.Calibration/Report.cs index a1ea46c..b16dda7 100644 --- a/tools/SignsOfAI.Calibration/Report.cs +++ b/tools/SignsOfAI.Calibration/Report.cs @@ -129,7 +129,7 @@ public static string Render(CalibrationResult r, CorpusManifest manifest, string "boundary from the table above rather than from anybody's judgement. Below it a document " + "gets its score and the reason it gets nothing more: a low score is not evidence that a " + "person wrote something, since a detector that detects nothing also returns a low score, " + - "and this project has deliberately never measured how much machine writing it catches. " + + "and on the public benchmark in DETECTION.md most machine-written text scores low as well. " + "The boundary moves when this page moves — including upward if a larger corpus turns out " + "to be less flattering."); sb.AppendLine(); @@ -138,7 +138,7 @@ public static string Render(CalibrationResult r, CorpusManifest manifest, string "and can say nothing whatever about how much further past it a score has travelled: no " + "text known to be human came close to the upper reaches, and grading \"moderate\" against " + "\"strong\" would need machine-written text, which the opening of this page argues against " + - "collecting. Interfaces do shade a high score more urgently than a low one, and those " + + "collecting; the benchmark in DETECTION.md was not used to set any band. Interfaces do shade a high score more urgently than a low one, and those " + "shades are a display convention — they are not on this page because nothing measured " + "them."); sb.AppendLine(); @@ -186,12 +186,15 @@ public static string Render(CalibrationResult r, CorpusManifest manifest, string sb.AppendLine(); sb.AppendLine("- **Nothing about how much AI writing it catches.** That is the other half of the " + "picture and it is not measured here, deliberately. A tool that flags nothing has a " + - "perfect false-positive rate."); + "perfect false-positive rate. That half is measured separately, on a public benchmark " + + "nobody here assembled, in [DETECTION.md](DETECTION.md)."); sb.AppendLine("- **Nothing about text unlike this corpus.** These are published articles and " + "the essays of adult learners in a university English programme. A first-year " + "essay by a native speaker is a different population again, and the rate on one " + "does not transfer to the other. Calibrating on your own students' pre-2022 work is " + - "the fix, and the same tool does it."); + "the fix, and the same tool does it. The first outside check, on long human writing " + + "from a public benchmark, is in [DETECTION.md](DETECTION.md): in that one sample the " + + "boundary did not hold for the news articles, and why was not measured."); sb.AppendLine("- **The affiliation groups are a proxy, not a fact.** Nobody's first language is " + "recorded in a DOI. The learner group is the exception — its corpus records each " + "writer's first language — which is why it exists. Elsewhere the manifest states " + diff --git a/tools/raid/README.md b/tools/raid/README.md new file mode 100644 index 0000000..20eb96f --- /dev/null +++ b/tools/raid/README.md @@ -0,0 +1,107 @@ +# Measuring detection on RAID + +The scripts behind [`Docs/DETECTION.md`](../../Docs/DETECTION.md): how much machine-written text the +score separates, measured on RAID (Dugan et al., ACL 2024), a benchmark nobody here assembled. + +The division of labour is the point. **This repository only produces scores.** RAID's own code, +`run_evaluation` from `raid-bench`, chooses every threshold and counts every hit, so nothing here can +shape the number in the engine's favour. + +## What you need + +- .NET 10, for `tools/SignsOfAI.Calibration`. +- **Python 3.12.** `raid-bench` pins numpy 1.26, which has no wheels for newer Pythons. +- About 1 GB of disk for the dataset. +- Network access: the dataset downloads from `dataset.raid-bench.xyz`, and RAID's + `alternative_spelling` attack fetches a JSON word list from GitHub when it starts. + +Keep the data, RAID's code and your scripts in separate directories, and run Python with `-I` so it +never imports a file from the directory it happens to be in. The layout used for the published run: + +``` +temp/raid/data/ the downloaded CSV +temp/raid/src/ RAID's repository (sparse: raid/ and generation/adversarial/) +temp/raid/venv/ Python 3.12 with tools/raid/requirements.txt +temp/raid/out/ everything this produces +``` + +`temp/` is ignored by git. + +## The run, step by step + +```sh +# 0. Environment +uv venv temp/raid/venv --python 3.12 +git clone --depth 1 --filter=blob:none --sparse https://github.com/liamdugan/raid.git temp/raid/src +git -C temp/raid/src sparse-checkout set raid generation/adversarial +VIRTUAL_ENV=temp/raid/venv uv pip install -e temp/raid/src -r tools/raid/requirements.txt +PY=temp/raid/venv/Scripts/python.exe # bin/python outside Windows + +# 1. Data: RAID-train without attacks (802 MB) +curl -L -o temp/raid/data/train_none.csv https://dataset.raid-bench.xyz/train_none.csv + +# 2. The sample: every human text, 200 machine texts per generator and domain, seed 20261117 +$PY -I tools/raid/sample.py --data temp/raid/data/train_none.csv --out temp/raid/out + +# 3. Scores from the shipped engine, unchanged +dotnet run --project tools/SignsOfAI.Calibration -c Release -- \ + raid --in temp/raid/out/sample.jsonl --out temp/raid/out/scores.jsonl + +# 4. The published calibration re-run with this engine, so the page can say what moved. +# `run` rewrites the engine's embedded JSON: copy it out, then restore it. +dotnet run --project tools/SignsOfAI.Calibration -c Release -- run --out temp/raid/cal-check.md +cp src/SignsOfAI.Core/Calibration/published-calibration.json Docs/Detection/calibration-rerun.json +git checkout -- src/SignsOfAI.Core/Calibration/published-calibration.json + +# 5. RAID's evaluation, plus the breakdowns +$PY -I tools/raid/evaluate.py --sample temp/raid/out/sample.csv --scores temp/raid/out/scores.jsonl \ + --texts temp/raid/out/sample.jsonl --manifest temp/raid/out/sample.json \ + --calibration src/SignsOfAI.Core/Calibration/published-calibration.json \ + --calibration-rerun Docs/Detection/calibration-rerun.json \ + --engine 0.8.1 --out Docs/Detection/results.json + +# 6. Attacks: RAID's own attack code, the eight that need no language model +$PY -I tools/raid/attack.py make --sample-csv temp/raid/out/sample.csv \ + --sample-jsonl temp/raid/out/sample.jsonl --adversarial temp/raid/src/generation/adversarial \ + --nltk-data temp/raid/nltk_data --out temp/raid/out/attacks +dotnet run --project tools/SignsOfAI.Calibration -c Release -- \ + raid --in temp/raid/out/attacks/attacked.jsonl --out temp/raid/out/attacks/scores.jsonl +$PY -I tools/raid/attack.py evaluate --attacked-csv temp/raid/out/attacks/attacked.csv \ + --attacked-scores temp/raid/out/attacks/scores.jsonl --sample-csv temp/raid/out/sample.csv \ + --scores temp/raid/out/scores.jsonl --results Docs/Detection/results.json \ + --setup temp/raid/out/attacks/attacked.json --attacked-jsonl temp/raid/out/attacks/attacked.jsonl \ + --adversarial temp/raid/src/generation/adversarial --out Docs/Detection/attacks.json + +# 7. For scale: the leaderboard's own numbers, from leaderboard/submissions//results.json +$PY -I tools/raid/leaderboard.py --dir --commit \ + --out Docs/Detection/leaderboard-scale.json + +# 8. The page +cp temp/raid/out/sample.json Docs/Detection/sample.json +$PY -I tools/raid/report.py --commit --run-date +``` + +Step 2 prints an id fingerprint. If yours matches the one in `Docs/Detection/sample.json`, you scored +the same texts. + +## Things that are easy to get wrong + +- **Scores go to RAID divided by 100.** RAID's threshold search starts at the mean of the human scores + and moves in steps of 0.5, which assumes a 0–1 range. On 0–100 it never reaches the cutoff, gives up + after 50 steps, and discards the domain. +- **RAID tunes a cutoff per domain by default** (`per_domain_tuning=True`). `evaluate.py` reports that + first, as the leaderboard does, and one cutoff for every domain second, which is closer to the + product. +- **RAID's `attack.py` cannot be imported without a GPU stack**: it imports every attacker at the top, + including the three that load a model. `attack.py` here imports the eight light ones one by one and + reads their strengths from RAID's `attack.py` source, so they cannot drift from what RAID uses. +- **The language is fixed to English**, as the calibration does. The web app would detect it instead. +- `--threads 1` on the `raid` verb scores in order. It exists to check that the parallel run gives + identical output, which it did. + +## What not to do with the result + +Do not change a rule or a threshold because of a RAID number inside the same measurement: that is +training on the test. A rule RAID suggests goes through the usual path, measured against the human +corpus before it is written. And nothing is submitted to the RAID leaderboard without the review +committee and the maintainer's decision: a merged `results.json` there stays public. diff --git a/tools/raid/attack.py b/tools/raid/attack.py new file mode 100644 index 0000000..8604519 --- /dev/null +++ b/tools/raid/attack.py @@ -0,0 +1,226 @@ +"""RAID's cheap attacks, applied to a subsample, and what they do to the score and to the scanner. + + make draws a seeded subsample from the RAID sample and attacks it with RAID's own attack code + evaluate reads the engine's scores for the attacked texts and reports two things kept apart: + how detection moves under each attack, and how often the character scanner reports it + +Only the attacks that need no language model are run. RAID's `attack.py` imports every attacker at +the top, including the ones that load a model, so the attackers are imported one by one and their +strength is read from `attack.py` itself rather than copied here, where it could drift. +""" + +import argparse +import importlib +import json +import random +import re +import sys +from pathlib import Path + +import pandas as pd + +# The attackers RAID ships that run without a model or a GPU. paraphrase, synonym and +# perplexity_misspelling need one and are left out; the results page says so. +LIGHT = { + "homoglyph": ("homoglyph", "HomoglyphAttack"), + "zero_width_space": ("zero_width_space", "ZeroWidthSpaceAttack"), + "whitespace": ("whitespace", "WhiteSpaceAttack"), + "upper_lower": ("upper_lower", "UpperLowerFlipAttack"), + "number": ("alter_number", "AlterNumbersAttack"), + "article_deletion": ("article_deletion", "ArticleDeletionAttack"), + "insert_paragraphs": ("insert_paragraphs", "InsertParagraphsAttack"), + "alternative_spelling": ("alternative_spelling", "AlternativeSpellingAttack"), +} + + +def strengths(attack_py: Path) -> dict: + """{ClassName: N} exactly as RAID's get_attack() constructs each attacker.""" + source = attack_py.read_text(encoding="utf-8") + return {cls: float(n) for cls, n in re.findall(r"return (\w+)\(([\d.]+)\)", source)} + + +def build(adversarial_dir: Path) -> dict: + sys.path.insert(0, str(adversarial_dir)) + n = strengths(adversarial_dir / "attack.py") + attackers = {} + for name, (module, cls) in LIGHT.items(): + mod = importlib.import_module(f"attackers.{module}") + attackers[name] = (getattr(mod, cls)(n[cls]), n[cls]) + return attackers + + +def make(args) -> None: + import nltk + + nltk_dir = Path(args.nltk_data) + nltk_dir.mkdir(parents=True, exist_ok=True) + nltk.data.path.insert(0, str(nltk_dir)) + for pkg in ("punkt", "punkt_tab"): + nltk.download(pkg, download_dir=str(nltk_dir), quiet=True) + + random.seed(args.seed) + meta = pd.read_csv(args.sample_csv) + texts = {} + with open(args.sample_jsonl, encoding="utf-8") as f: + for line in f: + row = json.loads(line) + texts[row["id"]] = row["text"] + + human = meta[meta["model"] == "human"].sample(n=args.per_class, random_state=args.seed) + machine = meta[meta["model"] != "human"].sample(n=args.per_class, random_state=args.seed) + base = pd.concat([human, machine]).sort_values("id") + + attackers = build(Path(args.adversarial)) + out = Path(args.out) + out.mkdir(parents=True, exist_ok=True) + rows = [] + with open(out / "attacked.jsonl", "w", encoding="utf-8", newline="\n") as f: + for name, (attacker, n) in attackers.items(): + for r in base.itertuples(index=False): + attacked = attacker.attack(texts[r.id])["generation"] + aid = f"{r.id}::{name}" + f.write(json.dumps({"id": aid, "text": attacked}, ensure_ascii=False) + "\n") + rows.append({"id": aid, "source": r.id, "model": r.model, "decoding": r.decoding, + "repetition_penalty": r.repetition_penalty, "domain": r.domain, "attack": name}) + pd.DataFrame(rows).to_csv(out / "attacked.csv", index=False) + (out / "attacked.json").write_text(json.dumps({ + "seed": args.seed, + "perClass": args.per_class, + "attacks": {k: v[1] for k, v in attackers.items()}, + "leftOut": ["paraphrase", "synonym", "perplexity_misspelling"], + }, indent=2) + "\n", encoding="utf-8") + print(f"{len(rows)} attacked texts from {len(base)} originals, {len(attackers)} attacks -> {out}") + + +def evaluate(args) -> None: + results = json.loads(Path(args.results).read_text(encoding="utf-8")) + thresholds = results["perDomainThreshold"]["thresholds"]["0.05"] + + meta = pd.read_csv(args.attacked_csv) + scores = pd.read_json(args.attacked_scores, lines=True) + if len(scores) != len(meta) or set(scores["id"]) != set(meta["id"]): + raise SystemExit("The attacked scores do not cover exactly the attacked ids.") + df = meta.merge(scores, on="id", how="left", validate="one_to_one") + sample = pd.read_csv(args.sample_csv) + original_scores = pd.read_json(args.scores, lines=True) + if len(original_scores) != len(sample) or set(original_scores["id"]) != set(sample["id"]): + raise SystemExit("The original scores do not cover exactly the sampled ids.") + original = sample.merge(original_scores, on="id", how="left", validate="one_to_one") + for frame, name in ((df, "attacked"), (original, "original")): + if frame["score"].isnull().any(): + raise SystemExit(f"{frame['score'].isnull().sum()} {name} rows have no score.") + if not set(df["source"]).issubset(set(original["id"])): + raise SystemExit("Some attacked texts have no original to compare with.") + by_id = original.set_index("id") + + def flagged(frame): + return (frame["score"] / 100.0) >= frame["domain"].map(thresholds) + + def scan(frame): + return { + "any": float((frame["artifacts"] > 0).mean()), + "strong": float((frame["strongArtifacts"] > 0).mean()), + "systematic": float((frame["artifactPattern"] == "Systematic").mean()), + } + + setup = json.loads(Path(args.setup).read_text(encoding="utf-8")) + report = {"setup": setup, "attacks": {}, "unattackedHumanScan": scan(original[original["model"] == "human"]), + "unattackedMachineScan": scan(original[original["model"] != "human"])} + for name, sub in df.groupby("attack"): + machine = sub[sub["model"] != "human"] + human = sub[sub["model"] == "human"] + before = by_id.loc[machine["source"]] + report["attacks"][name] = { + # Paired: the same machine texts, before and after the attack, at RAID's thresholds. + "tprBefore": float(flagged(before.reset_index()).mean()), + "tprAfter": float(flagged(machine).mean()), + "meanScoreBefore": float(before["score"].mean()), + "meanScoreAfter": float(machine["score"].mean()), + "scanMachine": scan(machine), + "scanHuman": scan(human), + } + def wilson(k, n, z=1.959964): + p = k / n + d = 1 + z * z / n + c = (p + z * z / (2 * n)) / d + m = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5) / d + return [max(0.0, c - m), min(1.0, c + m)] + + humans = original[original["model"] == "human"] + alarms = int((humans["artifacts"] > 0).sum()) + hg, zw = report["attacks"]["homoglyph"], report["attacks"]["zero_width_space"] + report["predictions"] = { + "P5": { + "homoglyphStrong": hg["scanMachine"]["strong"], + "zeroWidthStrong": zw["scanMachine"]["strong"], + "humanAlarms": alarms, "humanTexts": int(len(humans)), + "humanAlarmInterval": wilson(alarms, int(len(humans))), + "held": hg["scanMachine"]["strong"] >= 0.99 and zw["scanMachine"]["strong"] >= 0.99 + and alarms / len(humans) <= 0.01, + }, + "P6": {"scoreDrop": hg["meanScoreBefore"] - hg["meanScoreAfter"], + "held": hg["meanScoreAfter"] < hg["meanScoreBefore"]}, + } + # Which attacked texts the scanner did not report, by class, with a reason computed from the text: + # a letter that imitates nothing means the text is written in another script; no Latin letter at + # all means there was no Latin word left to disguise. + sys.path.insert(0, str(Path(args.adversarial))) + lookalikes = set() + for targets in importlib.import_module("attackers.homoglyph").HomoglyphAttack().mapping.values(): + lookalikes.update(targets) + texts = {} + with open(args.attacked_jsonl, encoding="utf-8") as f: + for line in f: + row = json.loads(line) + texts[row["id"]] = row["text"] + + def reason(text): + foreign = any(c.isalpha() and not c.isascii() and c not in lookalikes for c in text) + latin = sum(c.isascii() and c.isalpha() for c in text) + if foreign: + return "written in another script" + return "no Latin letter left" if latin == 0 else "unexplained" + + report["scannerMisses"] = { + name: [{"id": r.id, "class": "human" if r.model == "human" else "machine", "reason": reason(texts[r.id])} + for r in sub.loc[sub["strongArtifacts"] == 0].sort_values("id").itertuples(index=False)] + for name, sub in df.groupby("attack") if name in ("homoglyph", "zero_width_space") + } + Path(args.out).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + print(f"{'attack':<22} {'TPR before':>10} {'after':>7} {'score':>13} {'scan any/strong/sys (machine)':>32}") + for name, a in report["attacks"].items(): + s = a["scanMachine"] + print(f"{name:<22} {a['tprBefore']:>10.3f} {a['tprAfter']:>7.3f} {a['meanScoreBefore']:>6.1f}->{a['meanScoreAfter']:<5.1f} " + f"{s['any']:>10.3f} {s['strong']:>7.3f} {s['systematic']:>7.3f}") + h = report["unattackedHumanScan"] + print(f"unattacked humans flagged by the scan: any {h['any']:.4f} strong {h['strong']:.4f} systematic {h['systematic']:.4f}") + + +def main() -> None: + ap = argparse.ArgumentParser() + sub = ap.add_subparsers(dest="cmd", required=True) + m = sub.add_parser("make") + m.add_argument("--sample-csv", required=True) + m.add_argument("--sample-jsonl", required=True) + m.add_argument("--adversarial", required=True, help="RAID's generation/adversarial directory") + m.add_argument("--nltk-data", required=True) + m.add_argument("--out", required=True) + m.add_argument("--per-class", type=int, default=1000) + m.add_argument("--seed", type=int, default=20261117) + e = sub.add_parser("evaluate") + e.add_argument("--attacked-csv", required=True) + e.add_argument("--attacked-scores", required=True) + e.add_argument("--sample-csv", required=True) + e.add_argument("--scores", required=True) + e.add_argument("--results", required=True) + e.add_argument("--setup", required=True, help="attacked.json written by make") + e.add_argument("--attacked-jsonl", required=True, help="attacked.jsonl, read to explain scanner misses") + e.add_argument("--adversarial", required=True, help="RAID's generation/adversarial directory") + e.add_argument("--out", required=True) + args = ap.parse_args() + make(args) if args.cmd == "make" else evaluate(args) + + +if __name__ == "__main__": + main() diff --git a/tools/raid/evaluate.py b/tools/raid/evaluate.py new file mode 100644 index 0000000..16814b1 --- /dev/null +++ b/tools/raid/evaluate.py @@ -0,0 +1,398 @@ +"""Evaluate the engine's RAID scores with RAID's own code, and break the result down. + +The threshold search and the hit counting are RAID's `run_evaluation`, called unchanged. Everything +this script adds is bookkeeping on top of RAID's output, plus three things RAID does not report: +the product's own boundary, the length floor, and the prediction table in Docs/Detection/prediction.md. + +Scores reach RAID divided by 100. RAID's threshold search assumes a 0-1 range (it starts at the +mean and moves in steps of 0.5), and dividing by a constant changes no ranking and no AUROC. +""" + +import argparse +import hashlib +import json +import math +from pathlib import Path + +import numpy as np +import pandas as pd +from raid import run_evaluation +import raid + +CHAT = ["chatgpt", "gpt4", "llama-chat", "cohere-chat", "mpt-chat", "mistral-chat"] +BASE = ["gpt2", "mpt", "mistral", "cohere"] +BETWEEN = ["gpt3"] +GROUPS = {"chat": CHAT, "base": BASE, "between": BETWEEN} +LENGTH_BUCKETS = [(0, 199, "under 200"), (200, 399, "200-399"), (400, 648, "400-648"), (649, 10**9, "649 or more")] + + +def load(sample_csv: Path, scores_jsonl: Path, texts_jsonl: Path) -> pd.DataFrame: + meta = pd.read_csv(sample_csv) + scores = pd.read_json(scores_jsonl, lines=True) + if len(scores) != len(meta) or set(scores["id"]) != set(meta["id"]): + raise SystemExit("The scores do not cover exactly the sampled ids.") + df = meta.merge(scores, on="id", how="left", validate="one_to_one") + if df["score"].isnull().any(): + raise SystemExit(f"{df['score'].isnull().sum()} rows have no score.") + # RAID's human set repeats some texts verbatim. RAID counts rows and so does the headline; the + # checks this project adds on top count each distinct text once. + hashes = {} + with open(texts_jsonl, encoding="utf-8") as f: + for line in f: + row = json.loads(line) + hashes[row["id"]] = hashlib.sha256(row["text"].encode("utf-8")).hexdigest() + df["textHash"] = df["id"].map(hashes) + if df["textHash"].isnull().any(): + raise SystemExit("Some sampled ids have no text.") + return df + + +def lowest_counted(threshold: float) -> float: + """The lowest one-decimal engine score that RAID's cutoff counts, using RAID's own comparison + (score / 100 >= threshold). Printing the cutoff rounded can move a whole tie across it.""" + v = math.floor(threshold * 1000) / 10 + while (v / 100.0) < threshold: + v = round(v + 0.1, 1) + while v >= 0.1 and (round(v - 0.1, 1) / 100.0) >= threshold: + v = round(v - 0.1, 1) + return v + + +def pick(scores, **want): + """The one record RAID produced for this split, with every unnamed axis at 'all'.""" + keys = {"domain": "all", "model": "all", "decoding": "all", "repetition_penalty": "all", "attack": "all"} + keys.update(want) + for s in scores: + if all(s[k] == v for k, v in keys.items()): + return s + return None + + +def rate(record, fpr="0.05"): + if record is None or record["accuracy"].get(fpr) is None: + return None + return record["accuracy"][fpr]["accuracy"] + + +def counts(record, fpr="0.05"): + a = record["accuracy"][fpr] + return a["tp"], a["fn"] + + +def evaluate(df: pd.DataFrame, per_domain: bool) -> dict: + predictions = [{"id": i, "score": s / 100.0} for i, s in zip(df["id"], df["score"])] + raid_df = df[["id", "model", "decoding", "repetition_penalty", "attack", "domain"]].copy() + return run_evaluation(predictions, raid_df, target_fpr=[0.05, 0.01], per_domain_tuning=per_domain) + + +def summarise(result: dict, df: pd.DataFrame) -> dict: + scores = result["scores"] + machine = df[df["model"] != "human"] + out = { + "overall": { + "tpr05": rate(pick(scores)), + "tpr01": rate(pick(scores), "0.01"), + "auroc": pick(scores)["auroc"], + }, + "thresholds": result["thresholds"], + "lowestCounted": ({d: lowest_counted(t) for d, t in result["thresholds"]["0.05"].items()} + if isinstance(result["thresholds"]["0.05"], dict) + else lowest_counted(result["thresholds"]["0.05"])), + "trueFpr": result["fpr"], + "byModel": {}, + "byDomain": {}, + "byDecoding": {}, + "byRepetitionPenalty": {}, + "byGroup": {}, + } + for m in sorted(machine["model"].unique()): + r = pick(scores, model=m) + out["byModel"][m] = {"tpr05": rate(r), "tpr01": rate(r, "0.01"), "auroc": r["auroc"], "n": int((machine["model"] == m).sum())} + for d in sorted(machine["domain"].unique()): + r = pick(scores, domain=d) + out["byDomain"][d] = {"tpr05": rate(r), "tpr01": rate(r, "0.01"), "auroc": r["auroc"]} + for s in sorted(machine["decoding"].dropna().unique()): + r = pick(scores, decoding=s) + out["byDecoding"][s] = {"tpr05": rate(r), "auroc": r["auroc"]} + for p in sorted(machine["repetition_penalty"].dropna().unique()): + r = pick(scores, repetition_penalty=p) + out["byRepetitionPenalty"][p] = {"tpr05": rate(r), "auroc": r["auroc"]} + + # A group's rate is pooled from RAID's own per-model counts, not averaged, so a generator with + # more texts counts for more, exactly as it would in RAID's 'all'. + for name, models in GROUPS.items(): + tp = fn = 0 + for m in models: + r = pick(scores, model=m) + if r is None or r["accuracy"].get("0.05") is None: + continue + a, b = counts(r) + tp, fn = tp + a, fn + b + out["byGroup"][name] = {"tpr05": tp / (tp + fn) if tp + fn else None, "models": models, "n": tp + fn} + return out + + +def by_length(df: pd.DataFrame, thresholds05, boundary: float) -> list: + """RAID does not split by length. Same per-domain thresholds, applied by hand, and the product's + own boundary beside them.""" + rows = [] + t = df["domain"].map(thresholds05) if isinstance(thresholds05, dict) else thresholds05 + flagged = (df["score"] / 100.0) >= t + at_boundary = df["score"] >= boundary + for lo, hi, label in LENGTH_BUCKETS: + in_bucket = df["words"].between(lo, hi) + human = in_bucket & (df["model"] == "human") + machine = in_bucket & (df["model"] != "human") + rows.append({ + "bucket": label, + "machine": int(machine.sum()), + "human": int(human.sum()), + "tpr05": float(flagged[machine].mean()) if machine.any() else None, + "fpr05": float(flagged[human].mean()) if human.any() else None, + "humanAtBoundary": float(at_boundary[human].mean()) if human.any() else None, + "machineAtBoundary": float(at_boundary[machine].mean()) if machine.any() else None, + }) + return rows + + +def product_view(df: pd.DataFrame, boundary: float, floor: int) -> dict: + """What the shipped product would say: one boundary, and no verdict under the length floor.""" + human = df["model"] == "human" + machine = ~human + over = df["score"] >= boundary + measured = df["words"] >= floor + return { + "boundary": boundary, + "lengthFloorWords": floor, + "ignoringFloor": { + "fpr": float(over[human].mean()), + "tpr": float(over[machine].mean()), + }, + "shareAtOrAboveFloor": { + "human": float(measured[human].mean()), + "machine": float(measured[machine].mean()), + }, + "withFloor": { + "humanFlagged": float((over & measured)[human].mean()), + "machineFlagged": float((over & measured)[machine].mean()), + }, + "medianWords": { + "human": float(df.loc[human, "words"].median()), + "machine": float(df.loc[machine, "words"].median()), + }, + } + + +def matched(df: pd.DataFrame, thresholds05: dict, column: str) -> dict: + """A setting compared only across the generators that were run with both of its values, so a + difference is not just a difference in which generators are in each pool. Same cutoffs.""" + machine = df[df["model"] != "human"] + values = sorted(machine[column].dropna().unique()) + models = sorted(m for m, g in machine.groupby("model") if set(g[column].dropna().unique()) == set(values)) + sub = machine[machine["model"].isin(models)] + flagged = (sub["score"] / 100.0) >= sub["domain"].map(thresholds05) + out = {"models": models} + for v in values: + mask = sub[column] == v + out[v] = {"tpr05": float(flagged[mask].mean()), "n": int(mask.sum())} + return out + + +def wilson(k: int, n: int, z: float = 1.959964) -> tuple: + """The interval docs/CALIBRATION.md publishes (2 of 296 -> 0.19%-2.43%), so the two compare.""" + if n == 0: + return (None, None) + p = k / n + d = 1 + z * z / n + centre = (p + z * z / (2 * n)) / d + margin = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5) / d + return (max(0.0, centre - margin), min(1.0, centre + margin)) + + +def long_human_check(df: pd.DataFrame, boundary: float, floor: int, published: dict) -> dict: + """RAID's human texts at or above the floor: long human writing from registers the calibration + corpus does not contain (newswire, book summaries, film reviews, verse). Same boundary, same + interval, nothing tuned. An outside sample of a different population, not a replication.""" + rows = df[(df["model"] == "human") & (df["words"] >= floor)] + long_human = rows.drop_duplicates("textHash") + over = long_human["score"] >= boundary + k = int(over.sum()) + n = int(len(long_human)) + lo, hi = wilson(k, n) + by_domain = {} + for d, g in long_human.groupby("domain"): + dk, dn = int((g["score"] >= boundary).sum()), int(len(g)) + by_domain[d] = {"texts": dn, "flagged": dk, "interval": list(wilson(dk, dn))} + flagged_rules = long_human.loc[over, "rules"].explode().dropna().value_counts() + has_burst = long_human["rules"].apply(lambda rs: "stat.burstiness" in rs) + for d, g in long_human.groupby("domain"): + by_domain[d]["burstiness"] = float(has_burst[g.index].mean()) + not_news = long_human["domain"] != "news" + return { + "rows": int(len(rows)), + "texts": n, + "flagged": k, + "rate": k / n if n else None, + "interval": [lo, hi], + "byDomain": by_domain, + "rulesInFlagged": {r: int(c) for r, c in flagged_rules.items()}, + "burstinessAll": float(has_burst.mean()), + "withoutNews": {"texts": int(not_news.sum()), "flagged": int((over & not_news).sum())}, + "published": {"texts": published["texts"], "flagged": published["flaggedAtThreshold"], + "interval": [published["rateLow"], published["rateHigh"]]}, + } + + +def rule_lift(df: pd.DataFrame, min_texts: int = 50) -> list: + """For each rule: in what share of human texts, chat-tuned texts and base-model texts it fires. + Reported, never acted on inside this measurement.""" + groups = { + "human": df["model"] == "human", + "chat": df["model"].isin(CHAT), + "base": df["model"].isin(BASE), + } + exploded = df[["id", "model", "rules"]].explode("rules").dropna(subset=["rules"]) + rows = [] + for rule, sub in exploded.groupby("rules"): + ids = set(sub["id"]) + hit = df["id"].isin(ids) + if hit.sum() < min_texts: + continue + share = {g: float(hit[mask].mean()) for g, mask in groups.items()} + rows.append({ + "rule": rule, + "texts": int(hit.sum()), + "human": share["human"], + "chat": share["chat"], + "base": share["base"], + "chatLift": share["chat"] / share["human"] if share["human"] > 0 else None, + }) + rows.sort(key=lambda r: -(r["chatLift"] or 0)) + return rows + + +def domain_rules(df: pd.DataFrame, domain: str, top: int = 8) -> list: + """Why a domain separates well or badly: the rules whose share differs most between the two classes.""" + d = df[df["domain"] == domain] + human = d["model"] == "human" + exploded = d[["id", "rules"]].explode("rules").dropna(subset=["rules"]) + rows = [] + for rule, sub in exploded.groupby("rules"): + hit = d["id"].isin(set(sub["id"])) + rows.append({"rule": rule, "human": float(hit[human].mean()), "machine": float(hit[~human].mean())}) + rows.sort(key=lambda r: -(r["machine"] - r["human"])) + return rows[:top] + + +def rerun_summary(published: dict, rerun_path) -> dict | None: + """Which fields of the published calibration moved when the current engine re-ran it.""" + if not rerun_path: + return None + rerun = json.loads(Path(rerun_path).read_text(encoding="utf-8")) + return { + "engine": rerun["engine"], + "measuredOn": rerun["measuredOn"], + "fieldsChanged": sorted(k for k in published if published[k] != rerun.get(k)), + "file": "Docs/Detection/calibration-rerun.json", + } + + +def raid_mix(per_domain: dict, dataset_by_model: dict) -> float: + """The overall rate re-weighted to RAID's proportions instead of this sample's equal ones.""" + by_model = per_domain["byModel"] + total = sum(dataset_by_model[m] for m in by_model) + return sum(by_model[m]["tpr05"] * dataset_by_model[m] for m in by_model) / total + + +def predictions(per_domain: dict) -> list: + g = per_domain["byGroup"] + chat, base, between = g["chat"]["tpr05"], g["base"]["tpr05"], g["between"]["tpr05"] + domains = {d: v["tpr05"] for d, v in per_domain["byDomain"].items()} + top3 = sorted(domains, key=lambda d: -(domains[d] or 0))[:3] + lo, hi = min(chat, base), max(chat, base) + overall = per_domain["overall"]["tpr05"] + return [ + {"id": "P1", "claim": "chat-tuned rate at least twice the base-model rate", + "measured": f"chat {chat:.3f}, base {base:.3f}, ratio {chat / base:.2f}" if base else f"chat {chat:.3f}, base 0", + "held": bool(base == 0 or chat / base >= 2)}, + {"id": "P2", "claim": "base-model rate under 20%", "measured": f"{base:.3f}", "held": bool(base < 0.20)}, + {"id": "P3", "claim": "text-davinci-003 between the two groups", + "measured": f"{between:.3f} (range {lo:.3f}-{hi:.3f})", "held": bool(lo <= between <= hi)}, + {"id": "P4", "claim": "poetry and recipes not among the three best domains", + "measured": "best three: " + ", ".join(f"{d} {domains[d]:.3f}" for d in top3), + "held": bool("poetry" not in top3 and "recipes" not in top3)}, + {"id": "overall", "claim": "between 15% and 40% overall, without attacks", + "measured": f"{overall:.3f}", "held": bool(0.15 <= overall <= 0.40)}, + ] + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--sample", required=True, help="sample.csv") + ap.add_argument("--scores", required=True, help="scores.jsonl from `signsofai-calibrate raid`") + ap.add_argument("--texts", required=True, help="sample.jsonl, read only to recognise repeated texts") + ap.add_argument("--manifest", required=True, help="sample.json") + ap.add_argument("--calibration", required=True, help="published-calibration.json") + ap.add_argument("--calibration-rerun", help="published-calibration.json as regenerated by the current engine; " + "committed as Docs/Detection/calibration-rerun.json so the comparison has a source") + ap.add_argument("--engine", required=True, help="engine version that produced the scores") + ap.add_argument("--out", required=True) + args = ap.parse_args() + + df = load(Path(args.sample), Path(args.scores), Path(args.texts)) + calibration = json.loads(Path(args.calibration).read_text(encoding="utf-8")) + + per_domain = summarise(evaluate(df, per_domain=True), df) + single = summarise(evaluate(df, per_domain=False), df) + + results = { + "engine": args.engine, + "raidBench": raid.__version__, + "sample": json.loads(Path(args.manifest).read_text(encoding="utf-8")), + "language": "en", + "detectedNotEnglish": { + "human": float((df.loc[df["model"] == "human", "detectedLanguage"] != "en").mean()), + "machine": float((df.loc[df["model"] != "human", "detectedLanguage"] != "en").mean()), + }, + "perDomainThreshold": per_domain, + "singleThreshold": single, + "byLength": by_length(df, per_domain["thresholds"]["0.05"], calibration["recommendedThreshold"]), + "matchedRepetitionPenalty": matched(df, per_domain["thresholds"]["0.05"], "repetition_penalty"), + "matchedDecoding": matched(df, per_domain["thresholds"]["0.05"], "decoding"), + "humanTexts": {"rows": int((df["model"] == "human").sum()), + "distinct": int(df.loc[df["model"] == "human", "textHash"].nunique())}, + "product": product_view(df, calibration["recommendedThreshold"], calibration["shortestWords"]), + "calibration": { + "engine": calibration["engine"], + "measuredOn": calibration["measuredOn"], + "burstinessShare": next((r["textShare"] for r in calibration["noisiestRules"] + if r["ruleId"] == "stat.burstiness"), None), + "rerun": rerun_summary(calibration, args.calibration_rerun), + }, + "predictions": predictions(per_domain), + "predictionsSingleThreshold": predictions(single), + "raidMixTpr05": raid_mix(per_domain, json.loads(Path(args.manifest).read_text(encoding="utf-8"))["datasetMachineByModel"]), + "longHumanCheck": long_human_check(df, calibration["recommendedThreshold"], calibration["shortestWords"], calibration), + "rules": rule_lift(df), + "domainRules": {d: domain_rules(df, d) for d in ("poetry", "reviews", "recipes", "reddit")}, + "scoreQuantiles": { + "human": {q: float(np.quantile(df.loc[df["model"] == "human", "score"], q)) for q in (0.5, 0.9, 0.95, 0.99)}, + "machine": {q: float(np.quantile(df.loc[df["model"] != "human", "score"], q)) for q in (0.5, 0.9, 0.95, 0.99)}, + }, + } + Path(args.out).write_text(json.dumps(results, indent=2) + "\n", encoding="utf-8") + + o = per_domain["overall"] + print(f"RAID, per-domain threshold: TPR@5% {o['tpr05']:.3f} TPR@1% {o['tpr01']} AUROC {o['auroc']:.3f}") + s = single["overall"] + print(f"RAID, single threshold: TPR@5% {s['tpr05']} AUROC {s['auroc']:.3f}") + for name, v in per_domain["byGroup"].items(): + print(f" {name:<8} {v['tpr05']:.3f} (n={v['n']})") + lh = results["longHumanCheck"] + print(f" long humans: {lh['flagged']} of {lh['texts']} at the boundary, interval {lh['interval']}") + for p in results["predictions"]: + print(f" {p['id']:<8} {'HELD ' if p['held'] else 'FAILED '} {p['measured']}") + + +if __name__ == "__main__": + main() diff --git a/tools/raid/leaderboard.py b/tools/raid/leaderboard.py new file mode 100644 index 0000000..ee11f96 --- /dev/null +++ b/tools/raid/leaderboard.py @@ -0,0 +1,54 @@ +"""For scale only: what RAID's own authors measured for open detectors, from the leaderboard files. + +These numbers are on RAID-test; this project's are on a sample of RAID-train. Same distribution, +different texts, so they set a scale and are never a row in the same table. +""" + +import argparse +import json +from pathlib import Path + +ATTACKS = ["none", "homoglyph", "zero_width_space", "whitespace"] + + +def pick(scores, attack): + for s in scores: + if (s["domain"], s["model"], s["decoding"], s["repetition_penalty"], s["attack"]) == ("all", "all", "all", "all", attack): + a = s["accuracy"].get("0.05") + return a["accuracy"] if a else None + return None + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--dir", required=True, help="leaderboard results.json files, one per detector") + ap.add_argument("--commit", required=True, help="the RAID repository commit they were read at") + ap.add_argument("--out", required=True) + args = ap.parse_args() + + detectors, read, skipped = [], [], [] + for f in sorted(Path(args.dir).glob("*.json")): + d = json.loads(f.read_text(encoding="utf-8")) + read.append(f.name) + scores = d.get("scores") or [] + row = {a: pick(scores, a) for a in ATTACKS} + if row["none"] is None: + skipped.append({"file": f.name, "reason": "no score for the all-domains, no-attack split"}) + continue + detectors.append({ + "name": d["detector_name"], + "submittedBy": "RAID authors" if "RAID Authors" in str(d.get("contact_info")) else "others", + "tpr05": row, + }) + detectors.sort(key=lambda r: -r["tpr05"]["none"]) + out = {"source": "github.com/liamdugan/raid leaderboard/submissions/*/results.json", + "commit": args.commit, "split": "RAID-test", + "selection": "chosen by hand: the open detectors RAID's authors ran, and the highest entries on 2026-11-17", + "filesRead": read, "skipped": skipped, "detectors": detectors} + Path(args.out).write_text(json.dumps(out, indent=2) + "\n", encoding="utf-8") + for r in detectors: + print(f"{r['name']:<20} {r['submittedBy']:<13} " + " ".join(f"{(v or 0) * 100:6.1f}" for v in r["tpr05"].values())) + + +if __name__ == "__main__": + main() diff --git a/tools/raid/report.py b/tools/raid/report.py new file mode 100644 index 0000000..2db0d51 --- /dev/null +++ b/tools/raid/report.py @@ -0,0 +1,475 @@ +"""Write Docs/DETECTION.md from the measurement files in Docs/Detection/. + +Every number on the page comes from those files. Nothing here is typed by hand, so the page and the +data cannot drift apart: change the measurement, re-run this, and the page follows. +""" + +import argparse +import json +from pathlib import Path + +NAMES = { + "chatgpt": "ChatGPT", "gpt4": "GPT-4", "llama-chat": "Llama 2 70B Chat", "mistral-chat": "Mistral 7B Chat", + "mpt-chat": "MPT-30B Chat", "cohere-chat": "Cohere Chat", "gpt3": "GPT-3 (text-davinci-003)", + "gpt2": "GPT-2 XL", "mistral": "Mistral 7B", "cohere": "Cohere", "mpt": "MPT-30B", +} +GROUP = {**{m: "chat-tuned" for m in ["chatgpt", "gpt4", "llama-chat", "mistral-chat", "mpt-chat", "cohere-chat"]}, + **{m: "base" for m in ["gpt2", "mistral", "cohere", "mpt"]}, "gpt3": "instruction-tuned"} +DOMAINS = {"abstracts": "arXiv abstracts", "books": "book summaries", "news": "news articles", "poetry": "poetry", + "recipes": "recipes", "reddit": "Reddit posts", "reviews": "film reviews", "wiki": "Wikipedia"} + + +def pct(x, digits=1): + return "n/a" if x is None else f"{x * 100:.{digits}f}%" + + +def interval(pair, digits=1): + lo, hi = pair + return f"{lo * 100:.{digits}f}–{hi * 100:.{digits}f}%" + + +WORDS = {2: "two", 3: "three", 4: "four", 5: "five", 6: "six", 7: "seven", 8: "eight", 9: "nine", 10: "ten", + 11: "eleven", 12: "twelve", 13: "thirteen", 14: "fourteen", 15: "fifteen", 16: "sixteen", 20: "twenty"} + + +def few(x): + """A count of people a teacher can picture, not a fraction of a student.""" + if x < 1: + return "fewer than one" + if x < 2: + return "one or two" + return f"about {round(x)}" + + +def times(x): + n = round(x) + return {2: "twice", 3: "three times", 4: "four times", 5: "five times"}.get(n, f"{n} times") + + +def build(r: dict, a: dict, lb: dict, commit: str, run_date: str) -> str: + s = r["sample"] + pd_ = r["perDomainThreshold"] + single = r["singleThreshold"] + prod = r["product"] + lh = r["longHumanCheck"] + cal = r["calibration"] + groups = pd_["byGroup"] + by_model = pd_["byModel"] + rules = {x["rule"]: x for x in r["rules"]} + attacks = a["attacks"] + preds = {p["id"]: p for p in r["predictions"]} + p5, p6 = a["predictions"]["P5"], a["predictions"]["P6"] + hg, zw = attacks["homoglyph"], attacks["zero_width_space"] + base_models = [by_model[m]["tpr05"] for m in ["gpt2", "mistral", "cohere", "mpt"]] + burst = rules.get("stat.burstiness") + poetry_burst = next((x for x in r["domainRules"]["poetry"] if x["rule"] == "stat.burstiness"), None) + overall = pd_["overall"]["tpr05"] + human_rows = r["humanTexts"]["rows"] + human_verdicts = round(prod["withFloor"]["humanFlagged"] * human_rows) + news = lh["byDomain"]["news"] + wn = lh["withoutNews"] + pub = lh["published"] + pub_rate = pub["flagged"] / pub["texts"] + + L = [] + w = L.append + w("# How well the score tells machine-written text from human writing") + w("") + w("") + w("") + w("[CALIBRATION.md](CALIBRATION.md) measures how often this tool is wrong about a person, and says plainly what " + "it leaves out: nothing about how much AI writing it catches, because a tool that flags nothing has a perfect " + "false-positive rate. This page measures that other half.") + w("") + w("The calibration page also says why it avoids an accuracy figure: any collection of machine-written text ages, " + "and flatters whoever assembled it. So this page uses a collection nobody here assembled, " + "**RAID** (Dugan et al., ACL 2024, [arXiv 2405.07940](https://arxiv.org/abs/2405.07940)), which its authors " + "describe as the largest benchmark for detectors of machine-generated text, and it states the date. RAID's " + "generators are from 2023. The tells this tool looks for shift with every model, so this half of the picture " + "ages and the human half does not.") + w("") + w("## If you are a teacher, read this first") + w("") + w(f"- **A low score tells you nothing.** In this benchmark about three in four machine-written texts " + f"({pct(1 - overall)}) scored too low to stand out from human writing.") + w("- **None of the detection rates further down is what the product does.** They are counted at cutoffs the " + f"benchmark chose, on texts mostly under 400 words. The product gives no verdict under " + f"{prod['lengthFloorWords']} words, so on this benchmark it would have given its verdict on no machine-written " + f"text at all, and on {human_verdicts} texts written by people.") + w(f"- **For texts of {prod['lengthFloorWords']} words or more, where the product does give a verdict, how much " + "machine writing it finds has not been measured.** No machine text in this sample is that long. So for a " + "long text that crosses the boundary, this page cannot tell you how likely it is to be machine-written. On " + "this benchmark every long text that crossed was written by a person.") + w(f"- **The boundary was checked on long human writing it had never been measured on, and it did not hold for " + f"news articles in this sample**: {news['flagged']} of {news['texts']} crossed it, about one in " + f"{WORDS.get(round(news['texts'] / news['flagged']), round(news['texts'] / news['flagged']))}, where the " + "promise is at most one in twenty. The calibration corpus " + "holds research articles, encyclopedia entries and essays by adult learners of English at one university. " + "Anything else is unmeasured, and that includes ordinary secondary-school essays. Until a kind of writing is " + "measured, read a high score on it as saying nothing.") + k_hit = 3 * overall + k_false = 27 * 0.05 + w(f"- **A score above a cutoff is not a catch.** At the benchmark's cutoffs (not the product's), in a class of 30 " + f"where 3 texts were machine-written, you would expect {few(k_hit)} of the 3 above the cutoff, and " + f"{few(k_false)} of the 27 honest ones. That is arithmetic from {pct(overall)} and 5%, not a classroom " + "measurement.") + w("- **The machine text here is from 2023.** \"ChatGPT\" on this page is the 2023 model. Nothing here describes " + "what a student can use today.") + w("- **What the tool does show, at any length, is which tells a text carries and where.** That is a starting " + "point for a conversation with the writer, not a finding about them.") + w("") + w("Words used below. *Chat-tuned* means trained to answer in conversation, like ChatGPT; a *base* model only " + "continues a text. *AUROC* runs from 0.5 (the score cannot tell the two apart) to 1 (it always can). " + "*Above the cutoff* means only that a text scored above it; it does not mean anyone was identified.") + w("") + w("## What was measured") + w("") + w(f"- **Benchmark** RAID-train without attacks (`train_none.csv`, SHA-256 `{s['datasetSha256'][:16]}`), " + f"evaluated with raid-bench {r['raidBench']}") + w(f"- **Sample** {s['total']:,} texts: all {s['human']:,} human texts, and {s['machinePerGeneratorAndDomain']} " + f"machine texts for each of 11 generators in each of 8 domains ({s['machine']:,}); seed {s['seed']}, " + f"id fingerprint `{s['idFingerprintSha256'][:16]}`") + w(f"- **Engine** SignsOfAI.Core {r['engine']} at `{commit}`, rule packs unchanged, language set to English") + w(f"- **Run** {run_date}") + w("- **Counted by** RAID's own `run_evaluation`, at a 5% and a 1% false-positive rate") + w("") + w("**How RAID counts.** It takes the raw score and finds, separately for each domain, the cutoff that flags 5% " + "of its human texts. Then it counts how much machine text lands above that cutoff. The score is this tool's " + "0–100 divided by 100, because RAID's search assumes a 0–1 range; dividing by a constant changes no ranking. " + "So the product's own boundary and its length floor play no part here, and the cutoffs RAID finds are not " + "this tool's.") + w("") + w("## The headline") + w("") + w(f"- **{pct(overall)}** of the machine-written texts score above RAID's cutoffs at a 5% false-positive rate " + f"({pct(pd_['overall']['tpr01'])} at 1%; AUROC {pd_['overall']['auroc']:.2f}). Weighted to RAID's own mix of " + f"generators instead of this sample's equal one: {pct(r['raidMixTpr05'])}.") + w(f"- **Chat-tuned generators {pct(groups['chat']['tpr05'])}, base models {pct(groups['base']['tpr05'])}.** " + f"ChatGPT alone {pct(by_model['chatgpt']['tpr05'])}. The four base models sit between {pct(min(base_models))} " + f"and {pct(max(base_models))}, two to three times the 5% a score that knew nothing would reach, while their " + "AUROC is near 0.5: more of their text than of human text scores high, and across the rest of the range that " + "advantage is cancelled. Where was not measured.") + w(f"- **As shipped, the product would have given its verdict on none of the machine-written texts in this sample, " + f"and on {human_verdicts} texts written by people.** No machine-written text in the sample reaches " + f"{prod['lengthFloorWords']} words, the shortest text the published false-positive rate was measured on. Below " + "that length the product shows the score and the tells and withholds its verdict (#59).") + w(f"- **Found, not predicted:** on long human writing from registers the calibration never held, {lh['flagged']} " + f"of {lh['texts']} texts cross the product's boundary, {news['flagged']} of them news articles. Details under " + "*Long human writing* below.") + w("") + w("This is below the entries submitted by others at the top of RAID's leaderboard and below each of the open detectors " + "in the scale table further down, with the caveats given there. On this benchmark the tells are mostly " + "fingerprints of chat-tuning. They are evidence about how a text reads, not a way to tell who wrote it.") + w("") + w("## What we predicted, and what happened") + w("") + w("Written and committed before any RAID text was scored ([prediction.md](Detection/prediction.md), commit " + "`e007adc`). Each line is judged by the refutation criterion it was written with.") + w("") + w("| # | Prediction | Measured | Result |") + w("|---|---|---|---|") + w(f"| P1 | Chat-tuned rate at least twice the base-model rate | chat {pct(groups['chat']['tpr05'])}, " + f"base {pct(groups['base']['tpr05'])}, ratio {groups['chat']['tpr05'] / groups['base']['tpr05']:.2f} | " + f"{'held' if preds['P1']['held'] else '**failed**'} |") + w(f"| P2 | Base-model rate under 20% | {pct(groups['base']['tpr05'])} | {'held' if preds['P2']['held'] else '**failed**'} |") + w(f"| P3 | text-davinci-003 between the two groups | {pct(groups['between']['tpr05'])} | " + f"{'held' if preds['P3']['held'] else '**failed**'} |") + top3 = sorted(pd_["byDomain"].items(), key=lambda x: -x[1]["tpr05"])[:3] + w(f"| P4 | Poetry and recipes not among the three best domains | best three: " + f"{', '.join(DOMAINS[d] + ' ' + pct(v['tpr05']) for d, v in top3)} | " + f"{'held' if preds['P4']['held'] else '**failed**'} |") + w(f"| P5 | Scanner reports at least 99% of homoglyph and zero-width attacks, with false alarms on at most 1% of " + f"human texts | homoglyph {pct(p5['homoglyphStrong'])}, zero-width {pct(p5['zeroWidthStrong'])}; any " + f"artifact on {pct(p5['humanAlarms'] / p5['humanTexts'], 2)} of unattacked human texts " + f"({interval(p5['humanAlarmInterval'], 2)}) | " + f"{'held on the point estimate; the interval touches 1%' if p5['held'] and p5['humanAlarmInterval'][1] > 0.01 else ('held' if p5['held'] else '**failed**')} |") + w(f"| P6 | The homoglyph attack lowers the score | mean {hg['meanScoreBefore']:.1f} → {hg['meanScoreAfter']:.1f}; " + f"above the cutoff {pct(hg['tprBefore'])} → {pct(hg['tprAfter'])} | " + f"{'held, weakly (−' + format(p6['scoreDrop'], '.2f') + ' points)' if p6['held'] else '**failed**'} |") + w(f"| — | Between 15% and 40% overall, without attacks | {pct(overall)} | " + f"{'held' if preds['overall']['held'] else '**failed**'} |") + w("") + if poetry_burst: + w(f"**P4 failed.** One observation that may explain it, found afterwards: `stat.burstiness` fires on " + f"{pct(poetry_burst['machine'], 0)} of machine poems and {pct(poetry_burst['human'], 0)} of human ones. " + f"Recipes were as weak as predicted ({pct(pd_['byDomain']['recipes']['tpr05'])}).") + w("") + w(f"**P6 held, weakly.** The score dropped by {p6['scoreDrop']:.2f} points on average across the machine texts, " + "through the mechanism P6 named (see *Attacks*). The note added the same day predicted no drop; it was right " + "about the zero-width attack and wrong about homoglyphs. Between them the prediction and the note covered both " + "outcomes, so this line was not a test that could fail.") + w("") + ps = {p["id"]: p for p in r["predictionsSingleThreshold"]} + sg = single["byGroup"] + same = all(ps[k]["held"] == preds[k]["held"] for k in preds) + s_top3 = sorted(single["byDomain"].items(), key=lambda x: -x[1]["tpr05"])[:3] + w(f"At the single cutoff {'every line holds or fails as above' if same else 'some lines differ'} (P4's best three " + f"become {', '.join(DOMAINS[d] for d, _ in s_top3[:2])} and {DOMAINS[s_top3[2][0]]}): chat-tuned " + f"{pct(sg['chat']['tpr05'])}, base {pct(sg['base']['tpr05'])}, text-davinci-003 {pct(sg['between']['tpr05'])}, " + f"so P3 holds there by {(sg['between']['tpr05'] - sg['base']['tpr05']) * 100:.1f} points.") + w("") + w("## By generator") + w("") + w("| Generator | Kind | Above the cutoff at 5% | At 1% | AUROC |") + w("|---|---|---|---|---|") + for m, v in sorted(by_model.items(), key=lambda x: -x[1]["tpr05"]): + w(f"| {NAMES[m]} | {GROUP[m]} | {pct(v['tpr05'])} | {pct(v['tpr01'])} | {v['auroc']:.2f} |") + w("") + below = [NAMES[m] for m, v in by_model.items() if v["auroc"] < 0.5] + if below: + w(f"An AUROC below 0.5 ({', '.join(below)}) means a text from that generator scores lower than a human text " + "more often than it scores higher.") + w("") + w("## By domain") + w("") + w("| Domain | Above the cutoff at 5% | AUROC | Lowest score above RAID's cutoff (not the product's 30) |") + w("|---|---|---|---|") + low = pd_["lowestCounted"] + for d, v in sorted(pd_["byDomain"].items(), key=lambda x: -x[1]["tpr05"]): + w(f"| {DOMAINS[d]} | {pct(v['tpr05'])} | {v['auroc']:.2f} | {low[d]:.1f} |") + w("") + w("The engine scores to one decimal and RAID's cutoffs fall between those steps, so the last column gives the " + "lowest score each cutoff counts rather than the cutoff rounded: rounding can carry a whole tie across it. " + "The exact cutoffs are in `Docs/Detection/results.json`.") + w("") + md, mr = r["matchedDecoding"], r["matchedRepetitionPenalty"] + w(f"Greedy decoding scores above the cutoff more often than sampling ({pct(md['greedy']['tpr05'])} against " + f"{pct(md['sampling']['tpr05'])}, across all {len(md['models'])} generators). Only {len(mr['models'])} " + f"generators were run both with and without a repetition penalty; across those, texts generated with it are " + f"above the cutoff {pct(mr['yes']['tpr05'])} of the time and texts without it {pct(mr['no']['tpr05'])}. These " + "are observed differences between settings, not controlled experiments, and what causes them was not measured.") + w("") + w(f"With one cutoff for every domain, closer to how the product works, the rate is " + f"{pct(single['overall']['tpr05'])} (lowest score counted {single['lowestCounted']:.1f}).") + w("") + w("## Found along the way") + w("") + w("None of what follows was predicted. It is reported because it is true, and it should be read as " + "observation rather than as a test that could have failed.") + w("") + w("### The product's boundary on short text") + w("") + w(f"At the product's boundary of {prod['boundary']}/100, ignoring the length floor, " + f"{pct(prod['ignoringFloor']['fpr'])} of RAID's human texts score above it, against " + f"{pct(prod['ignoringFloor']['tpr'])} of the machine ones. The floor was not set from this number: it is the " + "shortest text the calibration measured (#59). This is the first outside look at what lies under it. The human " + "rate at the boundary falls as texts get longer, as the table shows, but this sample does not separate length " + "from register.") + w("") + w(f"| Length | Human texts | Machine texts | Human above {prod['boundary']} (no verdict under " + f"{prod['lengthFloorWords']} words) | Machine above {prod['boundary']} (same) " + "| Human above RAID's cutoff | Machine above RAID's cutoff |") + w("|---|---|---|---|---|---|---|") + for b in r["byLength"]: + w(f"| {b['bucket']} words | {b['human']:,} | {b['machine']:,} | {pct(b['humanAtBoundary'])} | " + f"{pct(b['machineAtBoundary'])} | {pct(b['fpr05'])} | {pct(b['tpr05'])} |") + w("") + w(f"Only {pct(prod['shareAtOrAboveFloor']['human'])} of the human texts and none of the machine texts reach " + f"{prod['lengthFloorWords']} words.") + w("") + w(f"The only rates that describe the product as shipped are the published {pct(pub_rate)} ({pub['flagged']} of " + f"{pub['texts']}) and, on this benchmark's long human texts, {pct(lh['rate'])} ({lh['flagged']} of " + f"{lh['texts']}). Every other percentage on this page is counted at a cutoff the product does not use, or on " + "texts it gives no verdict on.") + w("") + w("### Long human writing the calibration does not contain") + w("") + w(f"{lh['texts']} distinct human texts in the sample reach the floor ({lh['rows']} rows: RAID repeats a few " + "texts verbatim, and here each is counted once). They are an outside sample of long human writing, but " + "of a different population from the calibration corpus, which holds research articles, encyclopedia " + "revisions and learner essays. These are news articles, book summaries, film reviews and verse. At the " + f"product's boundary of {prod['boundary']}/100:") + w("") + w("| Domain | Texts | Crossing the boundary | 95% interval | `stat.burstiness` fires on |") + w("|---|---|---|---|---|") + for d, v in sorted(lh["byDomain"].items(), key=lambda x: -x[1]["flagged"]): + w(f"| {DOMAINS[d]} | {v['texts']} | {v['flagged']} | {interval(v['interval'])} | {pct(v['burstiness'], 0)} |") + w(f"| **all** | **{lh['texts']}** | **{lh['flagged']}** | **{interval(lh['interval'])}** | " + f"**{pct(lh['burstinessAll'], 0)}** |") + w("") + w(f"Overall that is {pct(lh['rate'])} ({lh['flagged']} of {lh['texts']}), against the published " + f"{pct(pub_rate)} ({pub['flagged']} of {pub['texts']}, {interval(pub['interval'])}): about " + f"{times(lh['rate'] / pub_rate)} the rate, a difference this sample is too small to establish. Without the " + f"news articles it is {wn['flagged']} of {wn['texts']} ({pct(wn['flagged'] / wn['texts'])}).") + w("") + w(f"**{news['flagged']} of {news['texts']} news articles cross the boundary** " + f"({pct(news['flagged'] / news['texts'])}, interval {interval(news['interval'])}). That interval lies wholly " + f"above the published one and reaches {times(news['interval'][1] / 0.05)} the 5% target: by the rule that set " + "the boundary, 30/100 is not supported for news. News was singled out after seeing the table, and a domain " + "chosen that way overstates itself, so this needs a second sample before it is quoted as a rate.") + w("") + nb = lh["rulesInFlagged"].get("stat.burstiness", 0) + w(f"{'All ' + str(nb) if nb == lh['flagged'] else str(nb) + ' of the ' + str(lh['flagged'])} texts that cross " + f"carry `stat.burstiness`. The rule fires on " + f"{pct(news['burstiness'], 0)} of the long news articles, crossing or not, against {pct(lh['burstinessAll'], 0)} " + f"of all the long human texts here and {pct(cal['burstinessShare'], 0)} of the calibration corpus. News " + "articles trip the sentence-rhythm rule far more often than the writing the boundary was measured on; why " + "was not measured.") + w("") + rr = cal.get("rerun") + if rr: + moved = [f for f in rr["fieldsChanged"] if f not in ("engine", "measuredOn")] + w(f"The published rate was measured with engine {cal['engine']} ({cal['measuredOn']}) and this sample with " + f"{r['engine']}. Re-running the calibration with {rr['engine']} " + + ("changed only its engine and date fields" if not moved else "changed " + ", ".join(moved)) + + " ([calibration-rerun.json](Detection/calibration-rerun.json)).") + else: + w(f"The published rate was measured with engine {cal['engine']} ({cal['measuredOn']}) and this sample with " + f"{r['engine']}; the calibration was not re-run with {r['engine']}.") + w("") + w("This does not move the published number, which describes the corpus it was measured on. Any change to it goes " + "through the calibration corpus and the review committee, not through RAID. What it does say: until a kind of " + "writing has been measured, a score above 30 on it should be read as saying nothing, and that includes every " + "kind absent from the calibration corpus, not only news. The fix is the one the calibration page gives: run the " + "calibration on your own students' pre-2022 work (`Docs/Calibration/README.md`). If you have no such work, " + "you have no boundary for your classroom.") + w("") + w("### Which rules carry the signal") + w("") + w("Share of texts each rule fires on, for the rules that fire most often on chat-tuned text relative to human " + "text (at least 50 texts):") + w("") + w("| Rule | Human | Chat-tuned | Base | Chat ÷ human |") + w("|---|---|---|---|---|") + for x in r["rules"][:12]: + w(f"| `{x['rule']}` | {pct(x['human'], 2)} | {pct(x['chat'], 2)} | {pct(x['base'], 2)} | {x['chatLift']:.0f}× |") + w("") + if burst: + w(f"Each of them is rare: strong evidence where it appears, absent from most texts. The rule with the largest " + f"absolute gap is `stat.burstiness`, which fires on {pct(burst['human'])} of human texts, " + f"{pct(burst['chat'])} of chat-tuned ones and {pct(burst['base'])} of base-model ones. No rule was removed " + "and re-scored, so how much of the separation each rule carries was not measured.") + w("") + wrong = [x for x in r["rules"] if x["chatLift"] is not None and x["chatLift"] < 0.5 and x["texts"] >= 200] + if wrong: + w("Some rules fire more often on human text than on RAID's chat-tuned output: " + + ", ".join(f"`{x['rule']}` ({pct(x['human'])} of human texts, {pct(x['chat'])} of chat-tuned)" for x in wrong) + + ". On RAID these rules point the wrong way. Whether they separate later models was not measured here, and " + "no rule was changed in response.") + w("") + w("### Attacks") + w("") + setup = a["setup"] + words = {7: "Seven", 8: "Eight", 9: "Nine"} + w(f"{words.get(len(setup['attacks']), len(setup['attacks']))} of RAID's eleven attacks need no language model. " + f"They were run, at the strengths RAID's own code uses, on {setup['perClass']:,} human and " + f"{setup['perClass']:,} machine texts drawn from the sample. Paraphrase, synonym swap and perplexity " + "misspelling need a model and were left out, and paraphrase is the attack most likely to hurt a detector " + "built on word choice. What follows is robustness to cheap attacks, not to rewriting.") + w("") + w("| Attack | Strength | Above the cutoff before | After | Mean score | Scanner reports a strong artifact |") + w("|---|---|---|---|---|---|") + for name, v in sorted(attacks.items()): + w(f"| {name.replace('_', ' ')} | {setup['attacks'][name]} | {pct(v['tprBefore'])} | {pct(v['tprAfter'])} | " + f"{v['meanScoreBefore']:.1f} → {v['meanScoreAfter']:.1f} | {pct(v['scanMachine']['strong'])} |") + w("") + misses = a["scannerMisses"]["homoglyph"] + n_machine = sum(m["class"] == "machine" for m in misses) + n_script = sum(m["reason"] == "written in another script" for m in misses) + n_nolatin = sum(m["reason"] == "no Latin letter left" for m in misses) + w(f"Counting only characters that typing does not produce, the scanner reports {pct(hg['scanMachine']['strong'])} " + f"of the homoglyph-attacked machine texts and {pct(hg['scanHuman']['strong'])} of the human ones, and " + f"{pct(zw['scanMachine']['strong'])} and {pct(zw['scanHuman']['strong'])} under the zero-width attack: it " + "describes the file, not who wrote it. On unattacked human texts it reports some artifact in " + f"{pct(a['unattackedHumanScan']['any'], 2)}, a strong one in {pct(a['unattackedHumanScan']['strong'], 2)}, and " + f"a systematic pattern in {pct(a['unattackedHumanScan']['systematic'], 2)}. The {len(misses)} homoglyph texts it " + f"did not report ({n_machine} machine, {len(misses) - n_machine} human) are listed in `attacks.json` with a " + f"reason computed from the text: {n_script} are written in another script, where those letters are the " + f"alphabet, and {n_nolatin} have no Latin letter left to disguise.") + w("") + w("**What the homoglyph attack still moves.** The scanner counts a disguised word only when Latin letters are at " + "least half of it, so that a Greek or Cyrillic word quoted in English is not called a disguise. RAID's attack " + "replaces every letter it can. In words such as *crucial*, *intricate* or a capitalised *Moreover* the " + "substitutes become the majority. Those words are neither reported nor restored, so the rules that need them " + "miss, while the document as a whole is still reported. Closing this is a change of behaviour, so it was not " + "made here: it needs its own measurement against human text, because a Greek or Cyrillic word is sometimes " + "exactly what the writer meant.") + w("") + w("### For scale: RAID's leaderboard") + w("") + skipped = ", ".join(f"`{x['file']}` ({x['reason']})" for x in lb.get("skipped", [])) + w(f"Measured on RAID-test, read from the leaderboard's files at commit `{lb['commit'][:7]}`. The files were " + f"{lb.get('selection', 'selected by hand')}; the list is in `leaderboard-scale.json`" + f"{', and ' + skipped + ' was skipped' if skipped else ''}. These are different texts and a different mix: " + "this sample weights the 11 generators equally, while RAID weights generators run with and without a " + f"repetition penalty twice (re-weighted to RAID's mix, the rate above is {pct(r['raidMixTpr05'])}). So they set " + "a scale and are not rows in the same table as the numbers above. Nothing from this project has been " + "submitted.") + w("") + w("| Detector, run by RAID's authors | No attack | Homoglyph | Zero-width | Whitespace |") + w("|---|---|---|---|---|") + for d in lb["detectors"]: + if d["submittedBy"] != "RAID authors": + continue + t = d["tpr05"] + w(f"| {d['name']} | {pct(t['none'])} | {pct(t['homoglyph'])} | {pct(t['zero_width_space'])} | {pct(t['whitespace'])} |") + w("") + others = [d["tpr05"]["none"] for d in lb["detectors"] if d["submittedBy"] != "RAID authors"] + if others: + w(f"Entries submitted by others sit between {pct(min(others))} and {pct(max(others))}. No result on this " + f"benchmark, theirs or the {pct(overall)} above, says how a detector does on other text.") + w("") + w("Under the zero-width attack Binoculars and GLTR rise to about 98%. RAID sets its cutoff on unattacked human " + "text only (`find_threshold` in RAID's `raid/evaluate.py`), so a detector whose score rises on any text " + "carrying these characters gains detection and is charged nothing for it. Whether that is what happens with " + "these two detectors was not tested here. This tool's scanner reports the attacked human files exactly as " + "often as the machine ones, and keeps the finding out of the score.") + w("") + w("## Where this departs from the plan") + w("") + w(f"- **All {s['human']:,} human texts** were used instead of 500 per domain. More human text makes the cutoff " + "steadier, and it cannot be tuned in anyone's favour.") + dn = r["detectedNotEnglish"] + w(f"- **Language fixed to English**, as the calibration does. The web app detects the language instead and would " + f"have misread {pct(dn['human'])} of the human texts and {pct(dn['machine'])} of the machine ones.") + w("- **Scores divided by 100** for RAID's threshold search, which assumes a 0–1 range.") + w("- **Both cutoffs reported**: RAID's per-domain cutoff first, then one cutoff for all domains.") + ht = r["humanTexts"] + w(f"- **Repeated texts.** RAID-train's human set holds {ht['distinct']:,} distinct texts in {ht['rows']:,} rows. " + "The headline counts rows, as RAID does; the long-text check counts each distinct text once.") + w(f"- **Generator mix.** The sample weights the 11 generators equally; RAID does not. Re-weighted to RAID's " + f"proportions the headline is {pct(r['raidMixTpr05'])} instead of {pct(overall)}.") + w("") + w("## What this does not tell you") + w("") + w(f"- **What the product does on long machine text.** No machine text in the sample reaches " + f"{prod['lengthFloorWords']} words, where the product starts giving verdicts.") + w("- **Anything about current models.** RAID's generators are from 2023. Newer models write differently and the " + "tells move with them. The right response to \"a newer model writes differently\" is to measure it.") + w("- **A leaderboard result.** This is a sample of RAID-train, not RAID-test, and nothing has been submitted.") + w("- **Anything about students.** RAID's human texts are published writing. The effect on second-language " + "learners (#75) is invisible here, which is what the calibration corpus is for.") + w("- **A cutoff tuned on other text.** RAID chooses each cutoff on the same human texts it then counts against. " + "That is its method, and it favours every detector equally.") + w("- **Robustness to rewriting.** Paraphrase, synonym swap and misspelling were not run.") + w("- **More than one run.** One seed, one sample. Intervals are given only where shown.") + w("- **An independent author.** The same model that wrote the rule packs wrote these scripts and this page. The " + "defence is that the counting is RAID's code, not ours, and that the predictions were committed first. The " + "page was reviewed adversarially before merging; the reviews are not published.") + w("") + w("## Re-run it yourself") + w("") + w("See [tools/raid/README.md](../tools/raid/README.md). In short: download `train_none.csv`, draw the sample with " + "`tools/raid/sample.py`, score it with `signsofai-calibrate raid`, count it with `tools/raid/evaluate.py`, " + "attack it with `tools/raid/attack.py`, and write this page with `tools/raid/report.py`. The id fingerprint " + "above says whether you drew the same sample.") + w("") + return "\n".join(L) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--dir", default="Docs/Detection") + ap.add_argument("--commit", required=True, help="engine commit the scores were produced at") + ap.add_argument("--run-date", required=True) + ap.add_argument("--out", default="Docs/DETECTION.md") + args = ap.parse_args() + d = Path(args.dir) + load = lambda n: json.loads((d / n).read_text(encoding="utf-8")) + page = build(load("results.json"), load("attacks.json"), load("leaderboard-scale.json"), args.commit, args.run_date) + Path(args.out).write_text(page, encoding="utf-8", newline="\n") + print(f"wrote {args.out} ({len(page.splitlines())} lines)") + + +if __name__ == "__main__": + main() diff --git a/tools/raid/requirements.txt b/tools/raid/requirements.txt new file mode 100644 index 0000000..5c3ff4e --- /dev/null +++ b/tools/raid/requirements.txt @@ -0,0 +1,5 @@ +# Python 3.12. raid-bench itself is installed from RAID's repository (see README.md), +# which pins numpy 1.26, pandas 2.2 and scikit-learn 1.3. +nltk==3.10.4 +regex==2026.11.15 +requests==2.34.2 diff --git a/tools/raid/sample.py b/tools/raid/sample.py new file mode 100644 index 0000000..7c68e0c --- /dev/null +++ b/tools/raid/sample.py @@ -0,0 +1,92 @@ +"""Draw the RAID sample this project measures, and fingerprint it. + +Every human text in RAID-train without attacks, and a fixed number of machine-written texts for each +generator and domain, chosen with a fixed seed. Writes: + + /sample.jsonl {id, text} per line, for `signsofai-calibrate raid` + /sample.csv the same rows with RAID's metadata and no text, for evaluation + /sample.json what was drawn and from what: dataset hash, seed, counts, id fingerprint + +The texts are not redistributed. Anyone with the same dataset file and this script gets the same ids, +and the fingerprint says whether they did. +""" + +import argparse +import hashlib +import json +from pathlib import Path + +import pandas as pd + +META = ["id", "model", "decoding", "repetition_penalty", "attack", "domain"] + + +def sha256_file(path: Path) -> str: + h = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--data", required=True, help="RAID train_none.csv") + ap.add_argument("--out", required=True, help="directory for the sample files") + ap.add_argument("--per-cell", type=int, default=200, help="machine texts per generator and domain") + ap.add_argument("--seed", type=int, default=20261117) + args = ap.parse_args() + + data = Path(args.data) + out = Path(args.out) + out.mkdir(parents=True, exist_ok=True) + + df = pd.read_csv(data) + if set(df["attack"].unique()) != {"none"}: + raise SystemExit("Expected RAID-train without attacks (train_none.csv).") + + human = df[df["model"] == "human"] + machine = df[df["model"] != "human"] + + # Sorting before sampling makes the draw depend on the rows, not on the order the file happens + # to be in. + machine = machine.sort_values("id") + drawn = ( + machine.groupby(["model", "domain"], group_keys=False) + .apply(lambda g: g.sample(n=min(args.per_cell, len(g)), random_state=args.seed)) + ) + + sample = pd.concat([human, drawn]).sort_values("id").reset_index(drop=True) + if sample["id"].duplicated().any(): + raise SystemExit("Duplicate ids in the sample.") + + with open(out / "sample.jsonl", "w", encoding="utf-8", newline="\n") as f: + for row in sample.itertuples(index=False): + f.write(json.dumps({"id": row.id, "text": row.generation}, ensure_ascii=False) + "\n") + + sample[META].to_csv(out / "sample.csv", index=False) + + fingerprint = hashlib.sha256("\n".join(sample["id"]).encode("utf-8")).hexdigest() + manifest = { + "dataset": "RAID-train without attacks (train_none.csv)", + "datasetUrl": "https://dataset.raid-bench.xyz/train_none.csv", + "datasetSha256": sha256_file(data), + "seed": args.seed, + "machinePerGeneratorAndDomain": args.per_cell, + "human": int(len(human)), + "machine": int(len(drawn)), + "total": int(len(sample)), + "humanByDomain": {k: int(v) for k, v in human["domain"].value_counts().sort_index().items()}, + "machineByModel": {k: int(v) for k, v in drawn["model"].value_counts().sort_index().items()}, + # RAID's own mix, which this sample does not follow: it weights generators run with and without + # a repetition penalty twice. Kept so a rate can be re-weighted to RAID's proportions. + "datasetMachineByModel": {k: int(v) for k, v in machine["model"].value_counts().sort_index().items()}, + "idFingerprintSha256": fingerprint, + "pandas": pd.__version__, + } + (out / "sample.json").write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + print(json.dumps(manifest, indent=2)) + + +if __name__ == "__main__": + main()