Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Docs/CALIBRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,9 +89,9 @@ Across these groups the median score runs from 9.5 (**en-second-language-learner

## What the product does with this number

The tool speaks at **30/100** and nowhere else, taking the boundary from the table above rather than from anybody's judgement. Below it a document gets its score and the reason it gets nothing more: a low score is not evidence that a person wrote something, since a detector that detects nothing also returns a low score, and this project has deliberately never measured how much machine writing it catches. The boundary moves when this page moves — including upward if a larger corpus turns out to be less flattering.
The tool speaks at **30/100** and nowhere else, taking the boundary from the table above rather than from anybody's judgement. Below it a document gets its score and the reason it gets nothing more: a low score is not evidence that a person wrote something, since a detector that detects nothing also returns a low score, and on the public benchmark in DETECTION.md most machine-written text scores low as well. The boundary moves when this page moves — including upward if a larger corpus turns out to be less flattering.

Above it there is **one** verdict, not a scale of them. This corpus can place a boundary and can say nothing whatever about how much further past it a score has travelled: no text known to be human came close to the upper reaches, and grading "moderate" against "strong" would need machine-written text, which the opening of this page argues against collecting. Interfaces do shade a high score more urgently than a low one, and those shades are a display convention — they are not on this page because nothing measured them.
Above it there is **one** verdict, not a scale of them. This corpus can place a boundary and can say nothing whatever about how much further past it a score has travelled: no text known to be human came close to the upper reaches, and grading "moderate" against "strong" would need machine-written text, which the opening of this page argues against collecting; the benchmark in DETECTION.md was not used to set any band. Interfaces do shade a high score more urgently than a low one, and those shades are a display convention — they are not on this page because nothing measured them.

A language absent from the corpus entirely gets no verdict at all, whatever it scores. A language present but too thin to bound its own rate borrows this boundary and carries its own figure beside it, so the reader weighs the rate that was measured for the writing in front of them rather than the pooled one.

Expand Down Expand Up @@ -131,8 +131,8 @@ A rule near the top is not automatically wrong. Some tells genuinely appear in h

## What this does not tell you

- **Nothing about how much AI writing it catches.** That is the other half of the picture and it is not measured here, deliberately. A tool that flags nothing has a perfect false-positive rate.
- **Nothing about text unlike this corpus.** These are published articles and the essays of adult learners in a university English programme. A first-year essay by a native speaker is a different population again, and the rate on one does not transfer to the other. Calibrating on your own students' pre-2022 work is the fix, and the same tool does it.
- **Nothing about how much AI writing it catches.** That is the other half of the picture and it is not measured here, deliberately. A tool that flags nothing has a perfect false-positive rate. That half is measured separately, on a public benchmark nobody here assembled, in [DETECTION.md](DETECTION.md).
- **Nothing about text unlike this corpus.** These are published articles and the essays of adult learners in a university English programme. A first-year essay by a native speaker is a different population again, and the rate on one does not transfer to the other. Calibrating on your own students' pre-2022 work is the fix, and the same tool does it. The first outside check, on long human writing from a public benchmark, is in [DETECTION.md](DETECTION.md): in that one sample the boundary did not hold for the news articles, and why was not measured.
- **The affiliation groups are a proxy, not a fact.** Nobody's first language is recorded in a DOI. The learner group is the exception — its corpus records each writer's first language — which is why it exists. Elsewhere the manifest states the reasoning per text so it can be argued with; where it is wrong, the number moves.
- **Hashes prove what *this* run measured**, not that another person extracting the same articles would get identical text. They would not: PDF and HTML extraction differ. Reproducing this needs the extracted texts, not just the manifest.

Expand Down
6 changes: 6 additions & 0 deletions Docs/Calibration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ A **false-positive rate** needs only writing known to be human. It does not go s
the harm this category actually causes: detectors flag 61% of essays by non-native English speakers,
and not one of them publishes that figure about itself.

The other half, how much machine-written text the score catches, is measured in `Docs/DETECTION.md`
on RAID, a public benchmark nobody here assembled. It answers one of the two objections above: nobody
here assembled it. It does not answer the other: its models are from 2023, and the number ages. It
reports a detection rate beside this false-positive rate rather than an accuracy figure that blends
the two.

## What makes a text admissible

**It was written before 2022.** That is the whole test. A paper with a 2019 DOI predates the models
Expand Down
Loading
Loading