Adaptive challenge cost: price by suspicion, on the clock not the hash - #27
Merged
Conversation
…t the hash Difficulty was a near-constant: 4, or 5 from a datacenter address, or up to 6 under a high request rate. That is the wrong knob. Browser JS runs 1-3M hash/s and native code 100-500M/s, so difficulty 6 costs a native solver about a millisecond and a budget Android phone about sixteen seconds. Escalating there taxes the slowest legitimate devices and constrains nobody. Wall-clock does not have that property. Nobody can make less time pass, so a minimum challenge age caps how fast one source can mint tokens regardless of the hardware it brings. Each challenge now carries a signed minAgeMs, and that is where the escalation lands. clean or unknown difficulty 4, 1500ms (unchanged for everyone) 1-2 strong bot verdicts difficulty 4, 4000ms 3-5 difficulty 5, 8000ms 6+ difficulty 5, 15000ms Difficulty now caps at 5 rather than 6. A datacenter address no longer raises it at all — a real person on a corporate VPN or iCloud Private Relay was paying several seconds of blocked hashing for a shared IP — and raises the time floor instead, which a hosted scraper feels as reduced throughput and a person filling in a form does not feel at all. At the 1.5s baseline one address mints roughly 40 tokens a minute; at 15s, four. The client is told minAgeMs and waits it out, so an ordinary visitor pays the elevated cost as a short delay rather than as a worse score. That distinction matters most for anyone sharing an egress address with whatever earned the delay. A client that submits early is still scored, but the below-baseline case keeps its existing strength while the between-baseline-and-floor case is contributory: an older cached client does not know to wait and should not be read as automation for it. Suspicion is held per (site key, address) for 15 minutes, records only verdicts at or above 0.8, and stores nothing but timestamps. It is not Workstream F 10.1 cross-session correlation and must not grow into it — same shape and privacy class as the rate limiter beside it. Bench: human FPR 0.00% (0/126), agent TPR 97.33%, both unchanged. Two harness defects surfaced on the way, both invisible until a signal was keyed by source: Sample addresses were a hash of the sample id, putting 180 samples into a 508-address space. They collided constantly — 25 shared addresses, 24 of them putting an agent and a human on the same IP — so the corpus was manufacturing shared egress its labels never claimed. The first source-keyed signal read 12.7% FPR on a panel whose real answer was zero. Addresses are now assigned by position and the pool throws rather than wrapping. Runs also inherited server-side state from each other: one signal fired on 2/75 agents against a fresh server and 74/75 against one already benchmarked. Each run now namespaces its own site key. bench/package.json pointed npm test at lib/lib.test.js, which has never existed; CI runs npm test in server-node, so nothing invoked it. Replaced with real tests for the address pool and wired into the bench workflow. Also completes the HMAC hardening pass in server-node/index.js, which it had missed: token signatures were truncated to 64 bits on both generation and verification while Go, Python and the standalone server used the full digest. Tokens issued by an older library build will not verify after this upgrade.
Three failures, all the same shape: a file the build needs is not where the build looks. `.gitignore` had `bench/*.json` to drop replay output, which also dropped bench/package.json. The harness could not be installed from a fresh clone, its playwright dependency was unrecorded, and CI could not run anything through npm in that directory — which is also why bench's `npm test` pointed at a file that had never existed without anyone noticing. Narrowed to `bench/*results*.json`. Both the Node and Python Dockerfiles enumerated the modules their entrypoint needs, under a comment instructing the reader to keep the list in step. The list went out of step immediately: neither image had the new module and both crashed on startup. That is the same failure as upstream #23, and the instruction did not prevent it because nothing about a Dockerfile review makes a missing require visible. They now copy the source and drop test files, so the list cannot drift again. Verified by building all three images and starting them: each serves /health and a challenge carrying difficulty and minAgeMs.
node --test only handles glob patterns itself from Node 22; CI runs Node 20 and received the literal string. Unquoted, so the shell expands it on both.
Neither had ever run in CI. Node's are covered by the benchmark workflow, so
every check on a pull request was green while two of the three implementations
were unverified. Among the tests that had never once executed there:
- TestOnlyJA4TLSIsImplemented, the only thing standing between this MIT
project and a FoxIO License 1.1 module that cannot legally ship in it
- TestWeightsSumToOne, guarding the scoring weights invariant
- every Python test, on an implementation whose only CI coverage was a
container smoke test
The sync rule says a change lands in all three servers, which is worth little if
only one is checked. Go now runs vet and `go test -race`; Python installs
requirements.txt and runs discovery.
Discovery had to start working first. The Python tests are plain functions
collected by a decorator rather than TestCase methods, and `unittest discover`
could not see them: test_clientip and test_sitekeys guarded their runner under
__main__ and reported "Ran 0 tests" without complaint, while test_detection and
test_inputforensics ran at import and called sys.exit mid-discovery, surfacing as
errors against suites that passed. Either way the count was wrong — 12 tests and
two errors, where there are 43.
testkit.py now holds the registry, so the same functions run both ways: readable
per-test output from a single file, and visible to unittest for discovery and CI.
It also replaces four copies of the same runner block. Verified that an injected
failure fails through both paths and is attributed to the file that owns it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Difficulty was effectively constant — 4, or 5 from a datacenter address, or up to 6 under load. That is the wrong knob to escalate.
Browser JS runs 1–3M hash/s; native code 100–500M/s. Difficulty 6 costs a native solver about a millisecond and a budget Android phone about sixteen seconds. Raising it taxes the slowest legitimate devices and constrains nobody.
Wall-clock has no such asymmetry — nobody can make less time pass. Each challenge now carries a signed
minAgeMs, and that is where escalation lands.A clean visitor pays exactly what they paid before. At the 1.5s baseline one address mints roughly 40 tokens a minute; at 15s, four.
Difficulty now caps at 5 rather than 6, and a datacenter address no longer raises it at all — a real person on a corporate VPN or iCloud Private Relay was paying several seconds of blocked hashing for a shared IP. It raises the time floor instead, which a hosted scraper feels as reduced throughput and someone filling in a form does not feel at all.
Cost lands as latency, not as a score
The client is told
minAgeMsand waits it out before submitting, so an ordinary visitor absorbs an elevated cost as a short delay rather than a worse verdict. That matters most for anyone sharing an egress address with whatever earned the delay.A client that submits early is still scored, but the two cases are separated. Below the universal 1500ms baseline keeps its existing strength — nothing legitimate happens that fast. Between the baseline and the source's own floor is contributory only, because an older cached client does not know to wait and should not be read as automation for it.
Scope of the memory
Per (site key, address), 15-minute window, records only verdicts at or above 0.8, stores nothing but timestamps. Same shape and privacy class as the rate limiter beside it. It is not Workstream F §10.1 cross-session correlation — no behavioral vectors, no per-fingerprint history — and the module comment says so, because those two should not be conflated on the grounds that both are "server-side memory".
Measurement
Human FPR 0.00% (0/126), agent TPR 97.33% — both unchanged.
Two harness defects surfaced, both invisible until a signal was keyed by source.
Sample addresses collided. They were a hash of the sample id, putting 180 samples into a 508-address space. By the birthday bound: 25 shared addresses, 24 of them putting an agent and a human on the same IP. The corpus was manufacturing shared egress its labels never claimed, and the first source-keyed signal read 12.7% FPR on a panel whose real answer was zero. Addresses are now assigned by position, and the pool throws rather than wrapping.
Runs inherited each other's server state. One signal fired on 2/75 agents against a fresh server and 74/75 against one already benchmarked. Each run now namespaces its own site key.
The replayer deliberately still does not honour
minAgeMs. A replayer that did would satisfy the timing gate by construction and the panel could never observe it firing on a human — the same trap as pinning a property during normalization and then reporting that nothing disagrees with it.bench/package.jsonpointednpm testatlib/lib.test.js, which has never existed; CI runsnpm testinserver-node, so nothing invoked it. Replaced with real tests for the address pool and wired into the bench workflow.Also in here
server-node/index.jswas missed by the pass that removed HMAC truncation elsewhere: token signatures were cut to 64 bits on both generation and verification, while Go, Python and the standalone server used the full digest. Now full-length. Tokens issued by an older library build will not verify after upgrading.Verification
go vetclean, all tests pass (10 new)npm test, which it was not