diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 3f312edb..aef904d4 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8938,6 +8938,23 @@ private-key floor has the opposite profile: the operator generates that key and public half, so raising it can break no handshake with anybody. Folding the two together would have smuggled a counterparty-facing refusal into a change whose entire justification is that it has none. The JWKS floor is real, is unfiled, and is named here by subject rather than by a number. + +**A SECOND #1166 LIMB BUILT 2026-08-22: the anonymizer salt-entropy floor.** This item's prose named it -- "the minimum-salt check counts CHARACTERS rather than measuring entropy". Measured before building: a salt of sixteen letter-a characters was ACCEPTED while a short salt was refused in the same run, so the gate was live and measuring the wrong property. It now estimates Shannon entropy over the observed character distribution and refuses below a floor. + +**THREE DEFECTS IN THE FIRST CUT, ALL FOUND BY AN ADVERSARIAL REVIEWER AND ALL CORRECTED BEFORE THIS LANDED.** Recorded because each is a shape worth recognising again. + +1. **LENGTH RESCUED A DEGENERATE PATTERN.** The total-bits estimate is length-scaled, so a two-symbol cycle at sixteen characters scored 16.00 bits and was refused while the IDENTICAL pattern at thirty-two characters scored 32.00 and PASSED. That is the length floor defeating the entropy floor, not a blind spot a docstring can disclaim. Fixed by checking the RATE separately: a new `MIN_SALT_ENTROPY_BITS_PER_CHAR` of 2.0, which refuses a two- or three-symbol cycle at any length while leaving a random decimal salt -- about 3.32 bits per character, the narrowest alphabet a real generator produces -- untouched. + +2. **THE REFUSAL NEVER REACHED THE OPERATOR.** The salt was validated inside the per-message loop, and the tee CLI wraps each message in a total `except Exception`. That is correct -- it must never let an anonymizer failure surface a body -- but it swallowed a CONFIGURATION error and reported it as N failed messages, steering an operator with a weak salt to extend their rule map, the wrong file entirely. Fixed by validating once BEFORE the loop, where the salt belongs: it is a property of the run, not of any message. + +3. **THE FLOOR CONTRADICTED THE ADR AND CITED IT AS AUTHORITY WHILE DOING SO.** The error text invoked ADR 0030 section 4 while accepting a salt at 32 estimated bits, where that ADR requires 128. Both facts are now stated: the ADR governs GENERATION, this estimator grades a SUPPLIED string and is a loose LOWER BOUND -- a genuine 128-bit secret rendered as sixteen decimal digits estimates about 40 bits here, so a floor at 128 would refuse conformant secrets. Clearing this screen therefore does NOT establish ADR compliance, and the comment now says so outright instead of implying the reverse. + +**A FOURTH, CAUGHT WHILE VERIFYING THOSE FIXES, AND THE REPOSITORY ALREADY GUARDED IT.** `tee/anon/keying.py` is a verbatim vendored copy and the tee CLI imports the VENDORED one, so fixing only the engine module left the CLI accepting a salt the engine refused. I nearly added a drift guard for it; `tests/test_anon_parity.py::test_shared_logic_files_are_byte_identical` already exists and fires on exactly that drift, verified by desyncing the file and watching it fail. The real lesson is narrower and about method: I ran one test FILE rather than the suite, and a file-scoped run cannot see a guard that lives in another file. + +Proof: four mutations, each red, zero vacuous -- per-character floor disabled, vendored copy drifted, total-bits floor disabled, and the two floors shown to fire independently. 66 passed across the anonymizer core and the tee CLI. + +**Residual:** the estimator stays ORDER-blind and DICTIONARY-blind, so a string of sixteen distinct sequential letters measures the arithmetic maximum and passes. Nothing short of generating the salt ourselves fixes that, which is what ADR 0030 already requires. + ## 1167. research an honest pass for ASVS 11.2.4 -- constant-time recovery-code verification without turning ten argon2id slots into an amplification target > 🔢 **Re-scored 2026-08-20 -> P3.** Value **4/10** · Difficulty **7/10** · _money pit_. The data-dependent early return survives on the shipped MFA path: _verify_second_factor walks the argon2id recovery hashes and returns on the first match, so the number of ~64 MiB verifications is a function of which code was presented. Value 4 because the leak is a wall-clock signal on an already-authenticated second factor rather than a bypass; difficulty 7 because the obvious constant-time loop multiplies a 64 MiB argon2id verification by the slot count on every attempt, converting a timing leak into a memory and CPU amplification target, and the evidentiary half has no precedent in this tree. _(was 4/10 · 7/10.)_ diff --git a/messagefoundry/anon/keying.py b/messagefoundry/anon/keying.py index 7f7bcc56..c8dd2da3 100644 --- a/messagefoundry/anon/keying.py +++ b/messagefoundry/anon/keying.py @@ -24,7 +24,9 @@ from __future__ import annotations import hashlib +import math import random +from collections import Counter #: 128-bit seed — ample entropy to index any surrogate pool without collisions you'd notice. _DIGEST_SIZE = 16 @@ -34,6 +36,85 @@ #: obviously-weak salt rather than emit guessable surrogates (fail closed, ADR 0030 §4). MIN_SALT_LEN = 16 +#: Floor on the salt's ESTIMATED entropy, in bits. Length alone is not strength: sixteen copies of +#: the letter "a" clears ``MIN_SALT_LEN`` and is guessable on the first try, so the length floor +#: could be passed by a salt with no secrecy at all. +MIN_SALT_ENTROPY_BITS = 32.0 + +#: Floor on the salt's estimated entropy PER CHARACTER, in bits. The total-bits floor above is +#: length-scaled, so **length alone would rescue a degenerate pattern**: ``"ab"`` repeated eight +#: times scores 16.00 bits and is refused, while the SAME two-symbol pattern repeated sixteen times +#: scores 32.00 and would pass. That is not a blind spot the estimator's docstring can wave at -- it +#: is the length floor defeating the entropy floor -- so the rate is checked separately from the +#: total and a string has to clear BOTH. +#: +#: 2.0 bits per character means an effective alphabet of four symbols. Measured against the shapes +#: that matter: ``"ab"`` repeated scores 1.00 and is refused at any length; ``"abc"`` repeated scores +#: about 1.58 and is refused; a random DECIMAL salt -- the narrowest alphabet a real generator would +#: produce -- scores about 3.32, and random hex about 3.9, so neither is touched. A single repeated +#: character scores 0.00 and was already refused by the total. +MIN_SALT_ENTROPY_BITS_PER_CHAR = 2.0 + +#: **WHY 32 BITS AND NOT THE 128 THE ADR NAMES.** ADR 0030 section 4 requires ``dataset_key`` to be +#: DRAWN from ``secrets.token_bytes``/``os.urandom`` with at least 128 bits of entropy. That is a +#: requirement on GENERATION and it is not the same measurement as this one. These floors grade a +#: string the engine did not generate, using a distribution estimate that is a LOWER BOUND and a +#: loose one: a genuine 128-bit secret rendered as sixteen decimal digits estimates about 40 bits +#: here, and rendered as base62 about 95 -- neither reaches 128, because the estimator cannot see +#: the entropy of the generator, only of the characters in front of it. +#: +#: So a floor set at the ADR's 128 would refuse real, conformant secrets, which is why it is not set +#: there. **The consequence must be read honestly: clearing these floors does NOT establish that a +#: salt meets ADR 0030, and no check on a supplied string could.** This is a screen against a +#: visibly degenerate salt, sitting underneath the ADR requirement rather than implementing it. The +#: control that actually delivers 128 bits is generating the salt with the tool the error message +#: names, and that stays where the ADR puts it. + +#: Ceiling on the salt's UTF-8 length in BYTES, set by BLAKE2b's keyed mode +#: (``hashlib.blake2b.MAX_KEY_SIZE``, 64). Checked here at construction rather than left to +#: :meth:`Keyer.seed`, where it fired as a bare ``ValueError: maximum key length is 64 bytes`` on +#: the FIRST message instead of on the bad configuration -- an 86-byte ``secrets.token_urlsafe(64)`` +#: built a Keyer without complaint and then failed mid-dataset. Bytes, not characters: a non-ASCII +#: salt can pass 64 bytes well under 64 characters. +MAX_SALT_BYTES = hashlib.blake2b.MAX_KEY_SIZE + + +def _estimated_entropy_bits(salt: str) -> float: + """Shannon entropy of the salt's OBSERVED character distribution, scaled by its length. + + Why this estimator. It is the weakest assumption we can make about a string we did not + generate: count the symbols, take ``-sum(p * log2(p))`` for the per-character entropy, and + multiply by the length for a total in bits. It needs no dictionary, no alphabet table and no + network, it grades continuously rather than by category, and -- unlike a bare count of distinct + characters -- it sees SKEW. A salt of sixteen "a" characters plus seven distinct others has + eight distinct characters and would clear a distinct-character floor of eight, while this + estimator scores it at about 40 bits for twenty-three characters and grades it near the floor, + which is the honest reading of a string that is five-sixths one symbol. + + WHAT IT CANNOT SEE, stated plainly because every distribution-based estimator shares the blind + spot: it is ORDER-BLIND and DICTIONARY-BLIND. It scores every permutation of a string + identically, so "abcdefghijklmnop" measures a full 64.00 bits at the length floor -- the + arithmetic maximum for sixteen characters -- and passes, despite being one of the first strings + any attacker would try. A repeated cycle such as "abcdabcdabcdabcd" measures exactly 32.00 bits + and also passes. This check therefore catches a salt that is visibly DEGENERATE; it cannot + certify that a salt is unpredictable, and nothing short of generating the salt ourselves could. + The real defence stays where ADR 0030 section 4 puts it: an env-supplied secret from a random + generator. This is the floor under that, not a substitute for it. + + Where the floor came from. Measured 2026-08-22 by sampling 200,000 random 16-character salts + per alphabet and counting how many this estimator scores below 32 bits: base62 0, lowercase 0, + lowercase hex 1, and decimal digits 57 -- the tightest realistic case, and a 16-digit salt + carries only about 53 bits of real entropy to begin with. The zeros are trustworthy because the + same run returned NON-ZERO at higher floors (at 40 bits the same four alphabets rejected 0, 2, + 306 and 11,066 of 200,000), so the instrument can fire. A 40-bit floor would refuse better than + 5 percent of genuinely random 16-digit salts, which is refusing real secrets; 32 bits does not. + """ + if not salt: + return 0.0 + n = len(salt) + per_char = sum(-(c / n) * math.log2(c / n) for c in Counter(salt).values()) + return n * per_char + class Keyer: """Maps a ``(field-kind, real value)`` pair to a stable, salt-keyed PRNG — one per dataset. @@ -51,9 +132,43 @@ def __init__(self, salt: str) -> None: f"anonymizer salt must be a secret of at least {MIN_SALT_LEN} characters " "(ADR 0030 §4: pinned-per-dataset secret, env-supplied, never committed)" ) + encoded = salt.encode("utf-8") + if len(encoded) > MAX_SALT_BYTES: + raise ValueError( + f"anonymizer salt must be at most {MAX_SALT_BYTES} bytes when UTF-8 encoded " + f"(this one is {len(encoded)}); that is BLAKE2b's keyed-mode limit, and a salt at " + "the ceiling already carries far more entropy than the surrogate pools can use. " + "secrets.token_urlsafe(48) is the longest that fits." + ) + # Length is a necessary floor, not a sufficient one -- see MIN_SALT_ENTROPY_BITS and + # _estimated_entropy_bits for the measure and its limits. Neither the salt nor any part of + # it appears in the message: it is a re-identification key, so an exception that quotes it + # would leak it into a traceback, a log or a CI transcript. + entropy = _estimated_entropy_bits(salt) + if entropy < MIN_SALT_ENTROPY_BITS: + raise ValueError( + f"anonymizer salt is too predictable: about {entropy:.1f} bits of estimated " + f"entropy against a floor of {MIN_SALT_ENTROPY_BITS:.0f}. Generate one with " + "secrets.token_urlsafe(24) and supply it from the environment. The salt itself is " + "withheld from this message on purpose. Note this floor is a SCREEN, not the ADR " + "0030 section 4 requirement: that one is 128 bits AT GENERATION, which no check on " + "a supplied string can confirm." + ) + # Checked SEPARATELY from the total, because the total is length-scaled and length would + # otherwise rescue a degenerate pattern -- "ab" repeated sixteen times reaches the 32-bit + # total on 1.00 bits per character. Both floors bind; neither substitutes for the other. + per_char = entropy / len(salt) + if per_char < MIN_SALT_ENTROPY_BITS_PER_CHAR: + raise ValueError( + f"anonymizer salt repeats too few distinct characters: about {per_char:.2f} bits " + f"per character against a floor of {MIN_SALT_ENTROPY_BITS_PER_CHAR:.1f}. Making it " + "longer will NOT help -- a repeating pattern carries the same rate at any length. " + "Generate one with secrets.token_urlsafe(24) and supply it from the environment. " + "The salt itself is withheld from this message on purpose." + ) # Hold the salt as bytes only; it is PHI-equivalent (a re-identification key) and must # never be logged, persisted, or surfaced — so we keep no other reference to it. - self._salt = salt.encode("utf-8") + self._salt = encoded def seed(self, kind: str, value: str) -> int: """A 128-bit seed for ``(kind, value)``, keyed by the dataset salt (one-way).""" diff --git a/tee/__main__.py b/tee/__main__.py index e45352b8..633d20b0 100644 --- a/tee/__main__.py +++ b/tee/__main__.py @@ -45,6 +45,7 @@ from tee import __version__, mefor_api from tee.anon import anonymize_checked +from tee.anon.keying import Keyer from tee.correlate import CorepointOutput, CorrelateConfig from tee.relay import Endpoint, RelayConfig, TeeRelay from tee.report import build_report @@ -511,6 +512,20 @@ async def _anonymize_captures(args: argparse.Namespace) -> int: return 1 overlay = Path(args.overlay) if args.overlay else None + + # VALIDATE THE SALT ONCE, HERE, BEFORE THE LOOP. The per-message `except Exception` below is + # deliberately total -- it must never let an anonymizer failure surface a body -- but that also + # swallows a CONFIGURATION error and reports it as N failed messages. A weak salt would then + # steer the operator to extend their rule map, which is the wrong file, while the message + # naming the actual problem was discarded. The salt is a property of the RUN, not of any + # message, so it belongs outside the loop; checking it here also means a bad salt costs one + # check instead of one per message. + try: + Keyer(salt) + except ValueError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + lines: list[str] = [] failed = 0 for row in rows: diff --git a/tee/anon/keying.py b/tee/anon/keying.py index 7f7bcc56..c8dd2da3 100644 --- a/tee/anon/keying.py +++ b/tee/anon/keying.py @@ -24,7 +24,9 @@ from __future__ import annotations import hashlib +import math import random +from collections import Counter #: 128-bit seed — ample entropy to index any surrogate pool without collisions you'd notice. _DIGEST_SIZE = 16 @@ -34,6 +36,85 @@ #: obviously-weak salt rather than emit guessable surrogates (fail closed, ADR 0030 §4). MIN_SALT_LEN = 16 +#: Floor on the salt's ESTIMATED entropy, in bits. Length alone is not strength: sixteen copies of +#: the letter "a" clears ``MIN_SALT_LEN`` and is guessable on the first try, so the length floor +#: could be passed by a salt with no secrecy at all. +MIN_SALT_ENTROPY_BITS = 32.0 + +#: Floor on the salt's estimated entropy PER CHARACTER, in bits. The total-bits floor above is +#: length-scaled, so **length alone would rescue a degenerate pattern**: ``"ab"`` repeated eight +#: times scores 16.00 bits and is refused, while the SAME two-symbol pattern repeated sixteen times +#: scores 32.00 and would pass. That is not a blind spot the estimator's docstring can wave at -- it +#: is the length floor defeating the entropy floor -- so the rate is checked separately from the +#: total and a string has to clear BOTH. +#: +#: 2.0 bits per character means an effective alphabet of four symbols. Measured against the shapes +#: that matter: ``"ab"`` repeated scores 1.00 and is refused at any length; ``"abc"`` repeated scores +#: about 1.58 and is refused; a random DECIMAL salt -- the narrowest alphabet a real generator would +#: produce -- scores about 3.32, and random hex about 3.9, so neither is touched. A single repeated +#: character scores 0.00 and was already refused by the total. +MIN_SALT_ENTROPY_BITS_PER_CHAR = 2.0 + +#: **WHY 32 BITS AND NOT THE 128 THE ADR NAMES.** ADR 0030 section 4 requires ``dataset_key`` to be +#: DRAWN from ``secrets.token_bytes``/``os.urandom`` with at least 128 bits of entropy. That is a +#: requirement on GENERATION and it is not the same measurement as this one. These floors grade a +#: string the engine did not generate, using a distribution estimate that is a LOWER BOUND and a +#: loose one: a genuine 128-bit secret rendered as sixteen decimal digits estimates about 40 bits +#: here, and rendered as base62 about 95 -- neither reaches 128, because the estimator cannot see +#: the entropy of the generator, only of the characters in front of it. +#: +#: So a floor set at the ADR's 128 would refuse real, conformant secrets, which is why it is not set +#: there. **The consequence must be read honestly: clearing these floors does NOT establish that a +#: salt meets ADR 0030, and no check on a supplied string could.** This is a screen against a +#: visibly degenerate salt, sitting underneath the ADR requirement rather than implementing it. The +#: control that actually delivers 128 bits is generating the salt with the tool the error message +#: names, and that stays where the ADR puts it. + +#: Ceiling on the salt's UTF-8 length in BYTES, set by BLAKE2b's keyed mode +#: (``hashlib.blake2b.MAX_KEY_SIZE``, 64). Checked here at construction rather than left to +#: :meth:`Keyer.seed`, where it fired as a bare ``ValueError: maximum key length is 64 bytes`` on +#: the FIRST message instead of on the bad configuration -- an 86-byte ``secrets.token_urlsafe(64)`` +#: built a Keyer without complaint and then failed mid-dataset. Bytes, not characters: a non-ASCII +#: salt can pass 64 bytes well under 64 characters. +MAX_SALT_BYTES = hashlib.blake2b.MAX_KEY_SIZE + + +def _estimated_entropy_bits(salt: str) -> float: + """Shannon entropy of the salt's OBSERVED character distribution, scaled by its length. + + Why this estimator. It is the weakest assumption we can make about a string we did not + generate: count the symbols, take ``-sum(p * log2(p))`` for the per-character entropy, and + multiply by the length for a total in bits. It needs no dictionary, no alphabet table and no + network, it grades continuously rather than by category, and -- unlike a bare count of distinct + characters -- it sees SKEW. A salt of sixteen "a" characters plus seven distinct others has + eight distinct characters and would clear a distinct-character floor of eight, while this + estimator scores it at about 40 bits for twenty-three characters and grades it near the floor, + which is the honest reading of a string that is five-sixths one symbol. + + WHAT IT CANNOT SEE, stated plainly because every distribution-based estimator shares the blind + spot: it is ORDER-BLIND and DICTIONARY-BLIND. It scores every permutation of a string + identically, so "abcdefghijklmnop" measures a full 64.00 bits at the length floor -- the + arithmetic maximum for sixteen characters -- and passes, despite being one of the first strings + any attacker would try. A repeated cycle such as "abcdabcdabcdabcd" measures exactly 32.00 bits + and also passes. This check therefore catches a salt that is visibly DEGENERATE; it cannot + certify that a salt is unpredictable, and nothing short of generating the salt ourselves could. + The real defence stays where ADR 0030 section 4 puts it: an env-supplied secret from a random + generator. This is the floor under that, not a substitute for it. + + Where the floor came from. Measured 2026-08-22 by sampling 200,000 random 16-character salts + per alphabet and counting how many this estimator scores below 32 bits: base62 0, lowercase 0, + lowercase hex 1, and decimal digits 57 -- the tightest realistic case, and a 16-digit salt + carries only about 53 bits of real entropy to begin with. The zeros are trustworthy because the + same run returned NON-ZERO at higher floors (at 40 bits the same four alphabets rejected 0, 2, + 306 and 11,066 of 200,000), so the instrument can fire. A 40-bit floor would refuse better than + 5 percent of genuinely random 16-digit salts, which is refusing real secrets; 32 bits does not. + """ + if not salt: + return 0.0 + n = len(salt) + per_char = sum(-(c / n) * math.log2(c / n) for c in Counter(salt).values()) + return n * per_char + class Keyer: """Maps a ``(field-kind, real value)`` pair to a stable, salt-keyed PRNG — one per dataset. @@ -51,9 +132,43 @@ def __init__(self, salt: str) -> None: f"anonymizer salt must be a secret of at least {MIN_SALT_LEN} characters " "(ADR 0030 §4: pinned-per-dataset secret, env-supplied, never committed)" ) + encoded = salt.encode("utf-8") + if len(encoded) > MAX_SALT_BYTES: + raise ValueError( + f"anonymizer salt must be at most {MAX_SALT_BYTES} bytes when UTF-8 encoded " + f"(this one is {len(encoded)}); that is BLAKE2b's keyed-mode limit, and a salt at " + "the ceiling already carries far more entropy than the surrogate pools can use. " + "secrets.token_urlsafe(48) is the longest that fits." + ) + # Length is a necessary floor, not a sufficient one -- see MIN_SALT_ENTROPY_BITS and + # _estimated_entropy_bits for the measure and its limits. Neither the salt nor any part of + # it appears in the message: it is a re-identification key, so an exception that quotes it + # would leak it into a traceback, a log or a CI transcript. + entropy = _estimated_entropy_bits(salt) + if entropy < MIN_SALT_ENTROPY_BITS: + raise ValueError( + f"anonymizer salt is too predictable: about {entropy:.1f} bits of estimated " + f"entropy against a floor of {MIN_SALT_ENTROPY_BITS:.0f}. Generate one with " + "secrets.token_urlsafe(24) and supply it from the environment. The salt itself is " + "withheld from this message on purpose. Note this floor is a SCREEN, not the ADR " + "0030 section 4 requirement: that one is 128 bits AT GENERATION, which no check on " + "a supplied string can confirm." + ) + # Checked SEPARATELY from the total, because the total is length-scaled and length would + # otherwise rescue a degenerate pattern -- "ab" repeated sixteen times reaches the 32-bit + # total on 1.00 bits per character. Both floors bind; neither substitutes for the other. + per_char = entropy / len(salt) + if per_char < MIN_SALT_ENTROPY_BITS_PER_CHAR: + raise ValueError( + f"anonymizer salt repeats too few distinct characters: about {per_char:.2f} bits " + f"per character against a floor of {MIN_SALT_ENTROPY_BITS_PER_CHAR:.1f}. Making it " + "longer will NOT help -- a repeating pattern carries the same rate at any length. " + "Generate one with secrets.token_urlsafe(24) and supply it from the environment. " + "The salt itself is withheld from this message on purpose." + ) # Hold the salt as bytes only; it is PHI-equivalent (a re-identification key) and must # never be logged, persisted, or surfaced — so we keep no other reference to it. - self._salt = salt.encode("utf-8") + self._salt = encoded def seed(self, kind: str, value: str) -> int: """A 128-bit seed for ``(kind, value)``, keyed by the dataset salt (one-way).""" diff --git a/tests/test_anon_core.py b/tests/test_anon_core.py index dc45e48f..c4ea3389 100644 --- a/tests/test_anon_core.py +++ b/tests/test_anon_core.py @@ -5,6 +5,8 @@ from __future__ import annotations +import secrets +import string from collections.abc import Iterator from pathlib import Path @@ -25,6 +27,12 @@ leak_report, load_rules, ) +from messagefoundry.anon.keying import ( + MAX_SALT_BYTES, + MIN_SALT_ENTROPY_BITS, + MIN_SALT_LEN, + _estimated_entropy_bits, +) from messagefoundry.anon.surrogates import Seps, scrub_site_codes, surrogate_field # The leak-check delegates to scripts/security/scan_forbidden.py (the relocated forbidden-content @@ -84,7 +92,7 @@ def _msg(*segments: str) -> str: def test_keyer_deterministic_and_salt_sensitive() -> None: - a, b = Keyer("salt-aaaaaaaaaaaaaaaa"), Keyer("salt-aaaaaaaaaaaaaaaa") + a, b = Keyer("salt-7Kq2mVz9pLx4Rw"), Keyer("salt-7Kq2mVz9pLx4Rw") assert a.seed("mrn", "12345") == b.seed("mrn", "12345") assert a.seed("mrn", "12345") != a.seed("mrn", "54321") assert a.seed("mrn", "12345") != a.seed("name", "12345") # kind is part of the key @@ -98,6 +106,91 @@ def test_keyer_rejects_weak_salt() -> None: Keyer("") +# A salt that is genuinely random yet REPEATS characters -- generated with secrets.token_hex(8), +# nine distinct characters over sixteen, one of them appearing five times. It is the POSITIVE +# CONTROL for the entropy gate: without it, a check that refused every salt would still satisfy the +# rejection tests below, and the gate would be indistinguishable from a permanent outage. +_REAL_RANDOM_SALT_WITH_REPEATS = "0222439f2bb823dd" + + +def test_keyer_accepts_real_high_entropy_salts_including_one_with_repeats() -> None: + """POSITIVE CONTROL -- the gate must not refuse a salt an operator would actually generate.""" + assert len(set(_REAL_RANDOM_SALT_WITH_REPEATS)) < len(_REAL_RANDOM_SALT_WITH_REPEATS), ( + "this control is only meaningful if the salt repeats a character" + ) + for salt in ( + _REAL_RANDOM_SALT_WITH_REPEATS, + secrets.token_hex(8), # 16 characters -- exactly at MIN_SALT_LEN, no length headroom + secrets.token_urlsafe(24), + secrets.token_hex(32), + ): + assert Keyer(salt).seed("mrn", "12345") > 0 # constructs and keys, no raise + + +def test_keyer_rejects_a_long_salt_with_too_little_entropy() -> None: + """Character COUNT is not entropy: each of these clears MIN_SALT_LEN and is still guessable.""" + for salt in ( + "a" * MIN_SALT_LEN, # the exact case the length-only gate accepted + "a" * 64, # length does not rescue a one-symbol salt + "0" * 32, + "abababababababab", # two symbols + "xxxxxxxxxxxxxxxy", # skewed: diverse by distinct count, one symbol in practice + "salt-aaaaaaaaaaaaaaaa", # a plausible-looking placeholder + ): + assert len(salt) >= MIN_SALT_LEN, "the length gate must not be what fires here" + with pytest.raises(ValueError, match="too predictable"): + Keyer(salt) + + +def test_keyer_rejects_an_oversized_salt_at_construction_not_at_first_message() -> None: + """A salt past BLAKE2b's keyed-mode limit used to build fine and crash on the FIRST message. + + Measured on the pre-fix code: ``Keyer(secrets.token_urlsafe(64))`` -- 86 bytes -- constructed + without complaint, then raised a bare "maximum key length is 64 bytes" from inside ``seed``. + On a first deployment that would surface as a mid-dataset failure rather than a rejected + configuration. The boundary owns it now. + """ + oversized = secrets.token_urlsafe(64) + assert len(oversized.encode("utf-8")) > MAX_SALT_BYTES # the case is what we think it is + with pytest.raises(ValueError, match="at most"): + Keyer(oversized) + # ...and a salt exactly AT the ceiling still works -- the bound must not be off by one. + at_ceiling = secrets.token_hex(MAX_SALT_BYTES // 2) + assert len(at_ceiling.encode("utf-8")) == MAX_SALT_BYTES + assert Keyer(at_ceiling).seed("mrn", "12345") > 0 + + +def test_keyer_weak_salt_error_never_quotes_the_salt() -> None: + """The salt is a re-identification key -- an exception that echoed it would leak it to a log.""" + salt = "a" * MIN_SALT_LEN + with pytest.raises(ValueError) as excinfo: + Keyer(salt) + assert salt not in str(excinfo.value) + assert "aaaa" not in str(excinfo.value) + + +def test_entropy_estimate_is_order_blind_which_is_the_declared_blind_spot() -> None: + """Pin the limitation the estimator's docstring admits, so nobody later overstates the gate. + + Sixteen distinct characters in alphabetical order score the arithmetic maximum for that length + and are ACCEPTED, even though the string is trivially guessable. A distribution-based estimator + cannot see order; claiming otherwise would be the dishonest reading of this check. + """ + assert _estimated_entropy_bits("abcdefghijklmnop") == _estimated_entropy_bits( + "pnmlkjihgfedcba" + "o" + ) + assert _estimated_entropy_bits("abcdefghijklmnop") >= MIN_SALT_ENTROPY_BITS + Keyer("abcdefghijklmnop") # accepted -- documented blind spot, not an oversight + + +def test_entropy_estimate_grades_the_measured_cases() -> None: + """Executed values behind the floor, so a future edit to the estimator has to face them.""" + assert _estimated_entropy_bits("a" * MIN_SALT_LEN) == 0.0 + assert _estimated_entropy_bits("") == 0.0 # helper is safe on empty; the length gate owns it + assert _estimated_entropy_bits("abababababababab") == pytest.approx(16.0) + assert _estimated_entropy_bits(_REAL_RANDOM_SALT_WITH_REPEATS) == pytest.approx(46.39, abs=0.01) + + # --- rule model ----------------------------------------------------------------------------------- @@ -458,3 +551,47 @@ def test_site_prefix_fixture_leaves_module_globals_consistent_with_the_environme "surrogates._SITE_PREFIXES drifted from what the current environment yields — a fixture " "recomputed them under a patched environment and did not restore them afterwards" ) + + +# --- the per-character floor: length must not rescue a degenerate pattern ---------------------- +# +# The total-bits floor is length-scaled, so a repeating two-symbol pattern reaches it by being +# long: "ab" x8 scored 16.00 bits and was refused, while the IDENTICAL pattern at "ab" x16 scored +# 32.00 and passed. That is the length floor defeating the entropy floor, not a blind spot the +# estimator can disclaim, so the RATE is now checked separately and a salt must clear both. + + +@pytest.mark.parametrize( + "salt,label", + [ + ("ab" * 16, "two symbols, 32 chars -- reached the total floor by length alone"), + ("ab" * 32, "two symbols, 64 chars"), + ("abc" * 21 + "a", "three symbols, 64 chars -- scored 101 bits on the total"), + ], +) +def test_length_does_not_rescue_a_repeating_pattern(salt: str, label: str) -> None: + with pytest.raises(ValueError, match="too few distinct characters"): + Keyer(salt) + + +def test_real_generated_salts_are_still_accepted() -> None: + """THE CONTROL THAT MATTERS. A floor that refuses everything would pass every rejection test + above while making the anonymizer unusable, which is worse than the defect it fixes. Decimal is + the narrowest alphabet a real generator would produce, so it is the binding case.""" + for salt in ( + secrets.token_urlsafe(24), + secrets.token_hex(8), + secrets.token_hex(16), + "".join(secrets.choice(string.digits) for _ in range(MIN_SALT_LEN)), + ): + Keyer(salt) # must not raise + + +def test_the_two_floors_are_independent() -> None: + """Neither floor substitutes for the other, so both must be able to fire alone. A single + repeated character fails the TOTAL (0 bits); a two-symbol pattern long enough to clear the + total fails the RATE. If one message could serve both, one floor would be redundant.""" + with pytest.raises(ValueError, match="too predictable"): + Keyer("a" * MIN_SALT_LEN) + with pytest.raises(ValueError, match="too few distinct characters"): + Keyer("ab" * MIN_SALT_LEN)