Skip to content

Percent-encoded delimiters defeat the leading \b on every secret token pattern, so a live PAT/JWT/AWS-key embedded in a URL-encoded URL (%3Dghp_…) survives history.Capture redaction and is written raw into the committed transcript #370

Description

@REPPL

Wrong assumption

Every bundled secret token pattern anchors its start on a leading \b (internal/adapter/scanner/patterns.go): \bghp_…, \bghs_/gho_/ghu_/ghr_…, \bgithub_pat_…, \bAKIA…, \bxox[baprs]-…, \bAIza…, \bsk_live_/sk_test_…, \bsk-ant-/sk-proj-/sk-svcacct-…, \beyJ….….…. RE2 \b is an ASCII word boundary. The implicit assumption is that a real leaked token is always immediately preceded by a non-word byte (=, :, ", /, space, newline), so the leading \b always holds.

That assumption breaks under percent-encoding. When a URL/JSON body is URL-encoded, every non-word delimiter becomes %XX whose final byte is a hex digit (a word char): =%3D, "%22, :%3A, /%2F, ?%3F. The token itself is made of unreserved URL characters and stays literal and in the clear. So the byte immediately before the literal token prefix is always [0-9A-Fa-f], \b fails, and the scanner's own regex — which would otherwise match the token verbatim — never fires. This is a validate-before-canonicalize gap: the redactor scans raw bytes and never percent-decodes.

Exact trigger (input → observed vs correct)

A session transcript line containing a percent-encoded URL that carries a real credential, e.g. an OAuth/redirect/magic-link:

...&redirect=%2Fauth%3Ftoken%3Dghp_<36+ alnum PAT>...
  • Observed: \bghp_[A-Za-z0-9]{36,} does not match (%3D leaves D, a word char, right before ghp_), so scanner.ScanText returns no finding, Redact changes nothing, and the raw live PAT is written into the committed (and typically pushed) transcript record.
  • Correct: the live token is detected and redacted; the store's documented guarantee — "a stored record can never contain a live secret" — holds.

Empirically verified against the actual compiled regexes (all return no match, while the plaintext token=ghp_… matches fine):

\bghp_[A-Za-z0-9]{36,}   vs  %3Dghp_<36 alnum>            → no match
                             %22ghp_… / %2Fghp_…          → no match
                             redirect%3Dghp_… (nested)    → no match
\bAKIA[0-9A-Z]{16}       vs  %3DAKIA<16>                  → no match
\beyJ…\.…\.…             vs  link%3Ftoken%3DeyJ….….…      → no match  (URL-encoded magic-link JWT)

Reachability (the two-stage fail-closed guard does NOT catch it)

internal/core/history/history.go:139-162Capture redacts raw session transcripts (attacker-influenceable content) at the write boundary:

  • text := string(raw) (line 139) — scanned raw; there is no percent-decode pre-pass anywhere in the scan pipeline (grep of scanner/history/lifeboat/launch is clean).
  • Stage one (140-141): ScanText + Redact — misses the encoded-delimiter token.
  • Stage one-and-a-half (150-155): a literal backstop, but it collapses only the caller's $HOME; secrets have no literal backstop.
  • Stage two / verify (159): re-scans the redacted text with the same detector, so the same leading-\b blind spot misses the token identically; blockingResidual finds nothing and the write proceeds.

The code comment at history.go:143-148 already acknowledges this exact failure mode — "a span that detector's trailing-boundary heuristic dropped would slip through both stages" — and only guards home paths against it. The same reachability applies to lifeboat pack and launch dry-run/ship, which redact planned bytes through the same scanner.

Why this is distinct from what's already fixed

internal/adapter/scanner/scanner.go:400-405 explicitly disclaims "a real secret preceded by content that is not itself a match found here" as an accepted \b-boundary limitation. The adjacency machinery (ledger iss-185 adjacencyProbe, iss-188 stolenJunctions) only recovers a token abutting a previously matched token. This report shows the accepted limitation is not merely inert "filler-text noise": percent-encoding is a systematic, deterministic, realistic instance of it that places a %XX hex byte before the literal token in any URL-encoded URL/JSON, turning an "accepted" gap into a reachable live-secret leak into a committed record. Not present in the issue tracker or the abcd ledger (the scanner ledger entries cover serialized cross-leak, adjacency, config-read, and degraded-scanner — none is about encoded delimiters).

Sibling sweep

Affected = every bundled token pattern whose prefix begins with a word char and carries a leading \b (patterns.go): github_pat (ghp_, line 64), github_server_token (ghs_, 69), github_oauth (gho_, 74), github_user_token (ghu_, 79), github_refresh (ghr_, 84), github_pat_finegrained (93), anthropic_key (sk-ant-, 109), openai_project_key (114), openai_service_account (119), aws_access_key (AKIA, 125), slack_token (xox, 137), google_api_key (AIza, 147), stripe_live_key (152), stripe_test_key (157), jwt_shaped (eyJ, 162). All 15 miss when a %XX delimiter precedes the token.

Not affected: pem_private_key (103) and rp_session_key (48) have no leading \b. The network patterns (net_ipv4/net_ipv6/net_mac) also carry a leading \b, but their separators (./:) are themselves encoded, changing the token shape — a separate, lesser concern worth a follow-up check, not part of this confirmed finding. The identity matchers (matchers.findings) are also cited in the scanner.go:400-405 disclaimer and should be checked for the same encoded-boundary interaction.

CWE anchor

CWE-180 (Incorrect Behavior Order: Validate Before Canonicalize) — the redactor scans before percent-decoding; CWE-176/CWE-20 (improper handling of an encoding / ASCII-only boundary); leaf impact CWE-312 / CWE-532 (sensitive information written into a stored file).

Adversarial validation (both lenses independently confirmed)

  • Reachability lens: "Reachability is real — history.go scans raw transcripts and stage-two re-scans with the same detector, so the missed token slips both stages; percent-encoding rewrites every delimiter into a %XX hex word-char while the token stays literal, so the leading \b can never hold — a systematic class (encoded OAuth/redirect/magic-link URLs), not a contrived string."
  • Correctness lens: "%3Dghp_REALTOKEN is a complete, valid, percent-decodable live credential the scanner's own regex would match but for the leading \b landing on the hex byte D; the false-positive objection only rules out the naive fix of dropping \b, not the defect's existence; against history's absolute guarantee that a stored record can never contain a live secret, writing a raw live JWT/PAT into the committed record is a genuine correctness defect."

Fix direction (one sentence)

Scan a percent-decoded copy of the text (or make the token start-boundary percent-encoding-aware) so an encoded delimiter cannot mask a literal token — without dropping the leading \b, which legitimately suppresses mid-blob false positives.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingneeds-fixvalidated bug awaiting a fixseverity:criticalsilent security/data-loss at a trust boundary

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions