Repository navigation
feat: central value-based secret redaction in the log publishers - #1588
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
Conversation
…rosoft#1141) Add redactSecrets/redactLogData in packages/shared/src/logging with a curated set of high-confidence token patterns (PEM private keys, Azure AccountKey and SAS sig, Bearer tokens, JWTs, AWS key ids, GitHub, Slack, OpenAI/Anthropic, Google, Stripe, npm and Hugging Face tokens). Both LogPublisher and RedisLogPublisher scrub the message and nested data strings before publishing to Redis/SSE and appending to Blob. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wassim Chegham (manekinekko)
left a comment
There was a problem hiding this comment.
Thank you Naveen Chatlapalli (@1aifanatic) for taking on central log redaction. Moving this protection to the publisher boundary, retaining useful prefixes, and testing both Redis and blob output are valuable improvements.
I reviewed 2d593d3, including an independent security pass. Two gaps remain in the protection claimed for recognized bearer credentials and PEM private keys; the concrete cases and suggested remedies are inline. These are incomplete coverage of pre-existing logging paths, not a claim that this PR created the underlying exposure. I am requesting changes so those paths are not presented as protected when they are not.
| # | Severity | File | Lines | Vulnerability | Confidence |
|---|---|---|---|---|---|
| 1 | 🟡 MEDIUM | packages/shared/src/logging/redact-secrets.ts | 43, 61-62 | Valid lowercase/mixed-case bearer schemes bypass the matcher and fast path | 9/10 |
| 2 | 🟡 MEDIUM | packages/shared/src/logging/redact-secrets.ts | 38 | A private key split across forwarded subprocess stderr events survives intact | 9/10 |
Validation using Node 22.22.0:
pnpm exec vitest run packages/shared/src/logging/redact-secrets.test.ts packages/shared/src/logging/log-publisher.test.ts --reporter=default— 31 tests passed.pnpm --filter shared build— passed.- Two additional scratch cases reproduced lowercase bearer and separate-event PEM bypasses. The security pass also exercised the production argument formatter and a real subprocess stderr pipe with an ephemeral in-memory key; it confirmed the split output could reconstruct the key. No real credentials were used or published.
Please add regression coverage through the actual publisher/stream path, not only complete-string helper inputs, and keep the documented guarantee aligned with the resulting coverage. PII, generic high-entropy detection, worker stdout, and writeToolCalls JSONL are explicitly out of scope here; I am not asking this PR to solve those separate issues. There are no UI changes requiring Storybook or UI tests.
Thanks again for the useful defense-in-depth work.
| { id: "azure-storage-key", pattern: /(?<prefix>\bAccountKey=)[A-Za-z0-9+/]{40,}={0,2}/g }, | ||
| // SAS signature only — the rest of the blob URL stays readable for debugging. | ||
| { id: "azure-sas-signature", pattern: /(?<prefix>[?&]sig=)[A-Za-z0-9%+/=]{16,}/g }, | ||
| { id: "authorization-bearer", pattern: /(?<prefix>\bBearer\s+)[A-Za-z0-9._~+/-]{16,}=*/g }, |
There was a problem hiding this comment.
[P2 / medium] Match HTTP bearer schemes case-insensitively in both the rule and prefilter. HTTP authentication schemes are case-insensitive, but this regex and the substring fast path only recognize Bearer with that exact capitalization.
For example, redactSecrets("curl -H 'Authorization: bearer syntheticOpaqueCredential1234567890' https://example.invalid") leaves the entire synthetic credential visible; changing only bearer to Bearer redacts it. The production name-based argument filter also leaves this intact because the enclosing argument is command, not a sensitive-looking field name.
Please make both scheme detection and the fast-path condition case-insensitive while preserving the original prefix, and add lowercase/uppercase/mixed-case publisher regressions. Otherwise valid credentials with unrecognized value shapes still reach Redis/SSE and run.jsonl despite the new bearer protection.
| const SECRET_RULES: readonly SecretRule[] = [ | ||
| { | ||
| id: "private-key", | ||
| pattern: /-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z0-9 ]*PRIVATE KEY-----/g, |
There was a problem hiding this comment.
[P2 / medium] Cover PEM blocks split across the real log stream. This rule requires BEGIN, key material, and END in one string, but the ACP subprocess stderr callback forwards each pipe chunk separately (apps/workers/coder-acp-copilot/src/acp-client.ts:573-575). Independent writes of the three PEM lines therefore produce events with no redaction marker, and log readers can reconstruct the complete key.
This was reproduced using a real subprocess stderr pipe and an ephemeral in-memory Ed25519 key: three chunks, zero redaction markers, and the reconstructed key still parsed. Passing the same key as one string was redacted. This is the forwarded stderr -> publisher path, not the explicitly excluded worker stdout or tool-call JSONL sink.
Please add stream-aware PEM suppression at an appropriate pre-publication boundary, including markers split between chunks and incomplete blocks, and test that path with separately emitted lines. If that requires a separately scoped change, the coverage/guarantee must at least be made explicit rather than claiming private-key protection for this delivery path.
Refs #1141
Summary
This adds value-based secret redaction, applied once at the log boundary, so a token is caught wherever it appears. That includes inside a
commandargument, in a free-text message, or in a nesteddatafield, not only under a sensitive-looking key.What changed
packages/shared/src/logging/redact-secrets.ts, exported fromshared:redactSecrets(text)replaces every match with[REDACTED:<rule-id>]. It keeps a descriptive prefix where there is one (Bearer,AccountKey=,?sig=), so the line still shows what was removed and SAS blob URLs stay readable.redactLogData(value)makes a recursive copy of plain objects and arrays with every string redacted. It never mutates the input, leaves non-plain objects (Dates, etc.) alone, and is depth-limited to 8.LogPublisher.publishandRedisLogPublisher.publishnow build their event through onebuildRedactedLogEventhelper. Every line that reaches Redis (and so SSE) or therun.jsonlappend blob is scrubbed exactly once, whichever worker or service emitted it.formatToolArgs/formatToolContent) is unchanged and stays as defense-in-depth, as the issue suggests.Rule set (curated, high-confidence)
In the spirit of OWASP
secrets-patterns-db. Every rule needs a distinctive prefix or structure, so ordinary debug text such as UUIDs, git SHAs, plain blob URLs andtask-…identifiers isn't touched (there are tests for this):private-key-----BEGIN … PRIVATE KEY----- … -----END … PRIVATE KEY-----azure-storage-keyAccountKey=<base64>azure-sas-signature?sig=/&sig=valueauthorization-bearerBearer <token>(16+ chars)jwteyJ….eyJ….<sig>aws-access-key-idAKIA/ASIA/ABIA/ACCA+ 16github-token,github-fine-grained-patgh[pousr]_…,github_pat_…slack-tokenxox[abposr]-…openai-anthropic-keysk-…,sk-proj-…,sk-ant-…google-api-key,stripe-key,npm-token,huggingface-tokenAIza…,sk_live_…/rk_test_…,npm_…,hf_…Hot-path cost
A substring pre-filter (
ghp_,sk-,eyJ,Bearer, …) skips every regex for the common case of a line with no candidate.Decisions left open (why this is "Refs", not "Fixes")
writeToolCalls) and worker stdout.Docs
docs/research/realtime-data-flow.md(Key Behaviors).Test plan
npx vitest run packages/shared/src: 75 files, 1093 testsredact-secrets.test.ts: each rule under an innocent context, prefix preservation, multiple matches, false-positive guards, deep/immutable data walk, depth limitRedisLogPublishertsc --noEmitforpackages/shared🤖 Generated with Claude Code