Skip to content

feat: central value-based secret redaction in the log publishers - #1588

Open
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
microsoft:mainfrom
1aifanatic:feat/1141-log-secret-redaction
Open

Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
microsoft:mainfrom
1aifanatic:feat/1141-log-secret-redaction

Conversation

@1aifanatic

Copy link
Copy Markdown

Refs #1141

Summary

This adds value-based secret redaction, applied once at the log boundary, so a token is caught wherever it appears. That includes inside a command argument, in a free-text message, or in a nested data field, not only under a sensitive-looking key.

What changed

  • New packages/shared/src/logging/redact-secrets.ts, exported from shared:
    • redactSecrets(text) replaces every match with [REDACTED:<rule-id>]. It keeps a descriptive prefix where there is one (Bearer , AccountKey=, ?sig=), so the line still shows what was removed and SAS blob URLs stay readable.
    • redactLogData(value) makes a recursive copy of plain objects and arrays with every string redacted. It never mutates the input, leaves non-plain objects (Dates, etc.) alone, and is depth-limited to 8.
  • LogPublisher.publish and RedisLogPublisher.publish now build their event through one buildRedactedLogEvent helper. Every line that reaches Redis (and so SSE) or the run.jsonl append blob is scrubbed exactly once, whichever worker or service emitted it.
  • The workers' existing name-based filter (formatToolArgs / formatToolContent) is unchanged and stays as defense-in-depth, as the issue suggests.

Rule set (curated, high-confidence)

In the spirit of OWASP secrets-patterns-db. Every rule needs a distinctive prefix or structure, so ordinary debug text such as UUIDs, git SHAs, plain blob URLs and task-… identifiers isn't touched (there are tests for this):

Rule Shape
private-key -----BEGIN … PRIVATE KEY----- … -----END … PRIVATE KEY-----
azure-storage-key AccountKey=<base64>
azure-sas-signature ?sig= / &sig= value
authorization-bearer Bearer <token> (16+ chars)
jwt eyJ….eyJ….<sig>
aws-access-key-id AKIA / ASIA / ABIA / ACCA + 16
github-token, github-fine-grained-pat gh[pousr]_…, github_pat_…
slack-token xox[abposr]-…
openai-anthropic-key sk-…, sk-proj-…, sk-ant-…
google-api-key, stripe-key, npm-token, huggingface-token AIza…, sk_live_…/rk_test_…, npm_…, hf_…

Hot-path cost

A substring pre-filter (ghp_, sk-, eyJ, Bearer, …) skips every regex for the common case of a line with no candidate.

Decisions left open (why this is "Refs", not "Fixes")

  • PII (emails, SSNs, cards) and generic high-entropy detection aren't included. They depend on the threat-model and false-positive decisions the issue lists as open. The module is built so more rules can be added.
  • Full redaction vs. masking: I chose full redaction with a rule-id marker over partial masking (last 4 chars).
  • Not scrubbed by this layer: tool-call JSONL (writeToolCalls) and worker stdout.

Docs

  • docs/research/realtime-data-flow.md (Key Behaviors).

Test plan

  • npx vitest run packages/shared/src: 75 files, 1093 tests
    • new redact-secrets.test.ts: each rule under an innocent context, prefix preservation, multiple matches, false-positive guards, deep/immutable data walk, depth limit
    • new publisher tests: the Redis and Blob payloads are both redacted, and so is RedisLogPublisher
  • tsc --noEmit for packages/shared
  • Fake tokens in the tests are built by string concatenation, so no literal secret shapes are committed.

🤖 Generated with Claude Code

…rosoft#1141)

Add redactSecrets/redactLogData in packages/shared/src/logging with a
curated set of high-confidence token patterns (PEM private keys, Azure
AccountKey and SAS sig, Bearer tokens, JWTs, AWS key ids, GitHub, Slack,
OpenAI/Anthropic, Google, Stripe, npm and Hugging Face tokens). Both
LogPublisher and RedisLogPublisher scrub the message and nested data
strings before publishing to Redis/SSE and appending to Blob.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@manekinekko Wassim Chegham (manekinekko) added area: shared Shared package: types, DB models, and queue/blob/redis clients used by all apps. community-contribution Pull requests submitted by external community contributors. language: javascript Work involving JavaScript code, tooling, or dependencies. topic: observability Logging, metrics, tracing, and operational visibility. topic: security Security protections, vulnerabilities, and risk reduction. topic: testing Test coverage, test infrastructure, and validation quality. type: documentation Documentation additions, corrections, and improvements. type: enhancement New functionality or improvements to existing behavior. labels Nov 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Naveen Chatlapalli (@1aifanatic) for taking on central log redaction. Moving this protection to the publisher boundary, retaining useful prefixes, and testing both Redis and blob output are valuable improvements.

I reviewed 2d593d3, including an independent security pass. Two gaps remain in the protection claimed for recognized bearer credentials and PEM private keys; the concrete cases and suggested remedies are inline. These are incomplete coverage of pre-existing logging paths, not a claim that this PR created the underlying exposure. I am requesting changes so those paths are not presented as protected when they are not.

# Severity File Lines Vulnerability Confidence
1 🟡 MEDIUM packages/shared/src/logging/redact-secrets.ts 43, 61-62 Valid lowercase/mixed-case bearer schemes bypass the matcher and fast path 9/10
2 🟡 MEDIUM packages/shared/src/logging/redact-secrets.ts 38 A private key split across forwarded subprocess stderr events survives intact 9/10

Validation using Node 22.22.0:

  • pnpm exec vitest run packages/shared/src/logging/redact-secrets.test.ts packages/shared/src/logging/log-publisher.test.ts --reporter=default — 31 tests passed.
  • pnpm --filter shared build — passed.
  • Two additional scratch cases reproduced lowercase bearer and separate-event PEM bypasses. The security pass also exercised the production argument formatter and a real subprocess stderr pipe with an ephemeral in-memory key; it confirmed the split output could reconstruct the key. No real credentials were used or published.

Please add regression coverage through the actual publisher/stream path, not only complete-string helper inputs, and keep the documented guarantee aligned with the resulting coverage. PII, generic high-entropy detection, worker stdout, and writeToolCalls JSONL are explicitly out of scope here; I am not asking this PR to solve those separate issues. There are no UI changes requiring Storybook or UI tests.

Thanks again for the useful defense-in-depth work.

{ id: "azure-storage-key", pattern: /(?<prefix>\bAccountKey=)[A-Za-z0-9+/]{40,}={0,2}/g },
// SAS signature only — the rest of the blob URL stays readable for debugging.
{ id: "azure-sas-signature", pattern: /(?<prefix>[?&]sig=)[A-Za-z0-9%+/=]{16,}/g },
{ id: "authorization-bearer", pattern: /(?<prefix>\bBearer\s+)[A-Za-z0-9._~+/-]{16,}=*/g },

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2 / medium] Match HTTP bearer schemes case-insensitively in both the rule and prefilter. HTTP authentication schemes are case-insensitive, but this regex and the substring fast path only recognize Bearer with that exact capitalization.

For example, redactSecrets("curl -H 'Authorization: bearer syntheticOpaqueCredential1234567890' https://example.invalid") leaves the entire synthetic credential visible; changing only bearer to Bearer redacts it. The production name-based argument filter also leaves this intact because the enclosing argument is command, not a sensitive-looking field name.

Please make both scheme detection and the fast-path condition case-insensitive while preserving the original prefix, and add lowercase/uppercase/mixed-case publisher regressions. Otherwise valid credentials with unrecognized value shapes still reach Redis/SSE and run.jsonl despite the new bearer protection.

const SECRET_RULES: readonly SecretRule[] = [
{
id: "private-key",
pattern: /-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----[\s\S]*?-----END [A-Z0-9 ]*PRIVATE KEY-----/g,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2 / medium] Cover PEM blocks split across the real log stream. This rule requires BEGIN, key material, and END in one string, but the ACP subprocess stderr callback forwards each pipe chunk separately (apps/workers/coder-acp-copilot/src/acp-client.ts:573-575). Independent writes of the three PEM lines therefore produce events with no redaction marker, and log readers can reconstruct the complete key.

This was reproduced using a real subprocess stderr pipe and an ephemeral in-memory Ed25519 key: three chunks, zero redaction markers, and the reconstructed key still parsed. Passing the same key as one string was redacted. This is the forwarded stderr -> publisher path, not the explicitly excluded worker stdout or tool-call JSONL sink.

Please add stream-aware PEM suppression at an appropriate pre-publication boundary, including markers split between chunks and incomplete blocks, and test that path with separately emitted lines. If that requires a separately scoped change, the coverage/guarantee must at least be made explicit rather than claiming private-key protection for this delivery path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: shared Shared package: types, DB models, and queue/blob/redis clients used by all apps. community-contribution Pull requests submitted by external community contributors. language: javascript Work involving JavaScript code, tooling, or dependencies. topic: observability Logging, metrics, tracing, and operational visibility. topic: security Security protections, vulnerabilities, and risk reduction. topic: testing Test coverage, test infrastructure, and validation quality. type: documentation Documentation additions, corrections, and improvements. type: enhancement New functionality or improvements to existing behavior.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants