Skip to content

Harden API + fix bugs (v3.1) - #6

Closed
willscott-v2 wants to merge 3 commits into
feature/si-design-systemfrom
fix/hardening-and-bugs
Closed

willscott-v2 wants to merge 3 commits into
feature/si-design-systemfrom
fix/hardening-and-bugs

Conversation

@willscott-v2

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #5 (SI design system). Fixes the security issues and bugs from the full code review. No visual changes — review this one for behavior.

Base is feature/si-design-system, not main so the diff stays focused on logic. Merge #5 first, then this retargets to main automatically.

Security

  • SSRF guards (api/chunk.js): http/https only; DNS-resolve the host and reject private/reserved ranges (127/10/172.16/192.168, 169.254 cloud metadata, 100.64 CGNAT, IPv6 loopback/ULA/link-local); every redirect hop re-validated (manual redirect, max 5); 5MB response cap; content-type allowlist. The endpoint was an open fetch proxy — anyone could scrape through our Vercel IPs or probe internal addresses.
  • XSS: escapeHtml now escapes " and '. It's used inside title="${fragment}" where fragment comes from the scraped page's heading id — a page with <h2 id='x" onmouseover=...'> could inject a live handler. Verified the payload now stays trapped in the attribute value.
  • Slack injection (api/feedback.js): escape &<> in message/email/pageUrl/user-agent — kills <!channel> pings and <url|label> link spoofing. pageUrl must be http(s), length-capped.
  • Input validation: allowlist chunkSize/strategy/extract/format/tokenizer → 400 instead of a confusing 500; overlap bounded 0–200.
  • Error leak: stop returning raw error.message to clients.

Bugs

  • Overlap slider was permanently stuck on "Auto" — the input handler's condition made the manual branch unreachable, so users could never set overlap from the UI. Verified fixed: dragging to 25 now sends overlap: 25.
  • Dead fetch timeout — timeout: 30000 is ignored by node-fetch v3. Replaced with native fetch + AbortSignal.timeout(25s), and dropped node-fetch entirely (engines already >=18).
  • extract=defuddle was a lie — it silently fell back to cheerio. Now fails 422.

Housekeeping

  • Removed scripts/build.js + vercel buildCommand. The build step mutated public/index.html in place (how the live GTM ID got committed). GTM-4G43 confirmed identical to the grader/ontologizer shared container; hardcoded.
  • Deleted ~1,900 lines of unused root design-system files (superseded by the canonical theme; recoverable from git).
  • README: MIT license (matched to package.json), searchinfluence/getchunks URLs, removed stale v1.2 section, honest "is my data secure" FAQ.

Testing

Local Vercel-style harness + Playwright:

  • SSRF: localhost / 127.0.0.1 / 169.254 metadata / 10.x / ftp:// / file:// all rejected 400
  • Live: example.com (cheerio fallback), wikipedia http→https redirect followed → 12 big / 29 small chunks, markdown format
  • Validation: bad chunkSize/overlap/type → 400; defuddle-only on thin page → 422
  • Slider fix and XSS containment verified in-browser

Deferred (not this PR)

  • Vercel WAF rate-limit rule (dashboard config — needs to be applied there)
  • Fixture/snapshot tests for split/merge/overlap
  • Nested-content extraction fix for the cheerio fallback path

🤖 Generated with Claude Code

willscott-v2 and others added 3 commits July 21, 2026 13:12
Security:
- SSRF guards in api/chunk.js: http/https only, DNS-resolve + reject
  private/reserved IP ranges (incl. 169.254 metadata, CGNAT, IPv6 ULA),
  per-hop redirect re-validation, 5MB body cap, content-type allowlist
- escapeHtml now escapes quotes — closes title="${fragment}" attribute
  XSS from scraped heading ids
- Slack mrkdwn escaping in feedback.js kills <!channel> pings + link
  spoofing; pageUrl scheme-checked and length-capped
- Input validation with allowlists (400 not 500); error details no longer
  leak to clients

Bugs:
- Overlap slider was permanently stuck on Auto (unreachable else branch)
- Dead node-fetch timeout replaced with native fetch + AbortSignal.timeout;
  node-fetch dependency removed
- extract=defuddle now fails 422 instead of silently falling back to cheerio

Housekeeping:
- Removed scripts/build.js + vercel buildCommand; GTM-4G43 (shared SI
  container, matches grader/ontologizer) hardcoded; GTM_SETUP.md rewritten
- Deleted ~1,900 lines of unused root design-system files
- README: MIT license, searchinfluence/getchunks URLs, v3/v3.1 history,
  honest data-security FAQ

Verified locally: SSRF/scheme/validation rejections, live extraction,
http->https redirect following, slider fix, and XSS containment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@willscott-v2
willscott-v2 deleted the branch feature/si-design-system July 21, 2026 23:55
@willscott-v2
willscott-v2 deleted the fix/hardening-and-bugs branch July 21, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant