diff --git a/README.md b/README.md index 247327a..3cee4a4 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,8 @@ A comprehensive TypeScript library for capturing, storing, and appending UTM tra - **Store** in sessionStorage or localStorage (with optional TTL) - **Attribution** — first-touch, last-touch, or both - **Form population** — inject UTM data into HTML forms (vanilla JS + React) +- **Rejection reporting** — tell "no campaign" apart from "campaign rejected" +- **Case folding** — normalize `LinkedIn` and `linkedin` to one campaign ### Outbound (Creating UTM-tagged links) - **Append** UTM parameters to share URLs @@ -28,6 +30,7 @@ A comprehensive TypeScript library for capturing, storing, and appending UTM tra - **React hook** and context provider - **Debug utilities** for troubleshooting - **SSR-safe** with graceful fallbacks +- **Server entry** (`/server`) — DOM-free normalization for public ingest endpoints - **Zero dependencies** (peer dependency on React is optional) ## Installation @@ -274,6 +277,49 @@ const params = captureUtmParameters(url, { }); ``` +#### Dropping instead of truncating + +By default an over-length value is truncated to `maxLength`. Truncation invents a value nobody sent, and two campaigns sharing a long prefix collapse into one. Set `onMaxLength: 'drop'` when values key a datastore — an absent parameter is honest, a fabricated one is not. + +```typescript +captureUtmParameters(url, { + sanitize: { enabled: true, maxLength: 64, onMaxLength: 'drop' }, +}); +// An over-length utm_source becomes '' rather than its first 64 characters +``` + +#### Gating values with `valuePattern` + +`valuePattern` is a positive allowlist: a value that does not match becomes `''`. Note how it differs from the two adjacent regex options: + +| Option | Effect | +|--------|--------| +| `sanitize.customPattern` | **Subtractive** — strips every match from the value | +| `sanitize.valuePattern` | **A gate** — keeps the value intact, or drops it whole | +| `piiFiltering.allowlistPattern` | The same gate, scoped to PII decisions, and able to produce `'[REDACTED]'` in redact mode | + +```typescript +captureUtmParameters(url, { + sanitize: { enabled: true, valuePattern: /^[a-z0-9_-]+$/ }, +}); +// 'spring-2025' survives; 'has spaces!' becomes '' +``` + +The gate is tested against the *trimmed* value, so surrounding whitespace never causes a rejection on its own. + +#### Folding case with `lowercaseValues` + +`LinkedIn` and `linkedin` are the same campaign. Anyone keying a store on captured values gets two rows unless the values are folded. `lowercaseValues` is the inbound counterpart of `buildUtmUrl`'s option of the same name: + +```typescript +captureUtmParameters('https://example.com?utm_source=LinkedIn', { + lowercaseValues: true, +}); +// { utm_source: 'linkedin' } +``` + +Folding runs **before** sanitization and PII filtering, so `customPattern`, `valuePattern` and `piiFiltering.allowlistPattern` all see the folded value and can be written without allowing uppercase. Keys are never folded, only values. It uses `toLowerCase()` rather than `toLocaleLowerCase()`, so the result does not depend on the host locale. + ### PII Filtering Detect and filter personally identifiable information (email addresses, phone numbers) from UTM parameter values. Prevents PII from leaking into analytics via misconfigured tracking links. Disabled by default. @@ -314,6 +360,79 @@ const params = captureUtmParameters(url, { Built-in PII patterns detect: email addresses, international phone numbers, UK phone numbers, and US phone numbers. +### Telling "No Campaign" Apart From "Campaign Rejected" + +`captureUtmParameters` returns `{}` for two completely different situations: genuine direct traffic, and a campaign link whose every parameter was filtered. Collapsing them inflates the direct-traffic denominator that every campaign share is measured against. + +`captureUtmParametersWithReport` separates them: + +```typescript +import { captureUtmParametersWithReport } from '@jackmisner/utm-toolkit'; + +const { params, rejected, invalidUrl } = captureUtmParametersWithReport(url, { + piiFiltering: { enabled: true }, +}); + +if (invalidUrl) { + // The URL could not be parsed at all — neither direct traffic nor a campaign +} else if (Object.keys(params).length === 0 && rejected.length > 0) { + // A campaign link arrived and every parameter was filtered. + // This is NOT direct traffic. + console.warn('rejected:', rejected); + // [{ key: 'utm_source', reason: 'pii', patternName: 'email' }] +} +``` + +Rejection reasons are `'allowedParameters'`, `'valuePattern'`, `'maxLength'`, `'allowlist'`, `'pii'` and `'notAString'` (server-side only). Rejections are recorded per parameter, so one bad parameter never costs you the whole campaign. + +> **The report deliberately omits the rejected value.** It carries the key, the reason, and for PII the matching pattern name — nothing else. A report struct containing the raw value would be handed straight to a logger by most consumers, which is exactly what PII filtering exists to prevent. + +> **But the `key` is not sanitized.** Any `utm_`-prefixed query parameter is captured, so the key comes straight from the URL and an attacker controls it — `?utm_someone@example.com=1` produces a rejection whose `key` contains an email address. `rejected` is also unbounded, one entry per offending parameter. Treat the report as untrusted input before logging it: filter to the keys you expect, and cap the length. + +`captureUtmParameters` delegates to this function and returns `.params`, so the two can never drift apart. + +### Sending Captured Params to a Server + +If you POST captured parameters to your own endpoint, there is a trap in `navigator.sendBeacon` worth knowing about before you hit it. + +**`sendBeacon` does not send JSON.** `navigator.sendBeacon(url, JSON.stringify(params))` sends `Content-Type: text/plain;charset=UTF-8`. A server that only parses `application/json` receives the body as a raw string, JSON parsing fails, and — if the server is written defensively — the campaign silently becomes empty while the endpoint still answers `204`. There is no error anywhere, and every campaign is attributed to direct traffic. + +Wrapping the body in a `Blob` typed `application/json` does not fix it. That makes the request non-simple, `sendBeacon` cannot perform the CORS preflight, and the browser drops the request entirely — also silently. + +There are two workable options, and the tradeoff is real: + +```typescript +// Option 1 — fetch with keepalive. Sends real JSON, survives page unload, +// but is subject to CORS preflight and a ~64KB keepalive body limit. +fetch('/api/utm', { + method: 'POST', + keepalive: true, + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(params), +}); + +// Option 2 — sendBeacon, and a server that accepts text/plain. +navigator.sendBeacon('/api/utm', JSON.stringify(params)); +``` + +If you choose option 2, the server has to opt into the content type it will actually receive: + +```typescript +// Fastify — register text/plain and parse it as JSON +fastify.addContentTypeParser('text/plain', { parseAs: 'string' }, (_req, body, done) => { + try { + done(null, JSON.parse(body as string)); + } catch { + done(null, {}); // Never throw on an untrusted body + } +}); + +// Express +app.use(express.text({ type: 'text/plain' })); +``` + +Either way, normalize what arrives — see [Server-Side Usage](#server-side-usage) below. The client-side pass cannot be trusted for a public endpoint, because anyone can POST to it directly. + ### Event Callbacks Hook into UTM lifecycle events for logging, analytics, or custom behavior. @@ -652,6 +771,56 @@ installDebugHelpers(); // Then use: window.utmDebug.state(), window.utmDebug.check() ``` +## Server-Side Usage + +`@jackmisner/utm-toolkit/server` is a DOM-free entry point for applying the same folding rules server-side. It exists because **the client-side pass cannot be trusted for a public endpoint** — anyone can POST to it directly. + +```typescript +import { normalizeUtmParams } from '@jackmisner/utm-toolkit/server'; + +app.post('/api/utm', async (req, res) => { + const { params, rejected } = normalizeUtmParams(req.body); + + // params is TOTAL: every allowed key is present, absent ones are '' + // { utm_source: 'linkedin', utm_medium: '', utm_campaign: '', ... } + await db.insertCampaignHit(params); + + if (rejected.length > 0) metrics.increment('utm.rejected', rejected.length); + res.status(204).end(); +}); +``` + +For a server that has a URL rather than a parsed body — a `Referer` header, a redirect target — use `normalizeUtmUrl(url, options)`. It has the same contract. + +### Why the output is total + +Every key in `allowedParameters` is always present, with absent parameters set to `absentValue` (default `''`). If you write these into a composite primary key, absence has to be a value that **groups**: `NULL` does not deduplicate in most stores, so a nullable column fragments one campaign into as many rows as it has absent parameters. Set `absentValue` to something unforgeable if `''` could collide with a real campaign value. + +### It never throws + +The argument is an untrusted HTTP body, so a throw is a 500 on somebody's first page load. `null`, `42`, `'a string'`, `[]`, `{ utm_source: ['a','b'] }` and a body with a `__proto__` key all produce a usable total result. Non-string values are **rejected rather than coerced** — `String(['a','b'])` is `'a,b'`, a value nobody sent. + +### Server defaults differ from browser defaults, deliberately + +| Option | Server default | Browser default | Why | +|--------|----------------|-----------------|-----| +| `lowercase` | `true` | `false` | `LinkedIn` and `linkedin` are one campaign | +| `onMaxLength` | `'drop'` | `'truncate'` | A truncated value is one nobody sent | +| `piiFiltering` | enabled | disabled | The endpoint is public | +| `allowedParameters` | all six standard params, **including `utm_id`** | same | Narrow it if you key fewer columns | + +The browser defaults are lenient because losing a campaign label client-side is cheap. A server keying a datastore needs determinism. + +`piiFiltering.mode` is deliberately **not** configurable here. `'[REDACTED]'` persisted as a campaign value is a campaign nobody ran, which is worse than dropping it; server-side filtering always rejects. + +> **On `utm_id`:** the server defaults to all six standard parameters, matching the browser. If your table keys five columns, pass `allowedParameters` explicitly — a consumer keying five against a library producing six gets a mystery extra row. + +### What this entry point cannot reach + +`/server` does not import storage, form population, link decoration, debug helpers or React. That restriction is enforced by a test that walks the module's runtime import graph, not just documented — so the guarantee cannot be quietly removed by a convenient re-export. + +Note the honest framing: the root entry does **not** crash in Node. The reasons to use `/server` are the documented DOM-free surface, the server-appropriate defaults, the totality contract, and a smaller install/parse surface — the server bundle is roughly 5KB against the root entry's 52KB. + ## Configuration Options | Option | Type | Default | Description | @@ -664,6 +833,7 @@ installDebugHelpers(); | `captureOnMount` | `boolean` | `true` | Auto-capture on React hook mount | | `appendToShares` | `boolean` | `true` | Append UTM params to share URLs | | `allowedParameters` | `string[]` | Standard UTM params | Params to capture | +| `lowercaseValues` | `boolean` | `false` | Fold captured values to lowercase (runs before sanitize and PII gates) | | `defaultParams` | `object` | `{}` | Fallback params when none captured | | `shareContextParams` | `object` | `{}` | Platform-specific params | | `excludeFromShares` | `string[]` | `[]` | Params to exclude from shares | @@ -676,6 +846,20 @@ installDebugHelpers(); | `onAppend` | `function` | `undefined` | Callback after UTM params are appended to a URL | | `onExpire` | `function` | `undefined` | Callback when stored params expire (TTL) | +### `SanitizeConfig` + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `enabled` | `boolean` | `false` | Enable sanitization | +| `stripHtml` | `boolean` | `true` | Strip HTML-significant characters: `<` `>` `"` `'` and backtick | +| `stripControlChars` | `boolean` | `true` | Strip control characters except tab/newline/CR | +| `maxLength` | `number` | `200` | Maximum value length | +| `onMaxLength` | `'truncate' \| 'drop'` | `'truncate'` | Truncate an over-length value, or drop it to `''` | +| `customPattern` | `RegExp` | `undefined` | **Subtractive** — strips every match from the value | +| `valuePattern` | `RegExp` | `undefined` | **A gate** — keeps the value whole, or drops it to `''` | + +Rules apply in order: `stripHtml` → `stripControlChars` → `customPattern` → trim → `valuePattern` → `maxLength`. + ## TypeScript Types ```typescript diff --git a/__tests__/config/loader.test.ts b/__tests__/config/loader.test.ts index bfa8f46..b8aa2b7 100644 --- a/__tests__/config/loader.test.ts +++ b/__tests__/config/loader.test.ts @@ -147,6 +147,38 @@ describe('loadConfigFromJson', () => { }) describe('validateConfig', () => { + it('rejects a non-boolean lowercaseValues', () => { + const errors = validateConfig({ lowercaseValues: 'yes' as unknown as boolean }) + expect(errors.some((e) => e.includes('lowercaseValues'))).toBe(true) + }) + + it('accepts a boolean lowercaseValues', () => { + expect(validateConfig({ lowercaseValues: true })).toEqual([]) + }) + + it('rejects an unknown sanitize.onMaxLength', () => { + const errors = validateConfig({ + sanitize: { onMaxLength: 'trunkate' as unknown as 'truncate' }, + }) + expect(errors.some((e) => e.includes('onMaxLength'))).toBe(true) + }) + + it('accepts valid sanitize.onMaxLength values', () => { + expect(validateConfig({ sanitize: { onMaxLength: 'drop' } })).toEqual([]) + expect(validateConfig({ sanitize: { onMaxLength: 'truncate' } })).toEqual([]) + }) + + it('rejects a non-RegExp sanitize.valuePattern', () => { + const errors = validateConfig({ + sanitize: { valuePattern: '^[a-z]+$' as unknown as RegExp }, + }) + expect(errors.some((e) => e.includes('valuePattern'))).toBe(true) + }) + + it('accepts a RegExp sanitize.valuePattern', () => { + expect(validateConfig({ sanitize: { valuePattern: /^[a-z]+$/ } })).toEqual([]) + }) + it('returns empty array for valid config', () => { const errors = validateConfig({ enabled: true, @@ -246,9 +278,45 @@ describe('sanitize config', () => { stripHtml: true, stripControlChars: true, maxLength: 200, + onMaxLength: 'truncate', }) }) + it('createConfig carries onMaxLength through', () => { + const config = createConfig({ sanitize: { enabled: true, onMaxLength: 'drop' } }) + expect(config.sanitize.onMaxLength).toBe('drop') + }) + + it('createConfig carries valuePattern through', () => { + const pattern = /^[a-z]+$/ + const config = createConfig({ sanitize: { enabled: true, valuePattern: pattern } }) + expect(config.sanitize.valuePattern).toBe(pattern) + }) + + it('createConfig({}) still resolves the onMaxLength default', () => { + // createConfig() with no argument early-returns the defaults without going + // through the merge, so it cannot catch a field the merge forgets to copy. + expect(createConfig({}).sanitize.onMaxLength).toBe('truncate') + }) + + it('createConfig keeps the onMaxLength default when other sanitize fields are set', () => { + const config = createConfig({ sanitize: { enabled: true } }) + expect(config.sanitize.onMaxLength).toBe('truncate') + }) + + it('mergeConfig carries onMaxLength and valuePattern through', () => { + const pattern = /^[a-z]+$/ + const base = createConfig({ sanitize: { enabled: true } }) + const merged = mergeConfig(base, { sanitize: { onMaxLength: 'drop', valuePattern: pattern } }) + expect(merged.sanitize.onMaxLength).toBe('drop') + expect(merged.sanitize.valuePattern).toBe(pattern) + }) + + it('createConfig carries lowercaseValues through', () => { + expect(createConfig({ lowercaseValues: true }).lowercaseValues).toBe(true) + expect(createConfig({}).lowercaseValues).toBe(false) + }) + it('createConfig merges partial sanitize config with defaults', () => { const config = createConfig({ sanitize: { enabled: true, maxLength: 100 }, diff --git a/__tests__/debug/diagnostics.test.ts b/__tests__/debug/diagnostics.test.ts new file mode 100644 index 0000000..bacf116 --- /dev/null +++ b/__tests__/debug/diagnostics.test.ts @@ -0,0 +1,47 @@ +import { describe, it, expect, vi } from 'vitest' +import { getDiagnostics } from '../../src/debug' +import { createConfig } from '../../src/config/loader' + +/** + * getDiagnostics exists to answer "why is the stored value not what I expect". + * It can only do that if it captures through the same options the real pipeline + * uses — otherwise it reports a value nobody's pipeline ever produced, and the + * tool that is supposed to resolve the confusion becomes the cause of it. + */ +describe('getDiagnostics urlParams reflects the configured pipeline', () => { + const stubUrl = (href: string): void => { + vi.stubGlobal('location', { href, search: href.slice(href.indexOf('?')) }) + } + + it('applies lowercaseValues from config', () => { + stubUrl('https://example.com?utm_source=LinkedIn') + const diagnostics = getDiagnostics(createConfig({ lowercaseValues: true })) + expect(diagnostics.urlParams.utm_source).toBe('linkedin') + }) + + it('leaves case alone when lowercaseValues is off', () => { + stubUrl('https://example.com?utm_source=LinkedIn') + const diagnostics = getDiagnostics(createConfig({})) + expect(diagnostics.urlParams.utm_source).toBe('LinkedIn') + }) + + it('applies sanitize from config', () => { + stubUrl('https://example.com?utm_source=%3Cb%3Ebold%3C%2Fb%3E') + const diagnostics = getDiagnostics(createConfig({ sanitize: { enabled: true } })) + expect(diagnostics.urlParams.utm_source).toBe('bbold/b') + }) + + it('applies PII filtering from config', () => { + stubUrl('https://example.com?utm_source=someone%40example.com') + const diagnostics = getDiagnostics(createConfig({ piiFiltering: { enabled: true } })) + expect(diagnostics.urlParams).toEqual({}) + }) + + it('still honours keyFormat and allowedParameters', () => { + stubUrl('https://example.com?utm_source=x&utm_term=y') + const diagnostics = getDiagnostics( + createConfig({ keyFormat: 'camelCase', allowedParameters: ['utm_source'] }), + ) + expect(diagnostics.urlParams).toEqual({ utmSource: 'x' }) + }) +}) diff --git a/__tests__/docs.md b/__tests__/docs.md index e45df2b..c6be775 100644 --- a/__tests__/docs.md +++ b/__tests__/docs.md @@ -4,13 +4,13 @@ Path: @/__tests__ ### Overview -- Test suite for the library, mirroring the `@/src` directory structure with subdirectories for `common/`, `inbound/`, `outbound/`, `config/`, and `react/`. +- Test suite for the library, mirroring the `@/src` directory structure one subdirectory per source folder. - Uses vitest with jsdom environment, `@testing-library/react` for React component/hook tests, and a global setup file for browser API mocks. - Coverage thresholds are enforced at 80% for statements, branches, functions, and lines (configured in `@/vitest.config.ts`). ### How it fits into the larger codebase -- Tests exercise all public API surfaces from `@/src/common`, `@/src/inbound`, `@/src/outbound`, `@/src/config`, and `@/src/react`. +- Tests exercise every public API surface in `@/src`, across all three package entry points (root, `/react`, `/server`). - Coverage excludes barrel `index.ts` files and the `@/src/types` directory, since these contain only re-exports and type definitions. - The test setup (`setup.ts`) provides the mock environment that all tests rely on: sessionStorage and localStorage mocks backed by plain objects with `vi.fn()` wrappers, and a `window.location` mock defaulting to `https://example.com`. - CI runs tests across Node 18, 20, and 22. @@ -19,15 +19,22 @@ Path: @/__tests__ - **`setup.ts`**: Creates fresh sessionStorage and localStorage mocks and a location mock in `beforeEach`, ensuring tests are isolated. Both storage mocks implement `getItem`, `setItem`, `removeItem`, `clear`, `length`, and `key`. Location is stubbed with `href`, `search`, `hash`, `pathname`, `protocol`, `host`, and `hostname`. - **`common/` tests**: Cover storage (write/read/clear, format conversion, validation of stored data, silent failure, localStorage backend, envelope format, TTL expiration with fake timers, backward compatibility with flat format data, availability checks, event callbacks), keys (bidirectional conversion, standard and custom keys, detection, validation), validator (protocol, domain, normalization, mutable default protocol), and event callback integration. -- **`inbound/` tests**: Cover capture (URL parsing, allowed parameters, key format conversion, SSR fallback, sanitization integration, PII filtering integration), sanitizer (HTML stripping, control character removal, custom patterns, truncation), pii-filter (pattern detection, reject/redact modes, allowlist, callback), attribution (first-touch/last-touch/both modes, write-once semantics), and form field population (name/data-attribute/auto-create strategies). +- **`inbound/` tests**: Cover capture (URL parsing, allowed parameters, key format conversion, SSR fallback, sanitization integration, PII filtering integration), sanitizer (HTML stripping, control character removal, custom patterns, truncation), pii-filter (pattern detection, reject/redact modes, allowlist, callback), attribution (first-touch/last-touch/both modes, write-once semantics), and form field population (name/data-attribute/auto-create strategies). Capture coverage includes the ordering-sensitive cases the two-phase pipeline exists for — duplicate parameters whose surviving occurrence is or is not PII, and malformed config degrading to empty rather than throwing. - **`outbound/` tests**: Cover appender (query/fragment placement, preserveExisting, remove, extract), builder (structured URL construction, validation, warnings, lowercase option), and decorator (link decoration, host filtering, skip-existing, MutationObserver). -- **`config/` tests**: Cover `createConfig` merging semantics (including `storageType`, `ttl`, attribution, and event callbacks), `validateConfig` error messages, `loadConfigFromJson` fallback behavior, and nested config merging for sanitize and piiFiltering. -- **`react/` tests**: Use `@testing-library/react` `renderHook` and `render` to test `useUtmTracking` (auto-capture, manual capture, clear, appendToUrl, `storageType` forwarding, attribution params), `UtmProvider`/`useUtmContext`, `UtmHiddenFields`, and `UtmLinkDecorator`. +- **`config/` tests**: Cover `createConfig` merging semantics (including `storageType`, `ttl`, attribution, and event callbacks), `validateConfig` error messages, `loadConfigFromJson` fallback behavior, and nested config merging for sanitize and piiFiltering. Merge coverage always passes an **explicit override object** — `createConfig()` with no argument early-returns the defaults without merging, so it cannot detect a field the merge functions drop. +- **`debug/` tests**: Assert that `getDiagnostics` reports what the real pipeline would produce (sanitisation, PII filtering and folding applied) and that it does **not** fire `onCapture`. +- **`react/` tests**: Use `@testing-library/react` `renderHook` and `render` to test `useUtmTracking` (auto-capture, manual capture, clear, appendToUrl, `storageType` forwarding, attribution params, capture-option forwarding such as `lowercaseValues`), `UtmProvider`/`useUtmContext`, `UtmHiddenFields`, and `UtmLinkDecorator`. +- **`server/` tests**: Cover `normalizeUtmParams`/`normalizeUtmUrl` behaviour (totality of the output record, non-string rejection rather than coercion, `__proto__` keys, server default divergence) plus an **architectural test** described below. The never-throws contract is exercised against the objects that can actually break it — throwing getters, hostile and revoked Proxies, malformed caller-supplied regexes, and a non-array `allowedParameters`. ### Things to Know - Both the sessionStorage and localStorage mocks use `vi.fn()` wrappers, which means tests can assert on call counts and arguments (e.g., `sessionStorage.setItem` or `localStorage.setItem` calls). - `window.location` is stubbed globally rather than using JSDOM's location, so tests that need specific URLs must override `location.href` and `location.search` in their setup. - The `beforeEach` in `setup.ts` resets both storage mocks and the location mock, so each test starts with empty storage and a clean `https://example.com` location. +- **`server/isolation.test.ts` tests architecture, not behaviour.** It reads source files off disk and walks the transitive **runtime** import graph out of `@/src/server/index.ts`, failing if it reaches a forbidden module (storage, form, attribution, appender, decorator, debug, react) or if any reachable file mentions `window`/`document`/`sessionStorage`/`localStorage`. The `/server` entry's whole value rests on an import restriction, which is only real if something checks it — otherwise the next convenient re-export silently removes the guarantee. +- The walk deliberately **does not follow `import type` / `export type`**, because those are erased at build and cannot pull browser code into the bundle. Following them would flag the legitimate type-only import of `UtmRejection` from `@/src/inbound/capture-report.ts`. Comments are stripped before scanning, so a docblock mentioning `sessionStorage` is not read as a use of it — with a guard so `https://` inside a string literal is not mistaken for a line comment, which would swallow the rest of that line and any `window` reference after it. +- **Everything the walk cannot see is a hole in the guarantee, so the walk is built to fail loudly rather than under-report.** It follows side-effect imports (`import './x'`) and dynamic `import('./x')` as well as ordinary statements — neither form exists in `@/src` today, but both are how someone reintroduces storage later. Bare (non-relative) specifiers are collected and asserted **empty**, since the package ships no runtime dependencies and a relative-path check cannot see `import 'react'`. An unresolvable relative specifier **throws**: a specifier the walk skipped is a blind spot, not a pass. +- **The isolation test needs its own positive control.** If the regexes parsed nothing, every "does not reach X" assertion would pass vacuously, so a separate case pins modules reached two hops out through real runtime imports (the sanitizer, PII filter and config defaults), not just the entry and its immediate neighbour. +- **An equivalence test between a wrapper and its delegate proves nothing.** `captureUtmParameters` *is* `captureUtmParametersWithReport(...).params`, so `inbound/capture-report.test.ts` comparing the two asserted `x === x` and passed against any pipeline however wrong — which is how a duplicate-parameter regression got through. It now checks **golden values** captured from the pre-refactor behaviour, including the duplicate cases, so it pins behaviour rather than self-consistency. Created and maintained by Nori. diff --git a/__tests__/inbound/capture-report.test.ts b/__tests__/inbound/capture-report.test.ts new file mode 100644 index 0000000..6761123 --- /dev/null +++ b/__tests__/inbound/capture-report.test.ts @@ -0,0 +1,258 @@ +import { describe, it, expect } from 'vitest' +import { captureUtmParameters } from '../../src/inbound/capture' +import { captureUtmParametersWithReport } from '../../src/inbound/capture-report' + +const EMAIL = 'someone@example.com' + +describe('captureUtmParametersWithReport', () => { + describe('absence versus rejection', () => { + it('reports zero rejections for a URL with no UTM parameters', () => { + const report = captureUtmParametersWithReport('https://example.com/page') + expect(report.params).toEqual({}) + expect(report.rejected).toEqual([]) + expect(report.invalidUrl).toBe(false) + }) + + it('reports a rejection when the only parameter is filtered as PII', () => { + const report = captureUtmParametersWithReport( + `https://example.com?utm_source=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true } }, + ) + expect(report.params).toEqual({}) + expect(report.rejected).toHaveLength(1) + expect(report.rejected[0].key).toBe('utm_source') + expect(report.rejected[0].reason).toBe('pii') + }) + + it('distinguishes the two: both produce empty params but different reports', () => { + const absent = captureUtmParametersWithReport('https://example.com/page') + const rejected = captureUtmParametersWithReport( + `https://example.com?utm_source=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true } }, + ) + expect(absent.params).toEqual(rejected.params) + expect(absent.rejected).toHaveLength(0) + expect(rejected.rejected).toHaveLength(1) + }) + }) + + describe('rejection reasons', () => { + it('names the matching PII pattern', () => { + const report = captureUtmParametersWithReport( + `https://example.com?utm_content=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true } }, + ) + expect(report.rejected[0].patternName).toBe('email') + }) + + it('reports maxLength when an over-length value is dropped', () => { + const report = captureUtmParametersWithReport( + `https://example.com?utm_source=${'a'.repeat(50)}`, + { sanitize: { enabled: true, maxLength: 10, onMaxLength: 'drop' } }, + ) + expect(report.rejected).toEqual([{ key: 'utm_source', reason: 'maxLength' }]) + }) + + it('does not report a rejection when an over-length value is merely truncated', () => { + const report = captureUtmParametersWithReport( + `https://example.com?utm_source=${'a'.repeat(50)}`, + { sanitize: { enabled: true, maxLength: 10, onMaxLength: 'truncate' } }, + ) + expect(report.rejected).toEqual([]) + expect(report.params.utm_source).toBe('a'.repeat(10)) + }) + + it('reports valuePattern when a value fails the gate', () => { + const report = captureUtmParametersWithReport('https://example.com?utm_source=has%20spaces', { + sanitize: { enabled: true, valuePattern: /^[a-z]+$/ }, + }) + expect(report.rejected).toEqual([{ key: 'utm_source', reason: 'valuePattern' }]) + }) + + it('reports allowlist when the PII allowlist pattern rejects a value', () => { + const report = captureUtmParametersWithReport('https://example.com?utm_source=UPPER', { + piiFiltering: { enabled: true, allowlistPattern: /^[a-z]+$/ }, + }) + expect(report.rejected).toEqual([{ key: 'utm_source', reason: 'allowlist' }]) + }) + + it('reports allowedParameters when a utm_ key is not in the allowlist', () => { + const report = captureUtmParametersWithReport( + 'https://example.com?utm_source=linkedin&utm_term=ignored', + { allowedParameters: ['utm_source'] }, + ) + expect(report.params).toEqual({ utm_source: 'linkedin' }) + expect(report.rejected).toEqual([{ key: 'utm_term', reason: 'allowedParameters' }]) + }) + + it('does not report a value that becomes empty through ordinary stripping', () => { + const report = captureUtmParametersWithReport( + 'https://example.com?utm_source=%3C%3E%22%27%60', + { sanitize: { enabled: true } }, + ) + expect(report.rejected).toEqual([]) + }) + }) + + describe('malformed URLs', () => { + it('flags an unparseable URL rather than reporting it as no campaign', () => { + const report = captureUtmParametersWithReport('not a url at all') + expect(report.invalidUrl).toBe(true) + expect(report.params).toEqual({}) + }) + + it('does not flag a well-formed URL', () => { + const report = captureUtmParametersWithReport('https://example.com?utm_source=linkedin') + expect(report.invalidUrl).toBe(false) + }) + }) + + describe('rejection is per-parameter, not per-request', () => { + it('keeps a good parameter while reporting a bad sibling', () => { + const report = captureUtmParametersWithReport( + `https://example.com?utm_source=linkedin&utm_content=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true } }, + ) + expect(report.params).toEqual({ utm_source: 'linkedin' }) + expect(report.rejected).toHaveLength(1) + expect(report.rejected[0].key).toBe('utm_content') + }) + }) + + describe('the report never carries the rejected value', () => { + it('does not contain the email anywhere in the serialised report', () => { + const report = captureUtmParametersWithReport( + `https://example.com?utm_content=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true } }, + ) + const serialised = JSON.stringify(report) + expect(serialised).not.toContain(EMAIL) + expect(serialised).not.toContain('someone') + expect(serialised).not.toContain('example.com') + }) + + it('does not contain an over-length value that was dropped', () => { + const secret = 'z'.repeat(50) + const report = captureUtmParametersWithReport(`https://example.com?utm_source=${secret}`, { + sanitize: { enabled: true, maxLength: 10, onMaxLength: 'drop' }, + }) + expect(JSON.stringify(report)).not.toContain(secret) + }) + }) + + describe('captureUtmParameters produces the documented results', () => { + // Golden values, NOT a comparison against captureUtmParametersWithReport. + // captureUtmParameters IS `captureUtmParametersWithReport(...).params`, so + // comparing the two asserts x === x and passes against any pipeline, however + // wrong. Every expectation below was taken from the pre-refactor + // implementation on main, so it pins behaviour rather than self-consistency. + const cases: Array< + [string, string, Parameters[1], Record] + > = [ + [ + 'plain capture', + 'https://example.com?utm_source=linkedin', + undefined, + { utm_source: 'linkedin' }, + ], + [ + 'camelCase keys', + 'https://example.com?utm_source=linkedin', + { keyFormat: 'camelCase' }, + { utmSource: 'linkedin' }, + ], + [ + 'allowlist filtering', + 'https://example.com?utm_source=a&utm_term=b', + { allowedParameters: ['utm_source'] }, + { utm_source: 'a' }, + ], + [ + 'sanitisation', + 'https://example.com?utm_source=%3Cb%3Ebold%3C%2Fb%3E', + { sanitize: { enabled: true } }, + { utm_source: 'bbold/b' }, + ], + [ + 'pii filtering rejects the key', + `https://example.com?utm_source=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true } }, + {}, + ], + [ + 'pii redact mode keeps the key', + `https://example.com?utm_source=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true, mode: 'redact' } }, + { utm_source: '[REDACTED]' }, + ], + [ + 'lowercasing', + 'https://example.com?utm_source=LinkedIn', + { lowercaseValues: true }, + { utm_source: 'linkedin' }, + ], + [ + 'duplicate, last wins', + 'https://example.com?utm_source=a&utm_source=b', + undefined, + { utm_source: 'b' }, + ], + [ + 'duplicate whose last occurrence is PII takes the key with it', + `https://example.com?utm_source=good&utm_source=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true } }, + {}, + ], + [ + 'duplicate whose first occurrence is PII keeps the last', + `https://example.com?utm_source=${encodeURIComponent(EMAIL)}&utm_source=good`, + { piiFiltering: { enabled: true } }, + { utm_source: 'good' }, + ], + [ + 'sanitize gate leaves an empty string', + 'https://example.com?utm_source=BAD', + { sanitize: { enabled: true, valuePattern: /^[a-z]+$/ } }, + { utm_source: '' }, + ], + [ + 'non-utm params ignored', + 'https://example.com?foo=bar&utm_source=x', + undefined, + { utm_source: 'x' }, + ], + ['no utm params', 'https://example.com/page', undefined, {}], + ['malformed url', 'not a url', undefined, {}], + ] + + it.each(cases)('%s', (_name, url, options, expected) => { + expect(captureUtmParameters(url, options)).toEqual(expected) + }) + + it.each(cases)( + 'the report carries the same params for: %s', + (_name, url, options, expected) => { + expect(captureUtmParametersWithReport(url, options).params).toEqual(expected) + }, + ) + }) + + describe('callbacks', () => { + it('fires onCapture exactly once with the surviving params', () => { + const seen: unknown[] = [] + captureUtmParametersWithReport('https://example.com?utm_source=linkedin', { + onCapture: (p) => seen.push(p), + }) + expect(seen).toEqual([{ utm_source: 'linkedin' }]) + }) + + it('does not fire onCapture when everything was rejected', () => { + const seen: unknown[] = [] + captureUtmParametersWithReport( + `https://example.com?utm_source=${encodeURIComponent(EMAIL)}`, + { piiFiltering: { enabled: true }, onCapture: (p) => seen.push(p) }, + ) + expect(seen).toEqual([]) + }) + }) +}) diff --git a/__tests__/inbound/capture.test.ts b/__tests__/inbound/capture.test.ts index d801aef..ccc05c6 100644 --- a/__tests__/inbound/capture.test.ts +++ b/__tests__/inbound/capture.test.ts @@ -230,6 +230,161 @@ describe('sanitization integration', () => { }) }) +describe('duplicate query parameters', () => { + // URLSearchParams yields every occurrence. Capture is last-wins, and a + // rejected last occurrence must not resurrect an earlier accepted one — + // otherwise a link carrying `?utm_source=good&utm_source=` smuggles a + // value past the filter. + it('is last-wins for plain duplicates', () => { + const result = captureUtmParameters('https://example.com?utm_source=a&utm_source=b') + expect(result.utm_source).toBe('b') + }) + + it('drops the key when the LAST duplicate is rejected as PII', () => { + const result = captureUtmParameters( + 'https://example.com?utm_source=good&utm_source=a%40b.com', + { piiFiltering: { enabled: true } }, + ) + expect(result).toEqual({}) + }) + + it('keeps the last duplicate when an EARLIER one is rejected as PII', () => { + const result = captureUtmParameters( + 'https://example.com?utm_source=a%40b.com&utm_source=good', + { piiFiltering: { enabled: true } }, + ) + expect(result).toEqual({ utm_source: 'good' }) + }) + + it('drops the key when the last duplicate fails the PII allowlist', () => { + const result = captureUtmParameters('https://example.com?utm_source=good&utm_source=BAD', { + piiFiltering: { enabled: true, allowlistPattern: /^[a-z]+$/ }, + }) + expect(result).toEqual({}) + }) + + it('is last-wins when the last duplicate is dropped by a sanitize gate', () => { + const result = captureUtmParameters('https://example.com?utm_source=good&utm_source=BAD', { + sanitize: { enabled: true, valuePattern: /^[a-z]+$/ }, + }) + expect(result.utm_source).toBe('') + }) +}) + +describe('malformed configuration does not break the caller', () => { + // The pipeline runs on a page-load path. A misconfigured pattern is a + // consumer bug, but throwing out of capture turns it into a broken page. + it('returns empty when a PII pattern entry has no regex', () => { + const result = captureUtmParameters('https://example.com?utm_source=x', { + piiFiltering: { + enabled: true, + patterns: [{ name: 'broken', enabled: true } as unknown as never], + }, + }) + expect(result).toEqual({}) + }) + + it('does not throw when customPattern is not a RegExp', () => { + expect(() => + captureUtmParameters('https://example.com?utm_source=x', { + sanitize: { enabled: true, customPattern: 'abc' as unknown as RegExp }, + }), + ).not.toThrow() + }) + + it('does not throw when valuePattern is not a RegExp', () => { + expect(() => + captureUtmParameters('https://example.com?utm_source=x', { + sanitize: { enabled: true, valuePattern: 42 as unknown as RegExp }, + }), + ).not.toThrow() + }) +}) + +describe('onPiiDetected call parity', () => { + // The callback receives the RAW value, which the docblock says must never be + // logged. Firing it for a duplicate occurrence that never reaches the output + // hands consumers strictly more raw PII than before. + it('does not fire for a superseded duplicate occurrence', () => { + const calls: string[] = [] + captureUtmParameters('https://example.com?utm_source=someone%40example.com&utm_source=clean', { + piiFiltering: { enabled: true, onPiiDetected: (key) => calls.push(key) }, + }) + expect(calls).toEqual([]) + }) + + it('fires once for a surviving rejected value', () => { + const calls: string[] = [] + captureUtmParameters('https://example.com?utm_source=someone%40example.com', { + piiFiltering: { enabled: true, onPiiDetected: (key) => calls.push(key) }, + }) + expect(calls).toEqual(['utm_source']) + }) +}) + +describe('lowercaseValues', () => { + it('does not lowercase by default, preserving current behaviour', () => { + const result = captureUtmParameters('https://example.com?utm_source=LinkedIn') + expect(result.utm_source).toBe('LinkedIn') + }) + + it('folds values to lowercase when enabled', () => { + const result = captureUtmParameters('https://example.com?utm_source=LinkedIn', { + lowercaseValues: true, + }) + expect(result.utm_source).toBe('linkedin') + }) + + it('folds every value, not just the first', () => { + const result = captureUtmParameters( + 'https://example.com?utm_source=LinkedIn&utm_campaign=Spring2025', + { lowercaseValues: true }, + ) + expect(result).toEqual({ utm_source: 'linkedin', utm_campaign: 'spring2025' }) + }) + + it('leaves keys untouched', () => { + const result = captureUtmParameters('https://example.com?utm_source=LinkedIn', { + lowercaseValues: true, + }) + expect(Object.keys(result)).toEqual(['utm_source']) + }) + + it('works without sanitization enabled', () => { + const result = captureUtmParameters('https://example.com?utm_source=LinkedIn', { + lowercaseValues: true, + sanitize: { enabled: false }, + }) + expect(result.utm_source).toBe('linkedin') + }) + + // This is the test that pins the pipeline order. It can only pass if folding + // runs before the PII allowlist gate; a refactor that reorders the stages + // fails here. + it('folds before the PII allowlist gate, so a lowercase-only pattern accepts it', () => { + const result = captureUtmParameters('https://example.com?utm_source=LinkedIn', { + lowercaseValues: true, + piiFiltering: { enabled: true, allowlistPattern: /^[a-z]+$/ }, + }) + expect(result.utm_source).toBe('linkedin') + }) + + it('folds before the sanitize valuePattern gate', () => { + const result = captureUtmParameters('https://example.com?utm_source=LinkedIn', { + lowercaseValues: true, + sanitize: { enabled: true, valuePattern: /^[a-z]+$/ }, + }) + expect(result.utm_source).toBe('linkedin') + }) + + it('rejects a mixed-case value under a lowercase-only gate when folding is off', () => { + const result = captureUtmParameters('https://example.com?utm_source=LinkedIn', { + sanitize: { enabled: true, valuePattern: /^[a-z]+$/ }, + }) + expect(result.utm_source).toBe('') + }) +}) + describe('PII filtering integration', () => { const piiFilterConfig = { enabled: true, diff --git a/__tests__/inbound/sanitizer.test.ts b/__tests__/inbound/sanitizer.test.ts index 3f1d166..9384925 100644 --- a/__tests__/inbound/sanitizer.test.ts +++ b/__tests__/inbound/sanitizer.test.ts @@ -112,6 +112,69 @@ describe('sanitizeValue', () => { }) }) + describe('onMaxLength', () => { + it('truncates by default, preserving current behaviour', () => { + const result = sanitizeValue('a'.repeat(250), defaultConfig) + expect(result).toBe('a'.repeat(200)) + }) + + it('truncates when explicitly set to truncate', () => { + const config: SanitizeConfig = { ...defaultConfig, onMaxLength: 'truncate' } + const result = sanitizeValue('a'.repeat(250), config) + expect(result).toBe('a'.repeat(200)) + }) + + it('drops the whole value when set to drop', () => { + const config: SanitizeConfig = { ...defaultConfig, onMaxLength: 'drop' } + const result = sanitizeValue('a'.repeat(250), config) + expect(result).toBe('') + }) + + it('leaves a value exactly at maxLength untouched under truncate', () => { + const config: SanitizeConfig = { ...defaultConfig, maxLength: 10, onMaxLength: 'truncate' } + expect(sanitizeValue('a'.repeat(10), config)).toBe('a'.repeat(10)) + }) + + it('leaves a value exactly at maxLength untouched under drop', () => { + const config: SanitizeConfig = { ...defaultConfig, maxLength: 10, onMaxLength: 'drop' } + expect(sanitizeValue('a'.repeat(10), config)).toBe('a'.repeat(10)) + }) + + it('drops a value one character over maxLength', () => { + const config: SanitizeConfig = { ...defaultConfig, maxLength: 10, onMaxLength: 'drop' } + expect(sanitizeValue('a'.repeat(11), config)).toBe('') + }) + }) + + describe('valuePattern', () => { + it('keeps a value that matches the pattern', () => { + const config: SanitizeConfig = { ...defaultConfig, valuePattern: /^[a-z0-9_-]+$/ } + expect(sanitizeValue('spring-2025_campaign', config)).toBe('spring-2025_campaign') + }) + + it('drops a value that does not match the pattern', () => { + const config: SanitizeConfig = { ...defaultConfig, valuePattern: /^[a-z]+$/ } + expect(sanitizeValue('has spaces and 123', config)).toBe('') + }) + + it('is undefined by default, so any value survives', () => { + expect(sanitizeValue('anything at all !@#', defaultConfig)).toBe('anything at all !@#') + }) + + it('gives the same answer twice with a g-flagged pattern', () => { + const config: SanitizeConfig = { ...defaultConfig, valuePattern: /^[a-z]+$/g } + const first = sanitizeValue('linkedin', config) + const second = sanitizeValue('linkedin', config) + expect(first).toBe('linkedin') + expect(second).toBe(first) + }) + + it('tests the trimmed value, not the raw one', () => { + const config: SanitizeConfig = { ...defaultConfig, valuePattern: /^[a-z]+$/ } + expect(sanitizeValue(' linkedin ', config)).toBe('linkedin') + }) + }) + describe('edge cases', () => { it('returns empty string when everything is stripped', () => { const result = sanitizeValue('<>"\'`', defaultConfig) diff --git a/__tests__/react/UtmProvider.test.tsx b/__tests__/react/UtmProvider.test.tsx index d4f7f52..420ad4b 100644 --- a/__tests__/react/UtmProvider.test.tsx +++ b/__tests__/react/UtmProvider.test.tsx @@ -74,6 +74,36 @@ describe('UtmProvider', () => { expect(screen.getByTestId('params').textContent).toBe('{"utm_source":"auto_capture"}') }) + it('forwards lowercaseValues from config to the capture pipeline', () => { + vi.stubGlobal('location', { + href: 'https://example.com?utm_source=LinkedIn', + search: '?utm_source=LinkedIn', + }) + + render( + + + , + ) + + expect(screen.getByTestId('params').textContent).toBe('{"utm_source":"linkedin"}') + }) + + it('does not lowercase via config by default', () => { + vi.stubGlobal('location', { + href: 'https://example.com?utm_source=LinkedIn', + search: '?utm_source=LinkedIn', + }) + + render( + + + , + ) + + expect(screen.getByTestId('params').textContent).toBe('{"utm_source":"LinkedIn"}') + }) + it('uses custom storage key', () => { sessionStorage.setItem('custom_provider_key', '{"utm_source":"custom"}') diff --git a/__tests__/server/isolation.test.ts b/__tests__/server/isolation.test.ts new file mode 100644 index 0000000..da5c009 --- /dev/null +++ b/__tests__/server/isolation.test.ts @@ -0,0 +1,165 @@ +import { describe, it, expect } from 'vitest' +import { readFileSync, existsSync } from 'node:fs' +import { dirname, resolve, relative } from 'node:path' + +/** + * The /server entry's value rests on it being unable to reach browser-coupled + * code. That is an import restriction, so it is only real if something checks + * it — otherwise the next person to add a convenient re-export silently removes + * the guarantee. + * + * This walks the actual transitive import graph from the entry point rather + * than asserting on a hand-maintained list. + */ + +const SRC = resolve(__dirname, '../../src') +const ENTRY = resolve(SRC, 'server/index.ts') + +const FORBIDDEN = [ + 'common/storage', + 'inbound/form', + 'inbound/attribution', + 'outbound/decorator', + 'outbound/appender', + 'debug/index', + 'react/index', +] + +/** Resolve a relative specifier to a concrete .ts/.tsx file, if one exists. */ +function resolveModule(fromFile: string, specifier: string): string | null { + if (!specifier.startsWith('.')) { + return null + } + const base = resolve(dirname(fromFile), specifier) + for (const candidate of [`${base}.ts`, `${base}.tsx`, `${base}/index.ts`]) { + if (existsSync(candidate)) { + return candidate + } + } + return null +} + +/** + * Strip comments so a docblock mentioning sessionStorage is not read as a use of it. + * + * The `(?; bare: Set } { + const seen = new Set() + const bare = new Set() + const queue = [entry] + + /** Queue a specifier, recording bare ones and refusing to silently skip a broken relative one. */ + const follow = (file: string, specifier: string): void => { + if (!specifier.startsWith('.')) { + bare.add(specifier) + return + } + const resolved = resolveModule(file, specifier) + if (resolved === null) { + // An unresolvable relative specifier means the walk has a blind spot. That + // is a bug in this test, not a pass — fail loudly rather than under-report. + throw new Error(`isolation walk could not resolve '${specifier}' from ${file}`) + } + queue.push(resolved) + } + + while (queue.length > 0) { + const file = queue.pop() as string + if (seen.has(file)) { + continue + } + seen.add(file) + + const source = stripComments(readFileSync(file, 'utf8')) + const statements = [ + ...source.matchAll( + /(?:^|\n)\s*(import|export)\s+(type\s+)?([\s\S]*?)from\s*['"]([^'"]+)['"]/g, + ), + ] + + // Forms with no `from` clause, which the statement regex above cannot see: + // side-effect imports (`import './polyfill'`) and dynamic `import('./lazy')`. + // Neither exists in src/ today, but either would bypass the guarantee silently. + for (const match of [ + ...source.matchAll(/(?:^|\n)\s*import\s*['"]([^'"]+)['"]/g), + ...source.matchAll(/\bimport\s*\(\s*['"]([^'"]+)['"]\s*\)/g), + ]) { + follow(file, match[1]) + } + + for (const [, , typeOnly, clause, specifier] of statements) { + // Skip `import type X from` and a clause that is entirely `{ type A, type B }`. + if (typeOnly !== undefined) { + continue + } + const names = clause.replace(/[{}]/g, '').trim() + if (names !== '' && names.split(',').every((n) => n.trim().startsWith('type '))) { + continue + } + follow(file, specifier) + } + } + + return { files: seen, bare } +} + +describe('server entry isolation', () => { + const { files: reachable, bare } = reachableFrom(ENTRY) + const relativePaths = [...reachable].map((f) => relative(SRC, f).replace(/\.tsx?$/, '')) + + // Positive control. Asserting only the entry and its one-hop neighbour would + // still pass if the regex parsed nothing, so this pins modules reached through + // a real runtime import two hops out. + it('reaches its transitive runtime dependencies, proving the walk works', () => { + expect(relativePaths).toContain('server/index') + expect(relativePaths).toContain('server/normalize') + expect(relativePaths).toContain('inbound/sanitizer') + expect(relativePaths).toContain('inbound/pii-filter') + expect(relativePaths).toContain('config/defaults') + }) + + it('pulls in no third-party runtime dependency', () => { + // The package ships zero runtime dependencies. A bare specifier here would + // mean the server entry started depending on one — including 'react', which + // the react/ path check below cannot see because it only matches relatives. + expect([...bare]).toEqual([]) + }) + + it.each(FORBIDDEN)('does not reach %s', (forbidden) => { + expect(relativePaths).not.toContain(forbidden) + }) + + it('does not reach any react module', () => { + expect(relativePaths.filter((p) => p.startsWith('react/'))).toEqual([]) + }) + + it('does not reach any debug module', () => { + expect(relativePaths.filter((p) => p.startsWith('debug/'))).toEqual([]) + }) + + it('never touches window, document or web storage in reachable runtime code', () => { + const offenders: string[] = [] + for (const file of reachable) { + const source = stripComments(readFileSync(file, 'utf8')) + if (/\b(window|document|sessionStorage|localStorage)\b/.test(source)) { + offenders.push(relative(SRC, file)) + } + } + expect(offenders).toEqual([]) + }) +}) diff --git a/__tests__/server/normalize.test.ts b/__tests__/server/normalize.test.ts new file mode 100644 index 0000000..9770417 --- /dev/null +++ b/__tests__/server/normalize.test.ts @@ -0,0 +1,360 @@ +import { describe, it, expect } from 'vitest' +import { normalizeUtmParams, normalizeUtmUrl } from '../../src/server/normalize' +import { STANDARD_UTM_PARAMETERS } from '../../src/config/defaults' + +const EMAIL = 'someone@example.com' +const ALL_KEYS = [...STANDARD_UTM_PARAMETERS] + +describe('normalizeUtmParams', () => { + describe('totality', () => { + it('returns every allowed key for a full input', () => { + const { params } = normalizeUtmParams({ utm_source: 'linkedin' }) + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + }) + + it('returns every allowed key for an empty object', () => { + const { params } = normalizeUtmParams({}) + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + }) + + it("defaults absent parameters to '' so a composite key groups", () => { + const { params } = normalizeUtmParams({ utm_source: 'linkedin' }) + expect(params.utm_source).toBe('linkedin') + expect(params.utm_medium).toBe('') + expect(params.utm_campaign).toBe('') + expect(params.utm_term).toBe('') + expect(params.utm_content).toBe('') + expect(params.utm_id).toBe('') + }) + + it('honours a custom absentValue', () => { + const { params } = normalizeUtmParams({ utm_source: 'linkedin' }, { absentValue: '(none)' }) + expect(params.utm_source).toBe('linkedin') + expect(params.utm_medium).toBe('(none)') + }) + + it('honours a narrowed allowedParameters list', () => { + const { params } = normalizeUtmParams( + { utm_source: 'linkedin', utm_id: 'abc' }, + { allowedParameters: ['utm_source', 'utm_medium'] }, + ) + expect(Object.keys(params).sort()).toEqual(['utm_medium', 'utm_source']) + }) + + it('never returns undefined for any allowed key', () => { + const { params } = normalizeUtmParams(undefined) + for (const key of ALL_KEYS) { + expect(params[key]).toBeDefined() + expect(typeof params[key]).toBe('string') + } + }) + }) + + describe('hostile input never throws', () => { + const hostile: Array<[string, unknown]> = [ + ['undefined', undefined], + ['null', null], + ['a number', 42], + ['a string', 'utm_source=linkedin'], + ['an array', []], + ['a populated array', [1, 2, 3]], + ['a nested array value', { utm_source: ['a', 'b'] }], + ['a null value', { utm_source: null }], + ['an undefined value', { utm_source: undefined }], + ['an object value', { utm_source: { nested: true } }], + ['a numeric value', { utm_source: 42 }], + ['a boolean value', { utm_source: true }], + ['a function value', { utm_source: () => 'x' }], + ['a prototype-polluting key', JSON.parse('{"__proto__":{"polluted":true}}')], + ['a constructor key', { constructor: 'x', utm_source: 'linkedin' }], + ['a symbol-keyed object', { [Symbol('s')]: 'x', utm_source: 'linkedin' }], + ] + + it.each(hostile)('does not throw on %s', (_name, input) => { + expect(() => normalizeUtmParams(input)).not.toThrow() + }) + + it.each(hostile)('returns a total result for %s', (_name, input) => { + const { params } = normalizeUtmParams(input) + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + }) + + // JSON.parse cannot produce these, but "untrusted HTTP body" reaches this + // function through custom content parsers, ORM entities and framework + // reactive proxies too. The docblock promises any input, so it must mean it. + describe('exotic objects', () => { + it('does not throw on a throwing getter', () => { + const body = {} + Object.defineProperty(body, 'utm_source', { + get() { + throw new Error('hostile getter') + }, + enumerable: true, + configurable: true, + }) + expect(() => normalizeUtmParams(body)).not.toThrow() + }) + + it('treats a throwing getter as absent', () => { + const body = {} + Object.defineProperty(body, 'utm_source', { + get() { + throw new Error('hostile getter') + }, + enumerable: true, + configurable: true, + }) + const { params } = normalizeUtmParams(body) + expect(params.utm_source).toBe('') + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + }) + + it('does not throw on a proxy with a hostile trap', () => { + const hostile = new Proxy( + {}, + { + getOwnPropertyDescriptor() { + throw new Error('hostile trap') + }, + get() { + throw new Error('hostile trap') + }, + }, + ) + expect(() => normalizeUtmParams(hostile)).not.toThrow() + }) + + it('does not throw on a revoked proxy', () => { + const { proxy, revoke } = Proxy.revocable({}, {}) + revoke() + expect(() => normalizeUtmParams(proxy)).not.toThrow() + }) + + it('returns a total result for a revoked proxy', () => { + const { proxy, revoke } = Proxy.revocable({}, {}) + revoke() + const { params } = normalizeUtmParams(proxy) + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + }) + }) + + describe('malformed options', () => { + it('does not throw when a PII pattern entry has no regex', () => { + expect(() => + normalizeUtmParams( + { utm_source: 'x' }, + { piiFiltering: { patterns: [{ name: 'broken', enabled: true } as unknown as never] } }, + ), + ).not.toThrow() + }) + + it('does not throw when valuePattern is not a RegExp', () => { + expect(() => + normalizeUtmParams({ utm_source: 'x' }, { valuePattern: 42 as unknown as RegExp }), + ).not.toThrow() + }) + + it('stays total when a key fails to process', () => { + const { params } = normalizeUtmParams( + { utm_source: 'x' }, + { piiFiltering: { patterns: [{ name: 'broken', enabled: true } as unknown as never] } }, + ) + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + }) + + it.each([ + ['null', null], + ['a number', 42], + ['a string', 'utm_source'], + ['an object', {}], + ])('falls back to the standard keys when allowedParameters is %s', (_name, bad) => { + const { params } = normalizeUtmParams( + { utm_source: 'linkedin' }, + { allowedParameters: bad as unknown as string[] }, + ) + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + expect(params.utm_source).toBe('linkedin') + }) + + it('ignores non-string entries inside allowedParameters', () => { + const { params } = normalizeUtmParams( + { utm_source: 'linkedin' }, + { allowedParameters: ['utm_source', 42 as unknown as string] }, + ) + expect(Object.keys(params)).toEqual(['utm_source']) + }) + }) + + it('does not pollute Object.prototype', () => { + normalizeUtmParams(JSON.parse('{"__proto__":{"polluted":true}}')) + expect(({} as Record).polluted).toBeUndefined() + }) + + it('coerces a non-string value to absent rather than stringifying it', () => { + // String(['a','b']) is 'a,b' — a value nobody sent. + const { params } = normalizeUtmParams({ utm_source: ['a', 'b'] }) + expect(params.utm_source).toBe('') + }) + + it('reports a non-string value as rejected', () => { + const { rejected } = normalizeUtmParams({ utm_source: ['a', 'b'] }) + expect(rejected).toEqual([{ key: 'utm_source', reason: 'notAString' }]) + }) + + it('does not report a merely absent parameter as rejected', () => { + const { rejected } = normalizeUtmParams({}) + expect(rejected).toEqual([]) + }) + }) + + describe('folding', () => { + it('folds case and trims whitespace by default', () => { + const { params } = normalizeUtmParams({ utm_source: ' LinkedIn ' }) + expect(params.utm_source).toBe('linkedin') + }) + + it('produces identical output for equivalent inputs', () => { + const a = normalizeUtmParams({ utm_source: ' LinkedIn ' }) + const b = normalizeUtmParams({ utm_source: 'linkedin' }) + expect(a.params).toEqual(b.params) + }) + + it('can have folding disabled', () => { + const { params } = normalizeUtmParams({ utm_source: 'LinkedIn' }, { lowercase: false }) + expect(params.utm_source).toBe('LinkedIn') + }) + }) + + describe('server defaults differ from browser defaults, deliberately', () => { + it('filters PII by default', () => { + const { params, rejected } = normalizeUtmParams({ utm_source: EMAIL }) + expect(params.utm_source).toBe('') + expect(rejected).toEqual([{ key: 'utm_source', reason: 'pii', patternName: 'email' }]) + }) + + it('lowercases by default', () => { + expect(normalizeUtmParams({ utm_source: 'LinkedIn' }).params.utm_source).toBe('linkedin') + }) + + it('drops rather than truncates an over-length value by default', () => { + const { params, rejected } = normalizeUtmParams( + { utm_source: 'a'.repeat(50) }, + { maxLength: 10 }, + ) + expect(params.utm_source).toBe('') + expect(rejected).toEqual([{ key: 'utm_source', reason: 'maxLength' }]) + }) + + it('can be told to truncate instead', () => { + const { params } = normalizeUtmParams( + { utm_source: 'a'.repeat(50) }, + { maxLength: 10, onMaxLength: 'truncate' }, + ) + expect(params.utm_source).toBe('a'.repeat(10)) + }) + + it('applies a valuePattern gate when given one', () => { + const { params, rejected } = normalizeUtmParams( + { utm_source: 'has spaces' }, + { valuePattern: /^[a-z0-9_-]+$/ }, + ) + expect(params.utm_source).toBe('') + expect(rejected).toEqual([{ key: 'utm_source', reason: 'valuePattern' }]) + }) + + it('reports allowlist when piiFiltering.allowlistPattern rejects a value', () => { + const { params, rejected } = normalizeUtmParams( + { utm_source: 'has spaces' }, + { piiFiltering: { allowlistPattern: /^[a-z]+$/ } }, + ) + expect(params.utm_source).toBe('') + expect(rejected).toEqual([{ key: 'utm_source', reason: 'allowlist' }]) + }) + + it('can have PII filtering disabled', () => { + const { params } = normalizeUtmParams( + { utm_source: EMAIL }, + { piiFiltering: { enabled: false } }, + ) + expect(params.utm_source).toBe(EMAIL.toLowerCase()) + }) + }) + + describe('rejection is per-parameter, not per-request', () => { + it('keeps a good parameter while rejecting a bad sibling', () => { + const { params, rejected } = normalizeUtmParams({ + utm_source: 'linkedin', + utm_content: EMAIL, + }) + expect(params.utm_source).toBe('linkedin') + expect(params.utm_content).toBe('') + expect(rejected).toHaveLength(1) + expect(rejected[0].key).toBe('utm_content') + }) + }) + + describe('the result never carries a rejected value', () => { + it('does not contain the email anywhere in the serialised result', () => { + const result = normalizeUtmParams({ utm_content: EMAIL }) + const serialised = JSON.stringify(result) + expect(serialised).not.toContain(EMAIL) + expect(serialised).not.toContain('someone') + }) + }) + + describe('unknown keys', () => { + it('ignores keys outside allowedParameters without reporting them', () => { + const { params, rejected } = normalizeUtmParams({ + utm_source: 'linkedin', + unrelated: 'x', + utm_custom: 'y', + }) + expect(params.utm_source).toBe('linkedin') + expect(Object.keys(params)).not.toContain('unrelated') + expect(Object.keys(params)).not.toContain('utm_custom') + expect(rejected).toEqual([]) + }) + }) +}) + +describe('normalizeUtmUrl', () => { + it('extracts and normalizes from a URL', () => { + const { params } = normalizeUtmUrl('https://example.com/?utm_source=LinkedIn&utm_medium=Social') + expect(params.utm_source).toBe('linkedin') + expect(params.utm_medium).toBe('social') + }) + + it('is total, like normalizeUtmParams', () => { + const { params } = normalizeUtmUrl('https://example.com/') + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + }) + + it('does not throw on a malformed URL', () => { + expect(() => normalizeUtmUrl('not a url')).not.toThrow() + }) + + it('returns a total result for a malformed URL', () => { + const { params } = normalizeUtmUrl('not a url') + expect(Object.keys(params).sort()).toEqual([...ALL_KEYS].sort()) + expect(params.utm_source).toBe('') + }) + + it('does not throw on non-string input', () => { + expect(() => normalizeUtmUrl(undefined as unknown as string)).not.toThrow() + expect(() => normalizeUtmUrl(null as unknown as string)).not.toThrow() + expect(() => normalizeUtmUrl(42 as unknown as string)).not.toThrow() + }) + + it('applies PII filtering like normalizeUtmParams', () => { + const { params, rejected } = normalizeUtmUrl( + `https://example.com/?utm_source=${encodeURIComponent(EMAIL)}`, + ) + expect(params.utm_source).toBe('') + expect(rejected[0].reason).toBe('pii') + }) + + it('last-wins on duplicate query parameters, matching URLSearchParams', () => { + const { params } = normalizeUtmUrl('https://example.com/?utm_source=a&utm_source=b') + expect(params.utm_source).toBe('b') + }) +}) diff --git a/package.json b/package.json index 9199350..fb68553 100644 --- a/package.json +++ b/package.json @@ -26,6 +26,16 @@ "types": "./dist/react/index.d.cts", "default": "./dist/react/index.cjs" } + }, + "./server": { + "import": { + "types": "./dist/server/index.d.ts", + "default": "./dist/server/index.js" + }, + "require": { + "types": "./dist/server/index.d.cts", + "default": "./dist/server/index.cjs" + } } }, "files": [ diff --git a/src/config/defaults.ts b/src/config/defaults.ts index ddb3739..9b379ae 100644 --- a/src/config/defaults.ts +++ b/src/config/defaults.ts @@ -31,6 +31,7 @@ export const DEFAULT_SANITIZE_CONFIG: SanitizeConfig = { stripHtml: true, stripControlChars: true, maxLength: 200, + onMaxLength: 'truncate', } /** @@ -107,6 +108,9 @@ export const DEFAULT_CONFIG: ResolvedUtmConfig = { /** Capture standard UTM parameters by default */ allowedParameters: [...STANDARD_UTM_PARAMETERS], + /** Values are not folded to lowercase by default */ + lowercaseValues: false, + /** No default parameters set */ defaultParams: {}, diff --git a/src/config/docs.md b/src/config/docs.md index 512a52f..1dc9cf4 100644 --- a/src/config/docs.md +++ b/src/config/docs.md @@ -14,23 +14,27 @@ Path: @/src/config - `@/src/debug` imports `getDefaultConfig()` from here as a fallback when no config is provided to diagnostic functions. - `DEFAULT_CONFIG` and `STANDARD_UTM_PARAMETERS` are the canonical definitions of default behavior (enabled, snake_case, `storageType: 'session'`, sessionStorage key `utm_parameters`, no TTL, attribution mode `'last'`, auto-capture on mount, append to shares, the 6 standard UTM params). - `DEFAULT_ATTRIBUTION_CONFIG` defines attribution defaults: `mode: 'last'` with suffixes `_first` and `_last`. This preserves existing last-touch-only behavior when attribution is not explicitly configured. -- `DEFAULT_SANITIZE_CONFIG` defines the sanitization defaults: disabled by default, but with safe-by-default values when enabled. `DEFAULT_PII_PATTERNS` and `DEFAULT_PII_FILTER_CONFIG` define PII detection defaults. +- `DEFAULT_SANITIZE_CONFIG` defines the sanitization defaults: disabled by default, but with safe-by-default values when enabled, including `onMaxLength: 'truncate'` (over-length values are cut, not dropped). `DEFAULT_PII_PATTERNS` and `DEFAULT_PII_FILTER_CONFIG` define PII detection defaults. +- `@/src/server` reads `STANDARD_UTM_PARAMETERS` and `DEFAULT_PII_FILTER_CONFIG` directly from `defaults.ts`. This folder is pure data and transformation with no browser coupling, so it is one of the few modules the DOM-free server entry is permitted to reach. It then *overrides* several defaults (lowercase on, `onMaxLength: 'drop'`, PII filtering on) because browser leniency is wrong for a datastore key — see `@/src/server/docs.md`. - Event callbacks (`onCapture`, `onStore`, `onClear`, `onAppend`, `onExpire`) are passed through from user config via `createConfig()` and `mergeConfig()` -- they have no defaults (undefined when not provided). - The config system does not perform side effects -- it is pure data transformation. ### Core Implementation -- `createConfig()` merges a partial user config with defaults using nullish coalescing (`??`) for scalar fields, including `storageType`, `ttl`, and event callbacks. Array fields (`allowedParameters`, `excludeFromShares`) are replaced wholesale when provided by the user, not merged. Object fields (`defaultParams`, `shareContextParams`) are shallow-merged. Nested config objects (`sanitize`, `piiFiltering`, `attribution`) each have dedicated merge functions that apply nullish coalescing per-field so partial overrides preserve unspecified defaults. +- `createConfig()` merges a partial user config with defaults using nullish coalescing (`??`) for scalar fields, including `storageType`, `ttl`, `lowercaseValues`, and event callbacks. Array fields (`allowedParameters`, `excludeFromShares`) are replaced wholesale when provided by the user, not merged. Object fields (`defaultParams`, `shareContextParams`) are shallow-merged. Nested config objects (`sanitize`, `piiFiltering`, `attribution`) each have dedicated merge functions that apply nullish coalescing per-field so partial overrides preserve unspecified defaults. - `mergeAttributionConfig()` merges `mode`, `firstTouchSuffix`, and `lastTouchSuffix` with nullish coalescing, following the same pattern as other nested configs. - `mergeConfig()` follows the same semantics but takes a `ResolvedUtmConfig` as the base instead of implicitly using defaults -- useful for layering configurations. It also forwards event callbacks with nullish coalescing. - `loadConfigFromJson()` accepts `unknown` input, validates it is a non-null non-array object, then delegates to `createConfig()`. Invalid input falls back to defaults with a `console.warn`. -- `validateConfig()` performs runtime type checking on each config field and returns an array of error message strings (empty array means valid). It validates `storageType` as `'session'` or `'local'`, `ttl` as a positive finite number, plus nested validation for `sanitize` and `piiFiltering` sub-objects. +- `validateConfig()` performs runtime type checking on each config field and returns an array of error message strings (empty array means valid). It validates `storageType` as `'session'` or `'local'`, `ttl` as a positive finite number, `lowercaseValues` as a boolean, plus nested validation for `sanitize` (including `valuePattern` as a `RegExp` and `onMaxLength` as `'truncate' | 'drop'`) and `piiFiltering` sub-objects. Since the JSON path is how consumers hand-write config, a field it does not know about validates clean whatever it contains. - `getDefaultConfig()` returns a shallow copy of `DEFAULT_CONFIG` with cloned arrays and objects (including deep-copied PII patterns and attribution config) to prevent mutation of the shared constant. ### Things to Know - Array replacement (not merge) for `allowedParameters` is intentional: if a consumer provides `allowedParameters: ['utm_source']`, they get only that parameter, not the union with defaults. This is a deliberate design choice. - `STANDARD_UTM_PARAMETERS` is declared `as const` and used both as the default `allowedParameters` value and as the source of truth in tests. +- **Adding a config option means editing every link in a plumbing chain, and a missed link fails silently.** A flat option (`lowercaseValues` is the current example) has to be threaded through `UtmConfig` and `ResolvedUtmConfig` (in `@/src/types`), `DEFAULT_CONFIG`, `createConfig()`, `mergeConfig()` and `validateConfig()`, then forwarded in `@/src/react/useUtmTracking.ts`. A **nested** option additionally needs its per-field merge function — `mergeSanitizeConfig()` and friends rebuild their object field by field, so a field they do not name is dropped rather than passed through. +- **That per-field rebuild is the reason `sanitize` options can be unreachable while appearing to work.** A `SanitizeConfig` field omitted from `mergeSanitizeConfig()` still functions when passed directly to `captureUtmParameters`, but silently vanishes on every route through this folder — `createConfig()`, `UtmProvider`, `loadConfigFromJson()`. Testing the direct-options path only cannot see it. - `validateConfig()` and `createConfig()` are independent -- `createConfig()` does not call `validateConfig()`. Validation is opt-in for consumers who want to check config before using it. +- **`createConfig()` with no argument early-returns the defaults without merging.** It never reaches the nested merge functions, so it exercises none of the merge behaviour it appears to — a test that calls it that way cannot detect a dropped field. Created and maintained by Nori. diff --git a/src/config/loader.ts b/src/config/loader.ts index 624e983..345572d 100644 --- a/src/config/loader.ts +++ b/src/config/loader.ts @@ -56,7 +56,9 @@ function mergeSanitizeConfig( stripHtml: override.stripHtml ?? base.stripHtml, stripControlChars: override.stripControlChars ?? base.stripControlChars, maxLength: override.maxLength ?? base.maxLength, + onMaxLength: override.onMaxLength ?? base.onMaxLength, customPattern: override.customPattern ?? base.customPattern, + valuePattern: override.valuePattern ?? base.valuePattern, } } @@ -119,6 +121,7 @@ export function createConfig(userConfig?: Partial): ResolvedUtmConfig ttl: userConfig.ttl ?? defaults.ttl, captureOnMount: userConfig.captureOnMount ?? defaults.captureOnMount, appendToShares: userConfig.appendToShares ?? defaults.appendToShares, + lowercaseValues: userConfig.lowercaseValues ?? defaults.lowercaseValues, allowedParameters: userConfig.allowedParameters ? [...userConfig.allowedParameters] : defaults.allowedParameters, @@ -177,6 +180,7 @@ export function mergeConfig( ttl: override.ttl ?? base.ttl, captureOnMount: override.captureOnMount ?? base.captureOnMount, appendToShares: override.appendToShares ?? base.appendToShares, + lowercaseValues: override.lowercaseValues ?? base.lowercaseValues, allowedParameters: override.allowedParameters ? [...override.allowedParameters] : [...base.allowedParameters], @@ -275,6 +279,10 @@ export function validateConfig(config: unknown): string[] { errors.push('appendToShares must be a boolean') } + if (c.lowercaseValues !== undefined && typeof c.lowercaseValues !== 'boolean') { + errors.push('lowercaseValues must be a boolean') + } + if (c.allowedParameters !== undefined) { if (!Array.isArray(c.allowedParameters)) { errors.push('allowedParameters must be an array') @@ -332,6 +340,12 @@ export function validateConfig(config: unknown): string[] { if (s.customPattern !== undefined && !(s.customPattern instanceof RegExp)) { errors.push('sanitize.customPattern must be a RegExp') } + if (s.valuePattern !== undefined && !(s.valuePattern instanceof RegExp)) { + errors.push('sanitize.valuePattern must be a RegExp') + } + if (s.onMaxLength !== undefined && s.onMaxLength !== 'truncate' && s.onMaxLength !== 'drop') { + errors.push('sanitize.onMaxLength must be "truncate" or "drop"') + } } } diff --git a/src/debug/docs.md b/src/debug/docs.md index c4a8eff..11991f9 100644 --- a/src/debug/docs.md +++ b/src/debug/docs.md @@ -18,12 +18,14 @@ Path: @/src/debug ### Core Implementation - `getDiagnostics()` assembles a `DiagnosticInfo` snapshot: resolves config, captures URL params via `captureUtmParameters`, reads stored params via `getStoredUtmParameters` (passing `storageType` from config), and checks `isStorageAvailable(config.storageType)`. SSR-safe (returns empty URL and empty params when `window` is unavailable). +- **The capture call forwards the whole capture-shaping config**, not just `keyFormat` and `allowedParameters`: `lowercaseValues`, `sanitize` and `piiFiltering` go through too. Forwarding a subset makes diagnostics report a value that no real pipeline produces whenever those options are configured, turning the tool into the cause of the confusion it exists to resolve. - `debugUtmState()` calls `getDiagnostics()` and formats output using `console.group`/`console.table`. Logs `storageType` alongside key format and storage key. - `checkUtmTracking()` calls `getDiagnostics()` and returns an array of status strings with emoji prefixes indicating state. The storage-unavailable warning message dynamically uses `localStorage` or `sessionStorage` based on `config.storageType`. - `installDebugHelpers()` checks for `?debug_utm=true` in the URL query string. If present, it attaches a `window.utmDebug` object with `state()`, `check()`, `diagnostics()`, and `raw()` methods. The `raw()` helper reads from the correct storage backend based on `config.storageType`. ### Things to Know +- **`onCapture` is deliberately not forwarded to `captureUtmParameters`.** Inspecting state must not fire a consumer's side effects — calling `debugUtmState()` in a console should not emit an analytics event. This is the one capture option intentionally withheld, and the reason "forward the config" is not simply "spread the config". - `installDebugHelpers()` is gated solely by the `debug_utm=true` URL parameter. It does not check `process.env` or `import.meta.env.DEV`. - The `window.utmDebug` object is attached via a cast to `Record` to avoid TypeScript errors on the global augmentation. - `checkUtmTracking()` detects a potential timing issue: when URL params exist but storage is empty and `captureOnMount` is enabled, it warns that the hook may not have initialized yet. diff --git a/src/debug/index.ts b/src/debug/index.ts index 0e1a82f..bb56d65 100644 --- a/src/debug/index.ts +++ b/src/debug/index.ts @@ -32,10 +32,18 @@ export function getDiagnostics(config?: ResolvedUtmConfig): DiagnosticInfo { const currentUrl = isBrowser ? window.location.href : '' // Capture params from current URL + // Forward the whole capture-shaping config, not just keys. Diagnostics that + // skip sanitisation, PII filtering or folding report a value no real pipeline + // produced, which makes this tool the cause of the confusion it exists to + // resolve. onCapture is deliberately NOT forwarded: inspecting state must not + // fire a consumer's side effects. const urlParams = isBrowser ? captureUtmParameters(currentUrl, { keyFormat: resolvedConfig.keyFormat, allowedParameters: resolvedConfig.allowedParameters, + lowercaseValues: resolvedConfig.lowercaseValues, + sanitize: resolvedConfig.sanitize, + piiFiltering: resolvedConfig.piiFiltering, }) : {} diff --git a/src/docs.md b/src/docs.md index 1eca0a5..95bdf03 100644 --- a/src/docs.md +++ b/src/docs.md @@ -5,15 +5,16 @@ Path: @/src ### Overview - Root source directory for `@jackmisner/utm-toolkit`, a TypeScript library for capturing, storing, and appending UTM tracking parameters. -- Organized by data flow direction: `@/src/inbound` (receiving UTM-tagged traffic), `@/src/outbound` (creating UTM-tagged links), and `@/src/common` (shared utilities). Supplemented by `@/src/config`, `@/src/debug`, `@/src/types`, and an optional `@/src/react` integration. -- Exposes two package entry points: `@/src/index.ts` (main, imported as `@jackmisner/utm-toolkit`) and `@/src/react/index.ts` (imported as `@jackmisner/utm-toolkit/react`). +- Organized by data flow direction: `@/src/inbound` (receiving UTM-tagged traffic), `@/src/outbound` (creating UTM-tagged links), and `@/src/common` (shared utilities). Supplemented by `@/src/config`, `@/src/debug`, `@/src/types`, an optional `@/src/react` integration, and a DOM-free `@/src/server` surface. +- Exposes three package entry points: `@/src/index.ts` (main, `@jackmisner/utm-toolkit`), `@/src/react/index.ts` (`.../react`), and `@/src/server/index.ts` (`.../server`). ### How it fits into the larger codebase - `@/src/index.ts` is the main barrel export that re-exports everything from `inbound`, `outbound`, `common`, `config`, `debug`, and `types`. This is what consumers get when they `import from '@jackmisner/utm-toolkit'`. -- `@/src/react/index.ts` is the second entry point for React-specific exports, built as a separate bundle with React externalized. -- `@/tsup.config.ts` defines these two entry points and produces dual ESM/CJS output with TypeScript declarations. -- `@/__tests__` mirrors this directory structure (`inbound/`, `outbound/`, `common/`, `config/`, `react/`) for testing. +- `@/src/react/index.ts` is the React entry point, built as a separate bundle with React externalized. +- `@/src/server/index.ts` is the server entry point: UTM normalisation for untrusted ingest endpoints, with a hard structural rule that it cannot reach browser-coupled modules. See `@/src/server/docs.md`. +- `@/tsup.config.ts` defines these entry points and produces dual ESM/CJS output with TypeScript declarations; `package.json` `exports` mirrors them as conditional exports. +- `@/__tests__` mirrors this directory structure (`inbound/`, `outbound/`, `common/`, `config/`, `react/`, `server/`) for testing. - The library has zero runtime dependencies. React is an optional peer dependency used only by `@/src/react`. ### Core Implementation @@ -26,8 +27,14 @@ Consumer API +--> src/index.ts (barrel) -----> inbound/ outbound/ common/ config/ debug/ types/ | +--> src/react/index.ts --------> react/ (useUtmTracking, UtmProvider, UtmHiddenFields, UtmLinkDecorator) + | | + | +--> inbound/ outbound/ common/ config/ types/ + | + +--> src/server/index.ts -------> server/ (normalizeUtmParams, normalizeUtmUrl) | - +--> inbound/ outbound/ common/ config/ types/ + +--> inbound/sanitizer inbound/pii-filter config/defaults + X common/storage inbound/form outbound/* debug/ react/ + (forbidden; enforced by an import-graph test) ``` - **types/** (`@/src/types`): Shared type definitions consumed by all other modules. Defines the dual key format system (snake_case/camelCase), storage type, attribution mode, event callbacks, and configuration interfaces. @@ -37,6 +44,7 @@ Consumer API - **outbound/** (`@/src/outbound`): Creating UTM-tagged links -- append params to URLs, structured UTM URL builder, and automatic link decoration. - **debug/** (`@/src/debug`): Development-time diagnostics. Assembles state snapshots and provides formatted console output and optional `window.utmDebug` helpers. - **react/** (`@/src/react`): React hooks, context provider, and components that orchestrate the core modules into stateful React APIs with auto-capture-on-mount behavior, form field rendering, and link decoration. +- **server/** (`@/src/server`): DOM-free, stateless normalisation of untrusted UTM input for ingest endpoints. Reuses only the pure value-level primitives from `inbound/`, applies server-appropriate defaults, and returns a *total* parameter record safe to use as a datastore key. **Key data flow**: URL with UTM params --> `capture` (with optional sanitization and PII filtering) --> `storeWithAttribution` or `store` in sessionStorage/localStorage (with optional TTL, envelope format) --> `appendToUrl` / `buildUtmUrl` / `decorateLinks` for outbound link generation. @@ -46,7 +54,9 @@ Consumer API - **Envelope storage format**: All stored data uses an envelope `{ params, iat, eat }` where `iat` is issued-at timestamp and `eat` is expires-at (null for no expiry). The storage module reads both envelope and flat formats for backward compatibility. - **SSR safety**: Every module that touches browser APIs (`window`, `sessionStorage`, `localStorage`, `URL`, `document`, `MutationObserver`) guards against their absence. The library can be imported and initialized on the server without errors. - **Event callbacks**: Lifecycle hooks (`onCapture`, `onStore`, `onClear`, `onAppend`, `onExpire`) are all wrapped in try-catch so a failing callback never breaks the data pipeline. -- **Two entry points**: The package.json `exports` map defines separate conditional exports for `.` and `./react`, each with ESM/CJS/types variants. React is externalized in the build so it is not bundled into the output. +- **Three entry points**: The package.json `exports` map defines conditional exports for `.`, `./react`, and `./server`, each with ESM/CJS/types variants. React is externalized in the build so it is not bundled into the output. +- **Client-side capture is never trusted server-side**: `@/src/inbound` runs the pipeline in the browser, but a public ingest endpoint can be POSTed to directly, so `@/src/server` runs equivalent rules again on untrusted input. Both sides share the same value-level primitives so the rules cannot drift. +- **`/server` isolation is structural and enforced**: `@/src/server` must not reach storage, DOM, or React modules at runtime. An import-graph test in `@/__tests__/server` walks the actual transitive runtime imports rather than a hand-maintained list, so a convenient re-export cannot silently break the guarantee. Note that the root entry does *not* crash in a DOM-free Node context — `/server` exists for the guaranteed surface, server-appropriate defaults, and bundle size, not to work around a crash. - **No runtime dependencies**: The library is self-contained. All functionality is implemented from scratch using standard Web APIs (`URL`, `URLSearchParams`, `sessionStorage`, `localStorage`, `MutationObserver`). Created and maintained by Nori. diff --git a/src/inbound/capture-report.ts b/src/inbound/capture-report.ts new file mode 100644 index 0000000..df869c6 --- /dev/null +++ b/src/inbound/capture-report.ts @@ -0,0 +1,260 @@ +/** + * UTM Capture Reporting + * + * Captures UTM parameters and reports what was rejected on the way, so a + * consumer can tell "no campaign" apart from "campaign rejected". + * + * An empty result means both things today: genuine direct traffic, and a + * misconfigured campaign link whose every parameter was filtered. Collapsing + * them inflates the direct-traffic denominator that every campaign share is + * measured against. + */ + +import type { PiiFilterConfig, SanitizeConfig, UtmParameters } from '../types' +import { DEFAULT_PII_FILTER_CONFIG, DEFAULT_SANITIZE_CONFIG } from '../config/defaults' +import { convertParams, isSnakeCaseUtmKey } from '../common/keys' +import { filterValueWithReport } from './pii-filter' +import { sanitizeValueWithReport } from './sanitizer' +// Type-only: erased at build time, so this does not create an import cycle with +// capture.ts, which imports this module's runtime function. +import type { CaptureOptions } from './capture' + +/** + * Why a UTM parameter was discarded during capture + * + * - `allowedParameters` — the key was not in the configured allowlist + * - `valuePattern` — the value failed `sanitize.valuePattern` + * - `maxLength` — the value exceeded `sanitize.maxLength` under `onMaxLength: 'drop'` + * - `allowlist` — the value failed `piiFiltering.allowlistPattern` + * - `pii` — a PII detection pattern matched; see `patternName` + * - `notAString` — the value was not a string (server-side only; a URL's + * search parameters are always strings, so this cannot arise from capture) + */ +export type UtmRejectionReason = + 'allowedParameters' | 'valuePattern' | 'maxLength' | 'allowlist' | 'pii' | 'notAString' + +/** + * A single rejected UTM parameter + * + * Carries the key and the reason only. The rejected VALUE is deliberately + * absent: `PiiFilterConfig.onPiiDetected` already warns that raw values must + * not be logged or transmitted, and a report struct carrying one would be + * handed straight to a logger by most consumers who use it. + */ +export interface UtmRejection { + /** + * The parameter key, e.g. 'utm_content'. + * + * UNTRUSTED. Any `utm_`-prefixed query parameter is captured, so this comes + * straight from the URL and an attacker controls it — `?utm_someone@example.com=1` + * yields a rejection whose key contains an email address. Filter to the keys + * you expect before logging a report. + */ + key: string + + /** Why the parameter was discarded */ + reason: UtmRejectionReason + + /** For reason 'pii', the name of the matching pattern ('email', 'phone_uk', …) */ + patternName?: string +} + +/** + * The result of a capture, plus what was rejected getting there + */ +export interface CaptureReport { + /** The parameters that survived the pipeline */ + params: UtmParameters + + /** + * Every parameter discarded during capture, in pipeline order. + * + * Unbounded: a URL carrying 500 offending parameters produces 500 entries. + * Cap it before logging or emitting metrics keyed on its contents. + */ + rejected: UtmRejection[] + + /** + * True when the URL could not be parsed at all. + * + * Distinct from an empty `rejected` list: an unparseable URL is neither + * direct traffic nor a rejected campaign, and reporting it as either would be + * wrong. + */ + invalidUrl: boolean +} + +/** + * Check if we're in a browser environment with access to window + */ +function isBrowser(): boolean { + return typeof window !== 'undefined' && typeof window.location !== 'undefined' +} + +/** + * Extract UTM parameters from a URL and report what was rejected + * + * Applies the same pipeline as `captureUtmParameters` — allowlist → lowercase → + * sanitize → PII filter → key format — and additionally records every parameter + * discarded along the way. + * + * @param url - The URL to extract UTM parameters from (defaults to window.location.href) + * @param options - Capture options, identical to `captureUtmParameters` + * @returns The surviving parameters, the rejections, and whether the URL parsed + * + * @example + * ```typescript + * const { params, rejected } = captureUtmParametersWithReport(url, { + * piiFiltering: { enabled: true }, + * }) + * if (Object.keys(params).length === 0 && rejected.length > 0) { + * // A campaign link arrived but every parameter was filtered — + * // this is NOT direct traffic. + * } + * ``` + */ +export function captureUtmParametersWithReport( + url?: string, + options: CaptureOptions = {}, +): CaptureReport { + const { + keyFormat = 'snake_case', + allowedParameters, + lowercaseValues = false, + sanitize, + piiFiltering, + onCapture, + } = options + + const rejected: UtmRejection[] = [] + + // Get URL, defaulting to current page URL in browser + const urlString = url ?? (isBrowser() ? window.location.href : '') + + // SSR safety: no URL is absence, not a parse failure + if (!urlString) { + return { params: {}, rejected, invalidUrl: false } + } + + let urlObj: URL + try { + urlObj = new URL(urlString) + } catch (error) { + if (typeof console !== 'undefined' && console.warn) { + console.warn( + 'Failed to parse URL for UTM parameters:', + error instanceof Error ? error.message : 'Unknown error', + ) + } + return { params: {}, rejected, invalidUrl: true } + } + + const resolvedSanitize: SanitizeConfig = { ...DEFAULT_SANITIZE_CONFIG, ...sanitize } + const resolvedPiiFilter: PiiFilterConfig = { + ...DEFAULT_PII_FILTER_CONFIG, + ...piiFiltering, + patterns: piiFiltering?.patterns ?? [...DEFAULT_PII_FILTER_CONFIG.patterns], + } + + const allowedSet = + allowedParameters && allowedParameters.length > 0 ? new Set(allowedParameters) : null + + // Everything past the URL parse runs inside one guard. A malformed + // caller-supplied regex or PII pattern is a consumer bug, but capture runs on + // a page-load path, so it must degrade to "no parameters" rather than throw — + // the behaviour the pre-refactor implementation had, and callers rely on. + try { + // PHASE 1: collect, last-wins. Duplicated query parameters must resolve + // BEFORE the gates run, so that a rejected later occurrence takes the key + // with it rather than leaving an earlier accepted duplicate standing, and so + // that onPiiDetected fires only for the value that actually survives. + const collected: Record = {} + const reportedKeys = new Set() + + for (const [key, rawValue] of urlObj.searchParams.entries()) { + // Only capture parameters that start with 'utm_' (case-sensitive) + if (!isSnakeCaseUtmKey(key)) { + continue + } + + if (allowedSet !== null && !allowedSet.has(key)) { + // Report a disallowed key once, however many times it appears. + if (!reportedKeys.has(key)) { + reportedKeys.add(key) + rejected.push({ key, reason: 'allowedParameters' }) + } + continue + } + + // Fold before every gate below, so patterns can assume lowercase input. + // toLowerCase(), never toLocaleLowerCase(): folding must not depend on locale. + collected[key] = lowercaseValues ? rawValue.toLowerCase() : rawValue + } + + // PHASE 2: gate the surviving value for each key. + const captured: Record = {} + + for (const [key, initialValue] of Object.entries(collected)) { + let value = initialValue + + if (resolvedSanitize.enabled) { + const result = sanitizeValueWithReport(value, resolvedSanitize) + if (result.rejected) { + rejected.push({ key, reason: result.rejected }) + } + // The key stays, carrying ''. hasUtmParameters already treats '' as absent, + // so it is the established "no value" sentinel; dropping the key here would + // be a second, inconsistent one. (PII reject mode does drop the key — that + // is pre-existing behaviour and is preserved below.) + value = result.value + } + + if (resolvedPiiFilter.enabled) { + const result = filterValueWithReport(key, value, resolvedPiiFilter) + if (result.rejected) { + rejected.push({ key, reason: result.rejected.reason, ...pattern(result.rejected) }) + } + // In redact mode a rejected value survives as '[REDACTED]'; in reject mode + // it is undefined and the key is dropped entirely. + if (result.value === undefined) { + continue + } + value = result.value + } + + captured[key] = value + } + + const params: UtmParameters = + keyFormat === 'camelCase' + ? convertParams(captured as UtmParameters, 'camelCase') + : (captured as UtmParameters) + + if (onCapture && Object.keys(params).length > 0) { + try { + onCapture(params) + } catch { + // Callbacks must not break the pipeline + } + } + + return { params, rejected, invalidUrl: false } + } catch (error) { + if (typeof console !== 'undefined' && console.warn) { + console.warn( + 'Failed to capture UTM parameters:', + error instanceof Error ? error.message : 'Unknown error', + ) + } + // invalidUrl stays false: the URL parsed fine, the pipeline did not. + return { params: {}, rejected, invalidUrl: false } + } +} + +/** + * Spread helper that omits `patternName` entirely when there isn't one, so the + * rejection object stays free of `undefined` keys that would surface in JSON. + */ +function pattern(rejection: { patternName?: string }): { patternName?: string } { + return rejection.patternName === undefined ? {} : { patternName: rejection.patternName } +} diff --git a/src/inbound/capture.ts b/src/inbound/capture.ts index 3e6b94f..dadad50 100644 --- a/src/inbound/capture.ts +++ b/src/inbound/capture.ts @@ -6,10 +6,7 @@ */ import type { KeyFormat, PiiFilterConfig, SanitizeConfig, UtmParameters } from '../types' -import { DEFAULT_PII_FILTER_CONFIG, DEFAULT_SANITIZE_CONFIG } from '../config/defaults' -import { convertParams, isSnakeCaseUtmKey } from '../common/keys' -import { filterParams } from './pii-filter' -import { sanitizeParams } from './sanitizer' +import { captureUtmParametersWithReport } from './capture-report' /** * Options for capturing UTM parameters @@ -21,6 +18,19 @@ export interface CaptureOptions { /** Allowlist of parameters to capture (snake_case format, e.g., ['utm_source', 'utm_campaign']) */ allowedParameters?: string[] + /** + * Lowercase all captured values (default: false) + * + * Mirrors `BuildUtmUrlOptions.lowercaseValues` on the outbound side. Applied + * before sanitization and PII filtering, so every downstream gate — + * `sanitize.customPattern`, `sanitize.valuePattern` and + * `piiFiltering.allowlistPattern` — sees the folded value and can be written + * without allowing uppercase. + * + * Keys are unaffected; only values are folded. + */ + lowercaseValues?: boolean + /** Sanitization configuration — when enabled, strips dangerous characters from values */ sanitize?: Partial @@ -71,76 +81,10 @@ function isBrowser(): boolean { * ``` */ export function captureUtmParameters(url?: string, options: CaptureOptions = {}): UtmParameters { - const { keyFormat = 'snake_case', allowedParameters, sanitize, piiFiltering, onCapture } = options - - // Get URL, defaulting to current page URL in browser - const urlString = url ?? (isBrowser() ? window.location.href : '') - - // SSR safety: return empty object if no URL available - if (!urlString) { - return {} - } - - try { - // Parse the URL to extract query parameters - const urlObj = new URL(urlString) - const params: Record = {} - - // Create a set of allowed parameters for O(1) lookup - const allowedSet = - allowedParameters && allowedParameters.length > 0 ? new Set(allowedParameters) : null - - // Iterate through all query parameters - for (const [key, value] of urlObj.searchParams.entries()) { - // Only capture parameters that start with 'utm_' (case-sensitive) - if (isSnakeCaseUtmKey(key)) { - // If allowedParameters is provided, check if this parameter is allowed - if (allowedSet === null || allowedSet.has(key)) { - params[key] = value - } - } - } - - // Apply sanitization if configured and enabled - const resolvedSanitize: SanitizeConfig = { ...DEFAULT_SANITIZE_CONFIG, ...sanitize } - const sanitized: UtmParameters = resolvedSanitize.enabled - ? sanitizeParams(params as UtmParameters, resolvedSanitize) - : (params as UtmParameters) - - // Apply PII filtering if configured and enabled - const resolvedPiiFilter: PiiFilterConfig = { - ...DEFAULT_PII_FILTER_CONFIG, - ...piiFiltering, - patterns: piiFiltering?.patterns ?? [...DEFAULT_PII_FILTER_CONFIG.patterns], - } - const captured: UtmParameters = resolvedPiiFilter.enabled - ? filterParams(sanitized, resolvedPiiFilter) - : sanitized - - // Convert to target format if needed - const result = keyFormat === 'camelCase' ? convertParams(captured, 'camelCase') : captured - - // Fire onCapture callback if params were found - if (onCapture && Object.keys(result).length > 0) { - try { - onCapture(result) - } catch { - // Callbacks must not break the pipeline - } - } - - return result - } catch (error) { - // If URL parsing fails, return empty object - // This ensures the function is robust and doesn't break the app - if (typeof console !== 'undefined' && console.warn) { - console.warn( - 'Failed to parse URL for UTM parameters:', - error instanceof Error ? error.message : 'Unknown error', - ) - } - return {} - } + // Delegates so there is exactly one capture pipeline. Use + // captureUtmParametersWithReport directly when you need to tell "no campaign" + // apart from "campaign rejected". + return captureUtmParametersWithReport(url, options).params } /** diff --git a/src/inbound/docs.md b/src/inbound/docs.md index fa8a4e8..14ec01b 100644 --- a/src/inbound/docs.md +++ b/src/inbound/docs.md @@ -5,31 +5,63 @@ Path: @/src/inbound ### Overview - Utilities for the inbound data path: receiving UTM-tagged traffic, processing captured parameters, and routing them into storage or form fields. -- Includes URL capture, value sanitization, PII filtering, first-touch/last-touch attribution, and form field population. +- Includes URL capture, capture rejection reporting, value sanitization, PII filtering, first-touch/last-touch attribution, and form field population. - All exports are re-exported through `@/src/index.ts` to package consumers. ### How it fits into the larger codebase - `@/src/react/useUtmTracking.ts` calls `captureUtmParameters` from this module during its mount-time capture flow. - `@/src/debug` imports `captureUtmParameters` to build diagnostic snapshots. -- Attribution (`attribution.ts`) and form (`form.ts`) import storage functions from `@/src/common/storage`. +- `@/src/server` reuses the *value-level* primitives here (`sanitizeValueWithReport`, `filterValueWithReport`) so server-side normalisation applies the same rules from the same code. It does not reuse the capture pipeline, which is URL-shaped and browser-defaulted. It also imports the `UtmRejection` type from `capture-report.ts` type-only, so server and browser rejections share one vocabulary without pulling browser-coupled code into the server bundle. +- Attribution (`attribution.ts`) and form (`form.ts`) import storage functions from `@/src/common/storage`. These modules are on the forbidden list for `@/src/server` and are structurally unreachable from it. - Sanitizer and PII filter are invoked during capture when their respective config options are enabled. The config objects (`SanitizeConfig`, `PiiFilterConfig`) come from `@/src/types` and are resolved in `@/src/config`. - Types (`AttributionConfig`, `TouchType`, `AttributionMode`, `PiiPattern`, etc.) come from `@/src/types`. ### Core Implementation -**Capture (`capture.ts`)** extracts UTM parameters from a URL string: -- Parses the URL via the `URL` constructor, iterates `searchParams`, and filters to allowed parameter names. -- Applies value sanitization (if `sanitize.enabled`) and PII filtering (if `piiFiltering.enabled`) as part of the capture pipeline. -- `hasUtmParameters()` checks a `UtmParameters` object for any defined values. +**Capture (`capture.ts` + `capture-report.ts`)** — there is exactly **one** capture pipeline, and it lives in `capture-report.ts`: -**Sanitizer (`sanitizer.ts`)** cleans UTM parameter values: -- `sanitizeValue()` strips HTML characters, control characters, applies custom patterns, and truncates to max length. -- `sanitizeParams()` applies sanitization to all values in a `UtmParameters` object. +```text +capture.ts :: captureUtmParameters(url, options) <-- thin wrapper, unchanged signature + | + +--> capture-report.ts :: captureUtmParametersWithReport(url, options).params + +URL parse --(throws)--> { params: {}, rejected: [], invalidUrl: TRUE } + | + v ......... everything below runs inside ONE try/catch ......... + PHASE 1 (per query parameter, in URL order) + isSnakeCaseUtmKey -> allowedParameters -> lowercaseValues -> collected[key] + | | LAST-WINS + | +-- rejected once per key (reportedKeys Set) + v + PHASE 2 (per surviving key, one value each) + sanitize -> PII filter -> captured[key] + | + v + keyFormat conversion -> onCapture -> { params, rejected, invalidUrl: false } + | + +--(any throw)--> console.warn, { params: {}, rejected, invalidUrl: FALSE } +``` + +- `captureUtmParametersWithReport` returns `{ params, rejected, invalidUrl }`. It exists so a consumer can tell *"no campaign"* apart from *"campaign arrived and every parameter was filtered"* — collapsing the two inflates the direct-traffic denominator that every campaign share is measured against. `invalidUrl` is a third state again: an unparseable URL is neither. +- **The two phases are ordered the way they are because duplicates must resolve before the gates run.** `?utm_source=good&utm_source=` has to yield `{}`: if each occurrence were gated as it was read, rejecting the later value would leave the earlier accepted `'good'` standing and a value would slip past the PII filter. Collecting last-wins first also means `onPiiDetected` — which receives the **raw** value — fires only for the value that actually survives, rather than for superseded occurrences, so a consumer is never handed raw PII for a value the pipeline discarded anyway. +- A key blocked by `allowedParameters` is reported **once**, however many times it appears in the query string, tracked by a `reportedKeys` Set during phase 1. +- `capture.ts` imports the runtime function from `capture-report.ts`; `capture-report.ts` imports the `CaptureOptions` **type** back from `capture.ts`. The back-edge is type-only, erased at build, so there is no runtime cycle. +- Per-key processing (not batch `sanitizeParams`/`filterParams`) is what makes rejection attributable to a key. +- `hasUtmParameters()` checks a `UtmParameters` object for any defined values, treating `''` as absent. + +**Sanitizer (`sanitizer.ts`)** cleans UTM parameter values. Rule order is: + +`stripHtml -> stripControlChars -> customPattern -> trim -> valuePattern -> maxLength` + +- `customPattern` is **subtractive** (every match is removed); `valuePattern` is a **positive allowlist gate** (the value is kept intact or dropped whole). `valuePattern` is tested *after* the trim, deliberately, so a value whose only offence is surrounding whitespace this function was about to remove is not rejected. +- `onMaxLength` selects `'truncate'` (cut to `maxLength`, the default and the pre-existing behaviour) or `'drop'` (replace with `''`). +- Rejected values become `''` rather than having the key removed: `hasUtmParameters` already treats `''` as absent, so it is the established "no value" sentinel and a second one would be inconsistent. PII reject mode *does* drop the key — that is pre-existing behaviour and is preserved. +- `sanitizeValueWithReport()` is the real implementation, returning `{ value, rejected? }` where `rejected` is `'maxLength' | 'valuePattern'`. `sanitizeValue()` is a thin wrapper returning `.value`. **PII Filter (`pii-filter.ts`)** detects and handles personally identifiable information in parameter values: - `detectPii()` checks a value against enabled patterns and returns the matching pattern name. -- `filterValue()` either rejects (returns undefined) or redacts (replaces with `[REDACTED]`) based on config mode. +- `filterValueWithReport()` is the real implementation, returning `{ value, rejected? }` where `rejected` is `{ reason: 'pii' | 'allowlist', patternName? }`. `filterValue()` is a thin wrapper returning `.value` — undefined in reject mode, `'[REDACTED]'` in redact mode. - `filterParams()` applies filtering to all values in a `UtmParameters` object. **Attribution (`attribution.ts`)** handles first-touch / last-touch storage: @@ -51,5 +83,12 @@ Path: @/src/inbound - **Attribution writes the main key in all modes**: Even in `'first'` and `'both'` modes, the main storage key (without suffix) is always written with the current params. The suffixed keys provide the historical first/last values. - **First-touch is write-once**: `storeWithAttribution` checks `hasStoredUtmParameters` for the first-touch key before writing. Once set, first-touch params are never overwritten. - **Data-attribute strategy strips utm_ prefix**: In the `'data-attribute'` strategy, `populateByDataAttribute` strips the `utm_` (or `utm`) prefix and lowercases the remainder to build the short name used in attribute matching (e.g., `utm_source` -> `source`). +- **Reports never carry the rejected value.** `UtmRejection` holds a key, a reason, and for PII the pattern *name* only. `PiiFilterConfig.onPiiDetected` already warns that raw values must not be logged or transmitted, and a report struct carrying one would be handed straight to a logger by most consumers who use it. `patternName` is omitted entirely rather than set to `undefined`, so it does not surface in JSON. +- **`lowercaseValues` is folded before every gate**, so `sanitize.customPattern`, `sanitize.valuePattern`, and `piiFiltering.allowlistPattern` can all be written assuming lowercase input. It uses `toLowerCase()`, never `toLocaleLowerCase()` — folding must not depend on locale. Keys are never folded. It lives on `CaptureOptions` rather than `SanitizeConfig` to mirror `BuildUtmUrlOptions.lowercaseValues` on the outbound side, which means it must also be threaded through `UtmConfig` in `@/src/config` for the React path. +- **A value reduced to `''` by ordinary stripping is not a rejection.** That outcome predates the report; reporting it would hand every consumer spurious rejections from long-standing behaviour. Only the gates report. +- **Capture never throws, because it runs on a page-load path.** The whole pipeline past the URL parse sits inside one guard: a consumer's malformed regex or half-built PII pattern (a `PiiPattern` with `enabled: true` and no `regex`, say) degrades to "no parameters" plus a `console.warn`, not a broken page. Its equivalent on the server side is the per-key guard in `@/src/server/normalize.ts`, which degrades one key to `absentValue` rather than failing the request. +- **`invalidUrl` means URL-parse failure specifically, not pipeline failure.** The pipeline guard returns `invalidUrl: false` — the URL parsed fine, the processing did not. A consumer branching on `invalidUrl` to decide "the caller handed us garbage" would be wrong to read it as "something went wrong". +- **`UtmRejection.key` is attacker-controlled and `CaptureReport.rejected` is unbounded.** Any `utm_`-prefixed query parameter is captured, so `?utm_someone@example.com=1` produces a rejection whose *key* contains an email, and a URL carrying hundreds of offending parameters produces hundreds of entries. The report withholds rejected **values** but not untrusted **keys**; filter to expected keys and cap the array before logging one or emitting metrics keyed on its contents. +- **The single-pipeline invariant matters.** Any change to capture semantics belongs in `capture-report.ts`; `captureUtmParameters` cannot drift from it because it is a delegation, not a copy. The flip side is that a test comparing the two proves nothing — see `@/__tests__/docs.md`. Created and maintained by Nori. diff --git a/src/inbound/index.ts b/src/inbound/index.ts index 2ea9292..e744dc3 100644 --- a/src/inbound/index.ts +++ b/src/inbound/index.ts @@ -13,11 +13,32 @@ export { type CaptureOptions, } from './capture' +// Capture reporting — tells "no campaign" apart from "campaign rejected" +export { + captureUtmParametersWithReport, + type UtmRejection, + type UtmRejectionReason, + type CaptureReport, +} from './capture-report' + // Sanitizer utilities -export { sanitizeValue, sanitizeParams } from './sanitizer' +export { + sanitizeValue, + sanitizeParams, + sanitizeValueWithReport, + type SanitizeRejection, + type SanitizeValueResult, +} from './sanitizer' // PII filter utilities -export { detectPii, filterValue, filterParams } from './pii-filter' +export { + detectPii, + filterValue, + filterParams, + filterValueWithReport, + type PiiRejection, + type FilterValueResult, +} from './pii-filter' // Form field population export { populateFormFields, createUtmHiddenFields, type FormPopulateOptions } from './form' diff --git a/src/inbound/pii-filter.ts b/src/inbound/pii-filter.ts index 3506d6d..09127f6 100644 --- a/src/inbound/pii-filter.ts +++ b/src/inbound/pii-filter.ts @@ -44,8 +44,49 @@ export function filterValue( value: string, config: PiiFilterConfig, ): string | undefined { + return filterValueWithReport(key, value, config).value +} + +/** + * Why the PII filter rejected a value. + * + * `'allowlist'` means the value failed `allowlistPattern`; `'pii'` means a + * detection pattern matched, and `patternName` names it. + */ +export interface PiiRejection { + reason: 'pii' | 'allowlist' + patternName?: string +} + +/** + * Result of PII-filtering a value, with the reason it was rejected if it was. + */ +export interface FilterValueResult { + /** The value, `undefined` in reject mode, or `'[REDACTED]'` in redact mode */ + value: string | undefined + /** Set only when the filter rejected the value */ + rejected?: PiiRejection +} + +/** + * Filter a value for PII and report why it was rejected + * + * Same rules as {@link filterValue}; this variant additionally reports which + * check rejected the value. The rejected value itself is never included in the + * result — see the warning on `PiiFilterConfig.onPiiDetected`. + * + * @param key - The parameter key (for callback reporting) + * @param value - The parameter value to check + * @param config - PII filter configuration + * @returns The filtered value plus an optional rejection reason + */ +export function filterValueWithReport( + key: string, + value: string, + config: PiiFilterConfig, +): FilterValueResult { if (!config.enabled) { - return value + return { value } } // Allowlist check takes precedence @@ -57,10 +98,13 @@ export function filterValue( } catch { // Callback errors should not break the filter pipeline } - return config.mode === 'redact' ? '[REDACTED]' : undefined + return { + value: config.mode === 'redact' ? '[REDACTED]' : undefined, + rejected: { reason: 'allowlist' }, + } } // Value passes allowlist — no further checks needed - return value + return { value } } // Pattern-based PII detection @@ -71,10 +115,13 @@ export function filterValue( } catch { // Callback errors should not break the filter pipeline } - return config.mode === 'redact' ? '[REDACTED]' : undefined + return { + value: config.mode === 'redact' ? '[REDACTED]' : undefined, + rejected: { reason: 'pii', patternName: detected.name }, + } } - return value + return { value } } /** diff --git a/src/inbound/sanitizer.ts b/src/inbound/sanitizer.ts index 318d224..9d24127 100644 --- a/src/inbound/sanitizer.ts +++ b/src/inbound/sanitizer.ts @@ -10,15 +10,53 @@ import type { SanitizeConfig, UtmParameters } from '../types' /** * Sanitize a single UTM parameter value * - * Applies stripping rules in order: HTML chars → control chars → custom pattern → trim → truncate. + * Applies rules in order: HTML chars → control chars → custom pattern → trim → + * value pattern gate → maxLength handling. * * @param value - The raw parameter value * @param config - Sanitization configuration * @returns Sanitized value */ export function sanitizeValue(value: string, config: SanitizeConfig): string { + return sanitizeValueWithReport(value, config).value +} + +/** + * Why `sanitizeValue` discarded a value outright. + * + * Only covers the two gates that reject a whole value. A value reduced to `''` + * by ordinary stripping is not a rejection — that has always been possible and + * reporting it would hand every consumer spurious rejections from behaviour + * that predates the report. + */ +export type SanitizeRejection = 'maxLength' | 'valuePattern' + +/** + * Result of sanitizing a value, with the reason it was dropped if it was. + */ +export interface SanitizeValueResult { + /** The sanitized value, or `''` if a gate rejected it */ + value: string + /** Set only when a gate rejected the value outright */ + rejected?: SanitizeRejection +} + +/** + * Sanitize a value and report which gate, if any, rejected it + * + * Same rules and ordering as {@link sanitizeValue}; this variant additionally + * distinguishes "dropped by a gate" from "reduced to empty by stripping". + * + * @param value - The raw parameter value + * @param config - Sanitization configuration + * @returns The sanitized value plus an optional rejection reason + */ +export function sanitizeValueWithReport( + value: string, + config: SanitizeConfig, +): SanitizeValueResult { if (!config.enabled) { - return value + return { value } } let result = value @@ -40,11 +78,25 @@ export function sanitizeValue(value: string, config: SanitizeConfig): string { result = result.trim() + // Gate on the trimmed value. Testing before the trim would reject values whose + // only offence is surrounding whitespace that this function was about to remove. + if (config.valuePattern) { + config.valuePattern.lastIndex = 0 + if (!config.valuePattern.test(result)) { + return { value: '', rejected: 'valuePattern' } + } + } + if (result.length > config.maxLength) { + // '' rather than removing the key: hasUtmParameters already treats '' as + // absent, so it is the established sentinel for "no value". + if (config.onMaxLength === 'drop') { + return { value: '', rejected: 'maxLength' } + } result = result.slice(0, config.maxLength) } - return result + return { value: result } } /** diff --git a/src/index.ts b/src/index.ts index ba10954..4449aab 100644 --- a/src/index.ts +++ b/src/index.ts @@ -15,14 +15,26 @@ export { captureWithReferrer, type CaptureOptions, + // Capture reporting + captureUtmParametersWithReport, + type UtmRejection, + type UtmRejectionReason, + type CaptureReport, + // Sanitizer sanitizeValue, sanitizeParams, + sanitizeValueWithReport, + type SanitizeRejection, + type SanitizeValueResult, // PII Filter detectPii, filterValue, filterParams, + filterValueWithReport, + type PiiRejection, + type FilterValueResult, // Form field population populateFormFields, diff --git a/src/react/docs.md b/src/react/docs.md index 2c65621..bf1f0fd 100644 --- a/src/react/docs.md +++ b/src/react/docs.md @@ -30,7 +30,11 @@ useState initializer --> getStoredUtmParameters({storageType}) --> initial state useEffect (once, via ref guard) --> if captureOnMount && enabled: | v -capture() --> captureUtmParameters(window.location.href, {sanitize, piiFiltering}) --> if has params: +capture() --> captureUtmParameters(window.location.href, ) + | where the options are keyFormat, allowedParameters, lowercaseValues, + | sanitize, piiFiltering and the onCapture callback + | + | --> if has params: | storeUtmParameters({storageType, ttl}) | setUtmParameters() | else if has defaultParams: @@ -57,6 +61,8 @@ URL with UTM params - **Initialization guard**: `hasInitialized.current` is a ref (not state), so the guard works correctly across strict mode double-effects without triggering re-renders. - **`appendToUrl` exclusion logic**: The `excludeFromShares` filter converts camelCase keys to snake_case using inline regex (not the `toSnakeCase` utility), so it duplicates some conversion logic from `@/src/common/keys.ts`. - **Storage options forwarding**: The hook passes `storageType` and `ttl` from the resolved config to `storeUtmParameters`, `getStoredUtmParameters`, and `clearStoredUtmParameters`. The `clear` callback passes `storageType` so it clears the correct backend. +- **Capture options forwarding is the React path's only route to `CaptureOptions`**: consumers of the hook configure capture through `UtmConfig`, so any `CaptureOptions` field must be mapped explicitly in `useUtmTracking`'s call *and* exist on `UtmConfig`/`ResolvedUtmConfig`/`DEFAULT_CONFIG` in `@/src/config`. A field present on `CaptureOptions` but not forwarded here is simply unreachable from React. +- **The hook uses `captureUtmParameters`, not `captureUtmParametersWithReport`**: the hook's public surface exposes captured params only, so rejection reporting is not surfaced through React state. Consumers wanting to distinguish "no campaign" from "campaign rejected" call `captureUtmParametersWithReport` from `@/src/inbound` directly. - **SSR safety**: The `useState` initializer checks `typeof window === 'undefined'` and returns `null` for server rendering. The `capture` callback also checks before accessing `window.location`. Form and decorator components/hooks guard against `document` being undefined. Created and maintained by Nori. diff --git a/src/react/useUtmTracking.ts b/src/react/useUtmTracking.ts index c51a753..756c0aa 100644 --- a/src/react/useUtmTracking.ts +++ b/src/react/useUtmTracking.ts @@ -119,6 +119,7 @@ export function useUtmTracking(options: UseUtmTrackingOptions = {}): UseUtmTrack const params = captureUtmParameters(window.location.href, { keyFormat: config.keyFormat, allowedParameters: config.allowedParameters, + lowercaseValues: config.lowercaseValues, sanitize: config.sanitize, piiFiltering: config.piiFiltering, onCapture: config.onCapture, diff --git a/src/server/docs.md b/src/server/docs.md new file mode 100644 index 0000000..a617842 --- /dev/null +++ b/src/server/docs.md @@ -0,0 +1,66 @@ +# Noridoc: server + +Path: @/src/server + +### Overview + +- DOM-free UTM normalisation for server-side ingest endpoints, exposed as the third package entry point (`@jackmisner/utm-toolkit/server`). +- Exists because a public ingest endpoint cannot trust the client-side capture pass — anyone can POST to the endpoint directly — so the same folding rules have to run again on fully untrusted input. +- Provides `normalizeUtmParams(input: unknown, options)` for parsed request bodies and `normalizeUtmUrl(url, options)` for URL strings (a `Referer` header, a redirect target). + +### How it fits into the larger codebase + +- This folder deliberately reuses the *pure* value-level primitives from `@/src/inbound` — `sanitizeValueWithReport` from `@/src/inbound/sanitizer.ts` and `filterValueWithReport` from `@/src/inbound/pii-filter.ts` — so server and browser apply the same rules from the same code. It does not reuse the capture pipeline itself, which is URL-shaped and browser-defaulted. +- **Structural isolation invariant**: `@/src/server` must not import (at runtime) from `@/src/common/storage`, `@/src/inbound/form`, `@/src/inbound/attribution`, `@/src/outbound/decorator`, `@/src/outbound/appender`, `@/src/debug`, or `@/src/react`. That import restriction *is* the guarantee that this entry cannot touch the DOM or web storage. +- The invariant is **enforced, not documented**: `@/__tests__/server/isolation.test.ts` walks the transitive runtime import graph from `@/src/server/index.ts` and fails on any forbidden module or any reference to `window`/`document`/`sessionStorage`/`localStorage` outside comments. Without it, the next convenient re-export would silently remove the guarantee. +- Type-only imports are exempt from the walk because they are erased at build. `ServerNormalizeResult.rejected` reuses the `UtmRejection` type from `@/src/inbound/capture-report.ts` via `import type`, and `@/src/server/index.ts` re-exports it, so server and browser rejections speak one vocabulary at zero runtime cost. +- `STANDARD_UTM_PARAMETERS` and `DEFAULT_PII_FILTER_CONFIG` come from `@/src/config/defaults.ts`, which is pure data and therefore safe to reach. +- Wired into the build by a `server/index` entry in `@/tsup.config.ts` and a `./server` condition in `package.json`, mirroring the existing `./react` shape (dual ESM/CJS with declarations). + +### Core Implementation + +```text +untrusted body (unknown) URL string + | | + | normalizeUtmUrl -> URL parse (never throws) + | | searchParams collected last-wins + v v + normalizeUtmParams(input, options) + | + for each key in allowedParameters <-- iterates the ALLOWLIST, not the input + | + present & string? -> lowercase -> sanitizeValueWithReport -> filterValueWithReport + | + v + { params: TOTAL record, rejected: UtmRejection[] } +``` + +- **Totality is the defining contract.** The result is built by iterating `allowedParameters`, never the input, so every allowed key is always present in `params`. Absent, non-string, and rejected parameters all carry `absentValue` (default `''`). The motivating consumer writes these into a composite primary key, where `NULL` does not deduplicate — a nullable column fragments one campaign into as many rows as it has absent parameters. +- Iterating the allowlist gives totality and prototype-pollution safety in one move: a key the caller did not allow can never become an output key. Assignment goes through `Object.defineProperty` rather than `params[key] = value`, because a plain assignment for the key `'__proto__'` sets the prototype instead of creating an own property, silently breaking totality. +- **Never throws, for any input — and the hostile cases are the point.** The argument is an untrusted HTTP body; a throw here would be a 500 on somebody's first page load. Non-record input (including arrays) is treated as an empty source, an unparseable URL yields a total all-absent result, and non-string values are *rejected rather than coerced* — `String(['a','b'])` is `'a,b'`, a value nobody sent. Three guards make the promise literally true rather than approximately true: + - `readProperty()` wraps the property read, because `source[key]` runs a getter and `hasOwnProperty` runs a Proxy trap, either of which can throw. Any failure is treated as **absence**. `JSON.parse` cannot produce such an object, but this function is also reachable through custom content-type parsers, ORM entities and framework reactive proxies. + - `isPlainRecord()` wraps `Array.isArray`, which throws on a revoked Proxy. A value that cannot even be classified is not a usable parameter map. + - The sanitize + PII step for each key sits in a `try`/`catch`. A malformed caller-supplied regex or PII pattern degrades **that key** to `absentValue` plus a rejection, keeping the output total, rather than failing the request. This is the server-side counterpart of the whole-pipeline guard in `@/src/inbound/capture-report.ts`. +- **`allowedParameters` is validated as an array of strings, not assumed to be one.** Server config typically arrives from env vars or JSON, so a non-array falls back to `STANDARD_UTM_PARAMETERS` and non-string entries are filtered out. A bare string would otherwise be iterated character by character, producing one output column per letter — a silently wrong schema rather than an error. +- **Server defaults deliberately diverge from browser defaults.** Browser defaults are lenient because losing a campaign label client-side is cheap; a server keying a datastore needs determinism. + + | Option | Browser default | Server default | Reason | + | --- | --- | --- | --- | + | lowercase | off | on | `LinkedIn` and `linkedin` are one campaign, two rows | + | onMaxLength | `truncate` | `drop` | a truncated value is one nobody sent; long shared prefixes collide | + | PII filtering | off | on | the endpoint is public | + +- `piiFiltering.mode` is intentionally **not** configurable here: `Omit` in the options type, and `mode: 'reject'` reapplied after the caller's spread. Redact mode would persist `'[REDACTED]'` as a campaign nobody ran. +- Rejections carry key, reason, and (for PII) the pattern name only — never the rejected value, matching the same rule enforced in `@/src/inbound/capture-report.ts`. + +### Things to Know + +- **A value stripped to nothing is absence, not rejection.** Sanitisation reducing a value to `''` predates this module and produces no `rejected` entry; only the gates (`valuePattern`, `maxLength` under `drop`, PII, `notAString`) do. +- **`absentValue` is configurable for a reason.** A consumer that must distinguish "genuinely absent" from a real empty value can supply an unforgeable sentinel that no campaign value could collide with. +- **`allowedParameters` defaults to all of `STANDARD_UTM_PARAMETERS`, including `utm_id`.** A consumer keying fewer columns than the library produces gets a mystery extra row, so narrowing this is a real configuration step rather than an optimisation. +- **The justification for a separate entry is not a crash.** Importing the root entry in a DOM-free Node context does not throw — this was verified against the built artifact. The case for `/server` is the documented DOM-free surface, server-appropriate defaults, the totality contract, the enforced structural isolation, and a substantially smaller bundle. +- **Rejection keys here are trusted, unlike the browser side.** Because the loop iterates `allowedParameters`, every `rejected.key` is a caller-configured key. The browser report captures whatever `utm_`-prefixed key the URL carried, so its keys are attacker-controlled — see `@/src/inbound/docs.md` before reusing log handling across the two. +- `normalizeUtmUrl` is last-wins on duplicate query parameters, matching `URLSearchParams` iteration and the browser-side behaviour in `@/src/inbound/capture-report.ts`. +- Folding uses `toLowerCase()`, never `toLocaleLowerCase()`, and happens **before** every gate, so `valuePattern` and `piiFiltering.allowlistPattern` can be written without allowing uppercase. This mirrors the ordering in the capture pipeline. + +Created and maintained by Nori. diff --git a/src/server/index.ts b/src/server/index.ts new file mode 100644 index 0000000..d60930d --- /dev/null +++ b/src/server/index.ts @@ -0,0 +1,24 @@ +/** + * @jackmisner/utm-toolkit/server + * + * DOM-free UTM normalisation for server-side ingest endpoints. + * + * This entry point must not reach browser-coupled code. It deliberately does + * NOT import from `common/storage`, `inbound/form`, `outbound/decorator`, + * `debug/` or `react/` — that import restriction is the structural guarantee + * that this module cannot touch storage or the DOM, and it is enforced by a + * test in `__tests__/server/isolation.test.ts`. + * + * @packageDocumentation + */ + +export { + normalizeUtmParams, + normalizeUtmUrl, + type ServerNormalizeOptions, + type ServerNormalizeResult, +} from './normalize' + +export type { UtmRejection, UtmRejectionReason } from '../inbound/capture-report' + +export { STANDARD_UTM_PARAMETERS } from '../config/defaults' diff --git a/src/server/normalize.ts b/src/server/normalize.ts new file mode 100644 index 0000000..f86581f --- /dev/null +++ b/src/server/normalize.ts @@ -0,0 +1,317 @@ +/** + * Server-side UTM normalisation + * + * A DOM-free surface for applying the same folding rules server-side that the + * browser applies client-side. A public ingest endpoint cannot trust the + * client-side pass — anyone can POST to it directly — so the rules have to run + * again on input that is entirely untrusted. + * + * Three properties make this different from `captureUtmParameters`: + * + * 1. **Total output.** Every allowed key is always present. A consumer writing + * these into a composite primary key needs "absent" to be a value that + * groups; `undefined`/`NULL` does not deduplicate, so a nullable column + * fragments one campaign into as many rows as it has absent parameters. + * 2. **Never throws, for any input.** The argument is an untrusted HTTP body. + * A throw here is a 500 on somebody's first page load. + * 3. **Server-appropriate defaults.** Lowercasing on, over-length dropped, PII + * filtering on. The browser defaults are lenient because losing a campaign + * label client-side is cheap; a server keying a store needs determinism. + */ + +import type { PiiFilterConfig, SanitizeConfig } from '../types' +import type { UtmRejection } from '../inbound/capture-report' +import { DEFAULT_PII_FILTER_CONFIG, STANDARD_UTM_PARAMETERS } from '../config/defaults' +import { filterValueWithReport } from '../inbound/pii-filter' +import { sanitizeValueWithReport } from '../inbound/sanitizer' + +/** + * Options for server-side normalisation + * + * Every default here is stated explicitly because several differ from the + * browser defaults on purpose. + */ +export interface ServerNormalizeOptions { + /** + * Keys to produce. Default: all six `STANDARD_UTM_PARAMETERS`, including + * `utm_id` — the same set the browser defaults to. + * + * Narrow it if you key fewer columns. A consumer keying five columns against + * a library producing six gets a mystery extra row, so this is stated rather + * than left to be discovered. + */ + allowedParameters?: string[] + + /** Maximum value length. Default: 200. */ + maxLength?: number + + /** + * What to do with an over-length value. Default: `'drop'`. + * + * Differs from the browser default (`'truncate'`) deliberately: a truncated + * value is one nobody sent, and two campaigns sharing a long prefix collapse + * into a single row. + */ + onMaxLength?: 'truncate' | 'drop' + + /** + * Fold values to lowercase. Default: `true`. + * + * Differs from the browser default (`false`) deliberately: `LinkedIn` and + * `linkedin` are one campaign, and a store keyed on the raw value gets two rows. + */ + lowercase?: boolean + + /** Positive allowlist for values. Default: undefined (accept anything that survives the rest). */ + valuePattern?: RegExp + + /** + * PII filtering. Default: enabled. + * + * Differs from the browser default (disabled) deliberately, because the + * endpoint is public. `mode` is deliberately not configurable: `'[REDACTED]'` + * stored as a campaign value is a campaign nobody ran, which is worse than + * dropping it. Rejection is always used server-side. + */ + piiFiltering?: Partial> + + /** + * What an absent or rejected parameter becomes. Default: `''`. + * + * Configurable because a consumer may want an unforgeable sentinel that no + * real campaign value could collide with. + */ + absentValue?: string +} + +/** + * Result of server-side normalisation + */ +export interface ServerNormalizeResult { + /** + * TOTAL: every key in `allowedParameters` is present. Absent and rejected + * parameters carry `absentValue`. + */ + params: Record + + /** Every parameter that was rejected, with the reason. Never carries the value. */ + rejected: UtmRejection[] +} + +/** + * True for objects that can be safely iterated as a string-keyed record. + * + * Arrays are excluded: an array body is not a parameter map, and treating it as + * one would silently read numeric indices as keys. + */ +function isPlainRecord(value: unknown): value is Record { + if (typeof value !== 'object' || value === null) { + return false + } + try { + return !Array.isArray(value) + } catch { + // Array.isArray throws on a revoked Proxy. Anything we cannot even classify + // is not a usable parameter map. + return false + } +} + +/** + * Read one property from an untrusted object, treating any failure as absence. + * + * `source[key]` runs a getter, and `hasOwnProperty` runs a Proxy trap; either + * can throw on a hostile or exotic object. `JSON.parse` cannot produce those, + * but this function is also reachable through custom content-type parsers, ORM + * entities and framework reactive proxies, and the never-throws guarantee is + * this module's headline promise. + */ +function readProperty(source: Record, key: string): unknown { + try { + return Object.prototype.hasOwnProperty.call(source, key) ? source[key] : undefined + } catch { + return undefined + } +} + +/** + * Assign a key without tripping over inherited setters. + * + * A plain `params[key] = value` for the key `'__proto__'` sets the prototype + * instead of creating an own property, which would silently break totality. + * `defineProperty` always creates the own property. + */ +function setParam(params: Record, key: string, value: string): void { + Object.defineProperty(params, key, { + value, + enumerable: true, + writable: true, + configurable: true, + }) +} + +/** + * Normalize UTM parameters from an untrusted request body + * + * Accepts input of any shape and never throws — including objects with throwing + * getters and hostile or revoked Proxies, which are treated as absent. Values + * that are not strings are rejected rather than coerced: `String(['a','b'])` is + * `'a,b'`, a value nobody sent. + * + * @param input - An untrusted request body, of any shape + * @param options - Normalisation options; see {@link ServerNormalizeOptions} + * @returns Total params keyed by `allowedParameters`, plus any rejections + * + * @example + * ```typescript + * const { params, rejected } = normalizeUtmParams(request.body) + * await db.insert(params) // every column present, safe to key on + * if (rejected.length > 0) metrics.increment('utm.rejected', rejected.length) + * ``` + */ +export function normalizeUtmParams( + input: unknown, + options: ServerNormalizeOptions = {}, +): ServerNormalizeResult { + const { + maxLength = 200, + onMaxLength = 'drop', + lowercase = true, + valuePattern, + piiFiltering, + absentValue = '', + } = options + + const sanitizeConfig: SanitizeConfig = { + enabled: true, + stripHtml: true, + stripControlChars: true, + maxLength, + onMaxLength, + valuePattern, + } + + const piiConfig: PiiFilterConfig = { + ...DEFAULT_PII_FILTER_CONFIG, + enabled: true, + ...piiFiltering, + patterns: piiFiltering?.patterns ?? [...DEFAULT_PII_FILTER_CONFIG.patterns], + // After the spread: redact mode is never used server-side. + mode: 'reject', + } + + // Server config often arrives from env or JSON, so allowedParameters cannot be + // trusted to be an array of strings. A non-array falls back to the standard + // keys; a string would otherwise be iterated character by character, producing + // one column per letter. + const allowedParameters = + Array.isArray(options.allowedParameters) && options.allowedParameters.length > 0 + ? options.allowedParameters.filter((key): key is string => typeof key === 'string') + : [...STANDARD_UTM_PARAMETERS] + + const rejected: UtmRejection[] = [] + const params: Record = {} + const source = isPlainRecord(input) ? input : {} + + // Iterate the ALLOWED keys, not the input. This gives totality and sidesteps + // prototype pollution in one move: a key the caller did not allow can never + // become an output key, whatever the body contains. + for (const key of allowedParameters) { + const raw = readProperty(source, key) + + if (raw === undefined || raw === null) { + setParam(params, key, absentValue) + continue + } + + if (typeof raw !== 'string') { + setParam(params, key, absentValue) + rejected.push({ key, reason: 'notAString' }) + continue + } + + // Fold before every gate below, so patterns can assume lowercase input. + // toLowerCase(), never toLocaleLowerCase(): folding must not depend on locale. + const folded = lowercase ? raw.toLowerCase() : raw + + let sanitized: ReturnType + let filtered: ReturnType + try { + sanitized = sanitizeValueWithReport(folded, sanitizeConfig) + if (sanitized.rejected) { + setParam(params, key, absentValue) + rejected.push({ key, reason: sanitized.rejected }) + continue + } + filtered = filterValueWithReport(key, sanitized.value, piiConfig) + } catch { + // A malformed caller-supplied regex or PII pattern is a config bug. Degrade + // this key to absent rather than 500 the request, and keep the output total. + setParam(params, key, absentValue) + rejected.push({ key, reason: 'notAString' }) + continue + } + + if (filtered.rejected) { + setParam(params, key, absentValue) + rejected.push({ + key, + reason: filtered.rejected.reason, + ...(filtered.rejected.patternName === undefined + ? {} + : { patternName: filtered.rejected.patternName }), + }) + continue + } + + // A value stripped to nothing is absent, not rejected — that outcome + // predates this module and is not a new rejection reason. + setParam( + params, + key, + filtered.value === undefined || filtered.value === '' ? absentValue : filtered.value, + ) + } + + return { params, rejected } +} + +/** + * Normalize UTM parameters from a URL string + * + * For servers that have a URL — a `Referer` header, a redirect target — rather + * than a parsed body. Never throws: a malformed URL yields a total result with + * every parameter absent. + * + * Duplicate query parameters are last-wins, matching `URLSearchParams` + * iteration and the browser-side behaviour. + * + * @param url - The URL to read parameters from + * @param options - Normalisation options; see {@link ServerNormalizeOptions} + * @returns Total params keyed by `allowedParameters`, plus any rejections + */ +export function normalizeUtmUrl( + url: string, + options: ServerNormalizeOptions = {}, +): ServerNormalizeResult { + if (typeof url !== 'string' || url === '') { + return normalizeUtmParams({}, options) + } + + let parsed: URL + try { + parsed = new URL(url) + } catch { + // A URL we cannot parse carries no parameters. Reported as absence rather + // than rejection: nothing was filtered, there was nothing to filter. + return normalizeUtmParams({}, options) + } + + const collected: Record = {} + for (const [key, value] of parsed.searchParams.entries()) { + // Last-wins on duplicates, matching URLSearchParams iteration order. + setParam(collected, key, value) + } + + return normalizeUtmParams(collected, options) +} diff --git a/src/types/docs.md b/src/types/docs.md index 68511a7..8a015e8 100644 --- a/src/types/docs.md +++ b/src/types/docs.md @@ -24,12 +24,16 @@ Path: @/src/types - `ResolvedUtmConfig` mirrors `UtmConfig` but with all fields required (except `ttl` and event callbacks, which remain optional) -- it represents the result of merging user-provided partial config with defaults. Includes `storageType` (defaulting to `'session'`), optional `ttl` (milliseconds, only meaningful for localStorage), `attribution` config, and lifecycle callbacks (`onCapture`, `onStore`, `onClear`, `onAppend`, `onExpire`). - Event callback signatures on `UtmConfig`/`ResolvedUtmConfig`: `onCapture(params)`, `onStore(params, meta)` where meta includes `storageType` and optional `touch`, `onClear()`, `onAppend(url, params)`, `onExpire(storageKey)`. - `SanitizeConfig` and `PiiFilterConfig` follow the partial-in/resolved-out pattern: `Partial<>` on `UtmConfig` (user input), required on `ResolvedUtmConfig` (resolved output). +- `SanitizeConfig` carries two distinct kinds of value rule. `customPattern` is **subtractive** (matches are stripped out of the value); `valuePattern` is a **positive allowlist gate** (the value is kept whole or replaced with `''`). `onMaxLength` (`'truncate' | 'drop'`) selects what happens to an over-length value — `'truncate'` is the default and preserves the original behaviour, `'drop'` is what a consumer keying a datastore wants, since a truncated value is one nobody sent. +- `UtmConfig.lowercaseValues` (mirrored required on `ResolvedUtmConfig`) is a **flat** field rather than part of `SanitizeConfig`, deliberately mirroring `BuildUtmUrlOptions.lowercaseValues` on the outbound side. It is forwarded to `CaptureOptions.lowercaseValues` by the React path. ### Things to Know - `UtmParameters` is a union, not an intersection. Code that receives it must handle either format, typically by detecting the format or converting via `@/src/common/keys.ts`. - `SharePlatform` is `'linkedin' | 'twitter' | 'facebook' | 'copy' | string` -- the named platforms are documentation aids, but any string is accepted. - `DiagnosticInfo` is only used by `@/src/debug` and is meant for development-time inspection, not production data flow. -- New features use a nested config object pattern (e.g., `sanitize: SanitizeConfig`, `attribution: AttributionConfig`) rather than adding flat fields to `UtmConfig`. The exceptions are `storageType`, `ttl`, and event callbacks, which exist as flat fields. +- New features use a nested config object pattern (e.g., `sanitize: SanitizeConfig`, `attribution: AttributionConfig`) rather than adding flat fields to `UtmConfig`. The exceptions are `storageType`, `ttl`, event callbacks, and `lowercaseValues`, which exist as flat fields. +- Rejection types (`UtmRejection`, `UtmRejectionReason`, `SanitizeRejection`, `PiiRejection`) are **not** defined here — they live next to the code that produces them in `@/src/inbound`, and `@/src/server` re-exports `UtmRejection` type-only so both entry points describe rejections with one vocabulary. +- Server-side normalisation options (`ServerNormalizeOptions`) are also defined locally in `@/src/server`, not here, because they are a different contract from `UtmConfig` — they configure a stateless pure function, not a stateful browser session. Created and maintained by Nori. diff --git a/src/types/index.ts b/src/types/index.ts index a079529..d307eae 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -144,8 +144,33 @@ export interface SanitizeConfig { /** Maximum allowed length for parameter values (default: 200) */ maxLength: number - /** Optional additional regex pattern to strip from values */ + /** + * What to do with a value longer than `maxLength` (default: 'truncate'). + * + * `'truncate'` cuts the value to `maxLength`. `'drop'` replaces it with `''`. + * Prefer `'drop'` when values key a datastore: a truncated value is one nobody + * sent, and two campaigns sharing a long prefix collapse into one. + */ + onMaxLength?: 'truncate' | 'drop' + + /** + * Optional additional regex pattern to strip from values. + * + * Subtractive: every match is removed from the value. Contrast `valuePattern`, + * which accepts or drops the value as a whole. + */ customPattern?: RegExp + + /** + * Optional positive allowlist for values. A value not matching becomes `''`. + * + * A gate, not a filter: the value is kept intact or dropped entirely. Tested + * against the trimmed value, so incidental whitespace does not cause a + * rejection. Contrast `customPattern` (subtractive) and + * `PiiFilterConfig.allowlistPattern` (the same gate, but scoped to PII + * decisions and able to produce `'[REDACTED]'` in redact mode). + */ + valuePattern?: RegExp } /** @@ -220,6 +245,13 @@ export interface UtmConfig { */ allowedParameters?: string[] + /** + * Lowercase all captured values (default: false) + * + * Forwarded to the capture pipeline; see `CaptureOptions.lowercaseValues`. + */ + lowercaseValues?: boolean + /** Default UTM parameters when none are captured */ defaultParams?: UtmParameters @@ -265,6 +297,7 @@ export interface ResolvedUtmConfig { captureOnMount: boolean appendToShares: boolean allowedParameters: string[] + lowercaseValues: boolean defaultParams: UtmParameters shareContextParams: ShareContextParams excludeFromShares: string[] diff --git a/tsup.config.ts b/tsup.config.ts index 9d9cdb5..f076411 100644 --- a/tsup.config.ts +++ b/tsup.config.ts @@ -4,6 +4,7 @@ export default defineConfig({ entry: { index: 'src/index.ts', 'react/index': 'src/react/index.ts', + 'server/index': 'src/server/index.ts', }, format: ['cjs', 'esm'], dts: true,