Skip to content

test: stop unit tests from sending events to production PostHog - #1373

Draft
posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/testdebug-stop-unit-test-runs-from-95fc99
Draft

posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/testdebug-stop-unit-test-runs-from-95fc99

Conversation

@posthog

@posthog posthog Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • Local unit test runs send events and exceptions to production PostHog. The team then triages test failures as if they were real user bugs.
  • Vitest does not mock the analytics singleton, so src/shared/utils/analytics.ts builds a real posthog-node client with the production write key.
  • Example: the debug.test.ts "blocker" test fails a log write on purpose. reportLogFailureOnce then calls analytics.captureException, and the real client files an EEXIST issue.
  • Draft #877 only gates exception autocapture. It does not stop explicit captureException calls.

Origin

  • Error tracking: issue
  • First signal: 2026-09-27
  • Inbox report: open
  • Likely cause: 2595fc0
  • Task started by: auto-start, after the report was rated P3 and ready to fix

Changes

  • New vitest.setup.ts automocks posthog-node. vitest.config.ts loads it for every test project. Explicit captures and autocaptured exceptions both go through that client, so both paths stop.
  • A per-file vi.mock('posthog-node') (for example in analytics.test.ts) still works as before.
  • Fix in pi/__tests__/tools.test.ts: the test created a rejected promise and did not handle it. The real client's unhandledRejection listener hid this from vitest. With the mock, vitest reports it, so the test now awaits the promise that it creates.

Note

Any future test that leaves an unhandled rejection now fails the run. Before this change, the real client hid it and sent it to error tracking.

Test plan

  • pnpm test: 204 files, 3434 tests pass, no unhandled errors.
  • pnpm typecheck, and eslint + prettier on the changed files.

LLM context

  • Agent-authored from a PostHog inbox report.
  • I chose a global posthog-node mock over a mock of ./analytics in debug.test.ts only, because the leak comes from many test files.

Created with PostHog Desktop from this inbox report.

🤖 Generated with Claude Code

Mock posthog-node in a global vitest setup file, so the analytics singleton and other clients never reach production error tracking from a test run.

The real client's exception autocapture hid an unhandled rejection in the pi check_env_keys test. The test now handles the promise it creates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Generated-By: PostHog Desktop
Task-Id: bcd9fae6-64f8-45bc-9515-41ba7a7bf497
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci feature-flags
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci feature-flags/django
  • /wizard-ci feature-flags/next-js
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant