Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 3 additions & 9 deletions .github/workflows/test-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -300,14 +300,6 @@ jobs:
key: ${{ github.repository }}-playwright-browsers-${{ github.event_name }}${{ github.event.pull_request.head.repo.fork && '-fork' || '' }}
path: ~/.cache/ms-playwright

- name: Restore E2E Next.js build cache
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
with:
path: ./apps/sim/.next/cache
key: ${{ runner.os }}-nextjs-e2e-hosted-billing-${{ hashFiles('bun.lock') }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-nextjs-e2e-hosted-billing-${{ hashFiles('bun.lock') }}-

- name: Install dependencies
run: bun install --frozen-lockfile

Expand All @@ -327,7 +319,7 @@ jobs:
run: bun run test:e2e

- name: Upload E2E diagnostics
if: failure() || cancelled()
if: failure()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: settings-e2e-${{ github.run_id }}
Expand All @@ -336,6 +328,8 @@ jobs:
apps/sim/test-results/
apps/sim/e2e/.runs/
!apps/sim/e2e/.runs/**/auth/**
!apps/sim/e2e/.runs/**/private/**
!apps/sim/e2e/.runs/**/homes/**
if-no-files-found: ignore
include-hidden-files: true
retention-days: 7
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ package-lock.json
/apps/**/test-results/
/apps/**/e2e/.runs/
/apps/**/e2e/.auth/
/apps/**/e2e/.cache/

# next.js
/.next/
Expand Down
99 changes: 92 additions & 7 deletions apps/sim/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,22 @@ E2E_PG_ADMIN_URL=postgresql://postgres:postgres@127.0.0.1:5432/postgres \
```

The runner creates a unique `sim_e2e_<runId>` database, migrates it, starts the
Stripe fake and realtime, builds and starts Next.js, runs Playwright on Node 22,
then stops services and drops only that guarded database.
Stripe fake and realtime, builds and starts Next.js, seeds the validated persona
world through production APIs plus narrow trusted arrangements, captures each
persona through the real login UI, runs Playwright on Node 22, then stops
services and drops only that guarded database.
Server telemetry currently shares the strict loopback Stripe-fake process; the
generic modular fake-service refactor remains scoped to the roadmap's
`e2e/06b-enterprise-integrations` phase.
An exclusive checkout-level orchestrator lock prevents concurrent runs from
racing on `.next`, the shared build cache, or the fixed app/realtime ports.

On interruption, the runner launches a detached cleanup supervisor before
exiting. It terminates managed process groups, force-drops the guarded database,
and removes temporary auth/cloud-config directories even if another Ctrl-C
terminates the foreground package runner.
Cleanup failures retain the lock and require the reported resources to be
inspected and cleaned before manually removing `e2e/.cache/orchestrator.lock`.

Pass Playwright arguments after `--`:

Expand All @@ -65,19 +74,61 @@ bun run test:e2e -- --project=hosted-billing-chromium-navigation
bun run test:e2e -- --grep "unauthenticated"
```

Projects form a dependency chain to keep shared boundaries serialized. Selecting
the personas project therefore also runs navigation and workflows by default,
and an upstream failure skips its dependents. For focused local iteration,
`--no-deps` is explicitly supported:

```bash
bun run test:e2e -- --project=hosted-billing-chromium-personas --no-deps
```

`--no-deps` requires exactly one explicit canonical project. It skips only
Playwright project dependencies; the guarded one-shot stack still performs full
seed and auth setup. Do not use it for full verification; the runner rejects it in CI.

For a local follow-up run, reuse only a verified build while still creating a
new database, Stripe fake, app, realtime process, and browser run:

```bash
bun run test:e2e -- --reuse-build --project=hosted-billing-chromium-navigation
```

The cache lives under ignored `e2e/.cache/builds/`. A hit requires matching
source contents (including uncommitted/untracked files), build/public profile,
Node/Bun/Next versions, platform, `BUILD_ID`, and the cached artifact checksum.
Any mismatch performs and caches a fresh local build; only the most recent cache
entry is retained because this application's production artifact is several
gigabytes. CI rejects `--reuse-build` and does not copy or hash a disposable
cache artifact. Plain local runs also skip cache identity computation,
multi-gigabyte copying, and cache population; only an explicit `--reuse-build`
request pays those costs. Cache restore also removes abandoned temporary stores
from interrupted writes before accepting a hit. `--skip-build` remains unsupported.

Keep-stack/rerun supervision is intentionally unavailable. The initial safety
experiment requires descriptor ownership, mutation observation, state snapshots,
and teardown to ship as one unit; until all of those are proven, each invocation
uses the normal one-shot lifecycle.

Do not invoke `playwright test` directly. Raw Playwright bypasses environment,
database, process, sharding, and teardown guards; the config rejects runs that
were not launched by the orchestrator. Report and trace viewer commands remain
safe because they do not execute tests.

Sharding is supported only for the navigation project. The runner rejects
`--shard` for `hosted-billing-chromium-workflows`.
`--shard` for workflows, persona contracts, and the dedicated two-worker
cross-world isolation project. Project dependencies serialize navigation,
workflows, and persona contracts before the isolation project opens its
two-worker pool.

## Diagnostics

- HTML report: `playwright-report/`
- Traces and screenshots: `test-results/`
- App, realtime, migration, and fake logs: `e2e/.runs/<runId>/logs/`
- App, realtime, migration, seed, auth-capture, and fake logs:
`e2e/.runs/<runId>/logs/`
- Non-secret persona manifest and auth-capture failure screenshots:
`e2e/.runs/<runId>/`

Open the report:

Expand All @@ -91,9 +142,43 @@ Open a trace:
node ../../node_modules/@playwright/test/cli.js show-trace test-results/<test>/trace.zip
```

The runner starts every child process from a fresh environment. It allowlists
only deterministic E2E values and shadows keys found in local `.env*` files, so
developer credentials are not used as test state or written to reports.
The runner starts every child process from a fresh, purpose-specific
environment. Next build receives deterministic sentinels instead of the run
database/fake endpoint; app, realtime, migrations, seeding, auth capture, and
Playwright each receive only their required values. Playwright receives the
non-secret manifest and storage-state directory, never passwords, the admin API
key, or the database URL. Next build/start shadow
keys found in local `.env*` files, while children that cannot load those files
omit denied keys entirely. Developer credentials are not used as test state or
written to reports. Named persona credentials and a separate all-synthetic-user
canary list live outside every child `HOME` in a private run directory. Auth
capture receives only the persona file, which is deleted immediately after
capture; storage states are mode `0600` and excluded from CI artifacts. Captured
passwords are loaded into orchestrator memory, then both secret files are
deleted before Playwright starts. After managed processes stop and logs flush,
the in-memory canary scans the manifest, logs, report files, and trace archives
and fails if a synthetic password, invitation token, or runtime secret escaped
the excluded private directories. Cancelled CI runs do not upload unscanned
diagnostics, and an unreadable canary or incomplete archive scan causes all
potentially unscanned diagnostic roots to be scrubbed. Storage-state session cookies are intentionally not canaried
because authenticated Playwright traces contain them by design; they are
synthetic and invalid once the run database is dropped.
Fresh-session recapture is deliberately deferred. Future membership-mutation
coverage must explicitly restore a private credential handoff and re-review its
access boundary rather than assuming credentials persist through Playwright.

E2E builds verify the pinned Bun executable plus reviewed sandbox-bundle
source, direct dependency, and output fingerprints and never regenerate
committed `.cjs` files.
`bun run build:sandbox-bundles:integrity` is the explicit maintenance command
that regenerates bundles and their reviewed integrity manifest together.
Unrelated monorepo lockfile changes do not invalidate the bundle fingerprint;
the committed output hashes still detect any transitive change that alters a
bundle.

Reset/reseed cleanup remains deferred with keep-stack supervision. Ordinary
runs own a unique guarded database and remove it wholesale rather than carrying
untested row-level deletion code.

Provider log scans are diagnostic tripwires, not proof of zero egress. The
primary boundaries are the default-deny child environment, provider disabling,
Expand Down
8 changes: 8 additions & 0 deletions apps/sim/e2e/fakes/stripe/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ export const STRIPE_FAKE_ENDPOINTS = {
health: '/health',
requestLog: '/__control/requests',
reset: '/__control/reset',
// Co-locate the only other server-side external boundary so the E2E stack needs one loopback fake.
telemetry: '/v1/traces',
} as const

const DEFAULT_MAX_BODY_BYTES = 64 * 1024
Expand Down Expand Up @@ -63,6 +65,7 @@ function isExpectedStripeRequest(method: string, path: string): boolean {
((method === 'GET' || method === 'POST') && path === '/v1/customers/search') ||
(method === 'GET' && path === '/v1/customers') ||
(method === 'POST' && path === '/v1/customers') ||
(method === 'POST' && path === STRIPE_FAKE_ENDPOINTS.telemetry) ||
(method === 'GET' && /^\/v1\/customers\/cus_e2e_[a-f0-9]+$/.test(path))
)
}
Expand Down Expand Up @@ -326,6 +329,11 @@ export function createStripeFakeServer(options: StripeFakeServerOptions): Stripe
unexpected: !expected,
})

if (method === 'POST' && url.pathname === STRIPE_FAKE_ENDPOINTS.telemetry) {
sendJson(response, 200, { partialSuccess: {} }, requestId)
return
}

if (!secureEqual(request.headers.authorization, expectedAuthorization)) {
sendStripeError(
response,
Expand Down
Loading
Loading