Skip to content

Fix workflow throughput admission#5791

Draft
akshay326 wants to merge 5 commits into
simstudioai:mainfrom
akshay326:fix/workflow-throughput-admission
Draft

Fix workflow throughput admission#5791
akshay326 wants to merge 5 commits into
simstudioai:mainfrom
akshay326:fix/workflow-throughput-admission

Conversation

@akshay326

@akshay326 akshay326 commented Jul 21, 2026

Copy link
Copy Markdown

Summary

This PR fixes an overload-control bug in the database-backed async workflow path: the admission permit was released after 202 Accepted while execution continued in the web process. The permit now remains held until inline execution finishes, so overload produces explicit backpressure instead of unbounded in-process work.

It also applies admission control to session-backed async requests, adds coverage, and documents a production-grounded method for measuring workflow throughput and locating bottlenecks.

Scope

  • This is a stability/backpressure fix, not a claim of higher workflow completion throughput.
  • Local database-backend execution and production Trigger.dev execution have different bottlenecks.
  • Throughput is measured from terminal execution rows grouped by workflow_execution_logs.ended_at; 202/sec is admission throughput only.
  • The committed one-pager preserves before/after sequence diagrams and adds a bottleneck register with falsifiable evidence and concise fixes.

Testing

  • bun run check:api-validation
  • bun vitest run app/api/workflows/[id]/execute/route.async.test.ts lib/core/async-jobs/backends/database.test.ts — 58 tests passed
  • bun run type-check
  • Targeted Biome checks and IDE lints
  • git diff --check

Review focus

  • Verify admission tickets are retained only for in-process execution and released on every terminal/error path.
  • Treat workflow.execution.count as secondary until production export and terminal-path coverage are validated.
  • Validate deployment values separately: code defaults to 10, while Helm/docker-compose currently override the gate to 500.

Follow-up experiments

The next evidence-gathering step is a fixed-workload local/staging matrix: work-duration sweep, database backend versus Trigger.dev, DB pool pressure, tenant fairness, sync/SSE web pressure, executor memory/fan-out, and Redis/storage/external dependency degradation. The goal is to prove the binding resource before proposing a fix.

Checklist

  • Code follows project style guidelines
  • Self-reviewed changes
  • Tests added/updated and passing
  • No new warnings introduced
  • Production/staging throughput matrix run (requires environment credentials and representative workflow fixture)
  • CLA confirmation

akshay326 and others added 4 commits July 21, 2026 01:00
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@vercel

vercel Bot commented Jul 21, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the Sim Team on Vercel.

A member of the Team first needs to authorize it.

Co-authored-by: Cursor <cursoragent@cursor.com>
@akshay326 akshay326 changed the title Fix/workflow throughput admission Fix workflow throughput admission Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant