Feature: active archive table#3
Open
tinchoz49 wants to merge 56 commits into
Open
Conversation
…y routing for getJobs/getJobById/retry
- Split jobs/steps into active and archive tables - Jobs move from active to archive on complete/fail/cancel - Single spans table without FK constraints - SQL-native archive operations with USING joins - Prune with orphan span cleanup - Time travel restores archived jobs to active - Dashboard archive page and filter toggles - Add archive-specific tests
The test was importing from 'duron' package which failed in CI because workspace resolution doesn't work during test execution. Moving it to the duron package allows using relative imports.
- Exclude expired active jobs from verify_concurrency CTE count - _recoverJobs now archives expired jobs as failed (not reset to created) - Non-expired jobs from unresponsive owners still reset to created - Add recoverJobsInterval option (default 60s) for periodic recovery - Recovery runs before fetch when interval has elapsed - Add test: expired jobs no longer block new job selection - Update docs: client-api.mdx and multi-worker.mdx
…ecovery test - Change processTimeout default from 5000ms to 500ms for faster orphan detection - Update docs (client-api, examples, multi-worker) with new default - Rename and improve test: 'should recover expired orphan jobs from dead processes' - Add multiProcessMode + recoverJobsInterval + fake client_id to simulate crash - Add detailed comment explaining the multi-process orphan recovery scenario - Keep fetch() recovery only in multiProcessMode to avoid single-process overhead
- Add status guard to _deleteJobs to always exclude active jobs - Fix sql.json() → JSON.stringify() for drizzle-orm 1.0.0-rc.4 compatibility - Add comprehensive tests for bulk delete safety Fixes #3 from FINDINGS.md
- Add FOR UPDATE to locked_source CTE to lock the source job row - Prevents concurrent retry calls from creating duplicate jobs Fixes #4 from FINDINGS.md
- Register wait in #pendingJobWaits BEFORE checking job status - If job is already terminal, remove wait and resolve immediately - Prevents NOTIFY from being missed during the status check await gap Fixes #5 from FINDINGS.md
- Remove redundant assignment that overwrote parsed input with raw input - Now input is properly validated by Zod schema before reaching action handlers - Invalid input will throw validation error instead of silently passing through Fixes #6 from FINDINGS.md
- End span and delete from #stepSpans before early return - Prevents memory leak and orphaned 'started but never ended' spans in telemetry Fixes #7 from FINDINGS.md
- Store listener references in fields for proper cleanup - Remove listeners in stop() before database.stop() to prevent use-after-close - Add guard to #setupPushListener to prevent duplicate registration - Reset listener setup flags on stop() for clean restart Fixes #8 from FINDINGS.md
- Add check for #starting at top of stop() - Prevents tearing down incomplete initialization when stop() is called during start() Fixes #9 from FINDINGS.md
- Add orphan span cleanup query after _deleteJobs - Removes spans whose job no longer exists in active or archive tables - Prevents orphan span accumulation after bulk deletion Fixes #10 from FINDINGS.md
- Add orphan span cleanup query at end of _truncateArchive - Removes spans whose job no longer exists in active or archive tables - Prevents permanent orphan spans after archive truncation Fixes #11 from FINDINGS.md
…utdown - Add optional signal parameter to RecoverJobsOptions - Check signal.aborted inside the ping loop to allow early termination - Prevents slow shutdown when stop() is called during recovery Fixes #12 from FINDINGS.md
- Replace sql.raw(timeoutMs.toString()) with parameterized ( * interval '1 millisecond') - Prevents potential SQL injection through timeout values - Improves code auditability Fixes #13 from FINDINGS.md
- Replace explicit field-by-field copy with { ...job } and { ...s }
- Prevents new columns from being silently dropped during time-travel restore
- More resilient to schema changes
Fixes #14 from FINDINGS.md
- These dependencies were not imported anywhere in the codebase - Reduces install size and attack surface Fixes #16 from FINDINGS.md
- Update @default from 5000 to 500 to match schema default - Prevents user confusion about actual timeout behavior Fixes #19 from FINDINGS.md
- Spread enumerable properties from Error instances into result - Preserves custom properties like 'code' from Node.js errors (ENOENT, ECONNREFUSED) - Improves debugging context in logs and persisted errors Fixes #20 from FINDINGS.md
- Remove 1801-line backup file from repository - Add *.backup to .gitignore to prevent future commits Fixes #21 from FINDINGS.md
- Document DATABASE_URL, JWT_SECRET, and OPENAI_API_KEY - Helps new contributors and users understand required configuration Fixes #31 from FINDINGS.md
- Add repository URL pointing to GitHub - Add bugs URL for issue tracking - Add homepage URL for documentation - Improves npm discoverability and user experience Fixes #32 from FINDINGS.md
- Make PostgresBaseAdapter class abstract to prevent direct instantiation - Make _initDb method abstract to enforce implementation in subclasses - Prevents runtime errors from missing implementations Fixes #33 from FINDINGS.md
Remove indexes that are never used in WHERE clauses or are redundant: jobs_active (15 → 8): - Remove description, started_at (dashboard-only) - Remove concurrency_limit, concurrency_step_limit (NEVER queried) - Remove action_status (redundant with single-column) - Remove input_fts, output_fts (output is NULL for active jobs) job_steps_active (8 → 5): - Remove job_status, job_name (redundant composites) - Remove output_fts (expensive GIN, rarely used) spans (11 → 7): - Remove kind, status_code (rarely queried) - Remove attributes, events (expensive GIN, dashboard-only) Total: 34 → 20 indexes (~40% reduction in write overhead) Fixes #34 from FINDINGS.md
- Add onConflictDoNothing() to job and step restore inserts - Prevents unhandled PK violations on concurrent time-travel of same job - Gracefully handles race condition as no-op Fixes #15 from FINDINGS.md
- Remove _notify calls from completeJobStep, failJobStep, delayJobStep, cancelJobStep - Remove step-status-changed and step-delayed event type definitions - Eliminates wasted DB work from unhandled NOTIFYs Fixes #17 from FINDINGS.md
- Add JSDoc explaining that bulk delete only affects active (non-running) jobs - Archive jobs are not affected by bulk delete - Document that active jobs are always excluded for safety Fixes #18 from FINDINGS.md
- Move job existence check before step update - Prevents wasted DB work when cancelling non-existent jobs Fixes #26 from FINDINGS.md
- Replace djb2 hash with fixed key (42) - Simpler and avoids potential lock contention from hash collisions Fixes #24 from FINDINGS.md
- Rename column in schema.ts for both active and archive step tables - Update all raw SQL references in base.ts - TypeScript field was already 'parallel', now DB column matches Fixes #25 from FINDINGS.md
- Mark Finding #30 as already fixed in FINDINGS.md - Regenerate migrations for branch→parallel rename and index removals - Add improvement plans directory
- Remove double-check of concurrency limit after acquiring lock - Use count from eligible_groups instead of re-counting - Eliminates correlated subquery per candidate job Fixes #23 from FINDINGS.md
…rips - Query both active and archive tables in single query - Add NULL as jobFinishedAt placeholder for active table - Return camelCase column names for Zod validation Fixes #27 from FINDINGS.md
…y leak - Store abort listener reference - Remove listener when release() is called - Prevents listener from staying attached after resolution Fixes #28 from FINDINGS.md
- Add .catch() to getJobStatus/getJobById chain in waitForJob - Settle wait as null on error instead of hanging forever - Clean up pending wait entry on failure - Remove unused registeredResolve variable Found in post-fix re-audit
- Use explicit column list in outer SELECT instead of SELECT * - output stays in subquery only for the search filter - Postgres no longer transfers output jsonb to the client - Drop unnecessary NULL jobFinishedAt placeholder Found in post-fix re-audit
…rter support - Remove telemetry.local option and LocalSpanExporter - Remove spans table from schema - Remove /jobs/:id/spans and /steps/:id/spans API endpoints - Remove dashboard SpansPanel, SpansContext, useJobSpans/useStepSpans - Remove spansEnabled getter, getJobSpans(), getStepSpans(), flushTelemetry() - Move @opentelemetry/sdk-trace-base and sdk-trace-node to optional peer deps - Add traceExporter array support (SpanExporter | SpanExporter[]) - Add batchDelayMs option for configuring export interval - Add otel-integration.test.ts with in-memory exporter tests - Add ADR 0001 documenting the decision - Add CONTEXT.md with domain glossary
- Update telemetry.mdx to focus on external exporters - Remove spans table documentation from adapters.mdx - Remove getSpans, spansEnabled, flushTelemetry from client-api.mdx - Remove spansEnabled and spans endpoints from server-api.mdx - Remove spans visualization from dashboard.mdx
- Fix trailing comma in packages/duron/package.json exports (root cause of TS1295 errors) - Remove dead spans-related code from dashboard (SpansProvider, spans-panel, spans-context refs) - Remove unused imports/variables left from otelia removal (adapter.ts, schema.ts, base.ts, test files) - Run oxfmt on affected files
client.ts statically imports @opentelemetry/resources, @opentelemetry/semantic-conventions, sdk-trace-base, and sdk-trace-node. These must be in dependencies — the module loader resolves all static imports on first import of 'duron/client', regardless of whether telemetry is configured. Removes the optional peerDependencies since they're not optional.
- telemetry.mdx: update installation to only require exporter package - client-api.mdx: remove stale local/spanProcessors options, show current API
- Fix stale comment in client.ts referencing telemetry.local - Remove spansCount from ArchiveStatsResponse type (server no longer returns it) - Remove dead 'Archived Spans' card from archive page
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements an Active/Archive split for Duron's PostgreSQL adapter to eliminate hot-path bloat and improve query performance.
Breaking Changes
20260421153337_large_nitrospanstable (no active/archive split)What's Changed
Core Architecture:
jobsintojobs_activeandjobs_archivejob_stepsintojob_steps_activeandjob_steps_archivespanstable with no FK constraints to avoid cascade issuescreated/active→ active,completed/failed/cancelled→ archiveArchive Management:
_pruneArchive- scheduled cleanup with advisory locks_truncateArchive- clear all archive data_getArchiveStats- archive metricsBug Fixes:
Dashboard:
Developer Experience:
AGENTS.md: do not use git worktreesTesting
test/archive.test.tswith 18 archive-specific testspackages/shared-actions/test/process-order.test.tsintegration test