The background-workers plugin for NetScript: one install wires durable jobs, multi-runtime tasks, workflows, a Workers API service, worker CLI commands, and Aspire orchestration into your app.
Background processing is usually a shopping trip — a queue library here, a scheduler there, an admin
API you write yourself, and deployment glue nobody wants to own. @netscript/plugin-workers ships
the whole capability as one declarative manifest: netscript plugin install worker scaffolds a
workers workspace, registers the Workers API service, provisions the storage it needs, and adds the
worker processes to your Aspire AppHost so everything starts with the rest of your app.
The manifest is plain data. Hosts read it to generate files and wiring; nothing executes until your
app boots. The reusable job, task, and workflow builders live in
@netscript/plugin-workers-core — this package
binds them to a NetScript host.
- One manifest, whole capability —
workersPlugindeclares services, background processors, stream topics, database schema, runtime-config topics, contract versions, and Aspire resources as typed contribution axes the host turns into running processes. - Multi-runtime tasks — jobs and tasks execute across Deno, PowerShell, Python, and shell
runtimes with at-least-once delivery keyed on
idempotencyKey. - A real operations surface — the plugin CLI covers the job lifecycle end to end:
add-job,add-task,add-workflow,run,run-task,list-jobs,executions,logs,trigger,enable/disable, andcompile-registry. - Workers API included — the
workers-apiservice uses an Aspire-allocated endpoint and exposes job and execution introspection over a versioned contract, so dashboards and agents query state instead of scraping logs. - Durable by default —
./streamspublishes execution and job entities to durable stream topics, and worker metrics flow through the shared NetScript telemetry conventions. - Aspire-native —
./aspirecontributes the worker resources to the AppHost, so local orchestration and deployment see workers as first-class processes.
Install without example definitions, then generate the registry:
netscript plugin install workers --no-db --no-samples
netscript generate plugins
netscript plugin doctorAn installed plugin with zero definitions gets a typed empty registry. Doctor reports the empty
registry as a warning and exits successfully; a missing or stale registry remains an error. With
its configured dependencies available, the background runner starts idle and reports ready with
zero definitions. Add your first definition and run netscript generate plugins again. Removing
the last definition and regenerating keeps a valid empty registry.
netscript generate plugins selects definition modules from workers/jobs/*.ts and plugin-provided
job directories. Keep a helper or a dependency-injected factory beside those definitions by putting
this exact comment on its first line:
// @netscript-registry-ignore
export function scopedId(scope: string, name: string): string {
return `${scope}.${name}`;
}For example, add it to workers/jobs/helper.ts. The directive is shared by all plugin runtime
registries and standalone registry compilers. It excludes the whole module before generating
imports; ordinary imports from your application keep working. Use a separate module for a top-level
job definition that should remain registered.
The marker must be the complete first-line comment, before any license header or code. Indentation, a UTF-8 BOM and CRLF are accepted; a later comment, a string containing the marker, or extra text on the line does not opt out. Existing manifest suffix and filename exclusions still apply.
Regenerate and inspect the result:
netscript generate plugins
netscript plugin doctorDoctor names eligible modules carrying the directive as Explicitly excluded registry modules, on
both healthy and stale registries. Remove the directive and regenerate to register the module again.
Unmarked modules retain the plugin's existing definition validation. Keep the CLI, shared plugin
package and installed plugins on matching release versions so their selector and inspection protocol
agree.
A workers config entry pointing at an excluded job remains an unmatched-path error. Remove that config entry or restore a registrable module; exclusion does not silently disable configured jobs.
flowchart LR
M["workersPlugin<br/>(manifest: services, schema,<br/>topics, Aspire resources)"] --> H["NetScript host<br/>(plugin install + sync)"]
H --> A["workers-api<br/>Aspire-allocated endpoint"]
H --> W["Worker + Scheduler<br/>processes"]
W --> R["Runtimes<br/>Deno · PowerShell · Python · shell"]
W --> S["Durable streams<br/>executions · jobs"]
worker.pause() stops intake for jobs, tasks and queue triggers while admitted work keeps running
and renewing its leases. worker.resume() resumes intake after that listener generation has joined
its admitted work, so overlapping calls cannot exceed concurrency. Pause keeps worker.start()
pending and reports a healthy paused lifecycle through worker.healthStatus.lifecycle and
worker.isPaused. Repeated pause/resume calls are harmless. A claim already committed when pause
arrives can be returned without executing its handler; that ordinary requeue still counts.
await worker.stop() stops intake and gives admitted work time to finish normally and acknowledge.
It then aborts leftovers, waits for cleanup, and closes its queues within one total drainTimeoutMs
budget. The default is 25,000 ms: 19 s work, 5 s abort cleanup, 1 s settlement.
abortGracePeriodMs defaults to min(5_000, Math.floor(drainTimeoutMs / 5)); shorter totals also
reserve min(1_000, max(1, floor(total / 25))) milliseconds for settlement. Total must be an
integer from 1 to 2,147,483,647; grace must be an integer from zero to total. A hung handler,
storage call or queue close can make stop reject TimeoutError within this total. Stop is
idempotent and terminal for that Worker instance; create a new Worker to start again.
Set both durations in the project's workers section, including generated workers:
import { defineWorkers } from '@netscript/plugin-workers-core/config';
export const workers = defineWorkers({
drainTimeoutMs: 25_000,
abortGracePeriodMs: 5_000,
});startWorkerProcess and startCombinedProcess accept explicit duration overrides, otherwise use
validated project workers configuration, otherwise the same defaults. Direct new Worker(...)
accepts the same options. Its top-level grace overrides the legacy
workerPoolOptions.abortGracePeriodMs; pool-only customization remains supported. Keep the parent
process drain at least five seconds longer than the Worker total; raise the parent budget too when
raising this total. Redis/KV-polling listener drainTimeoutMs has a different scope: it bounds
callback waiting after intake stops. Worker gives those listeners an infinite wait and owns their
finite total itself, keeping receipts valid through pause and handback.
Jobs returned because their first cancellation cause is controlled shutdown retain their current
delivery count. Repeated clean restarts therefore do not exhaust maxDeliveries for healthy jobs.
Ordinary handler failure requeues still count. An earlier failure, deadline, operator cancellation
or lease loss cannot become a free retry merely because shutdown happens later. Delivery remains at
least once: uncertain acknowledgement or persistence can replay an effect, so use idempotent effects
and stable message/domain keys.
Deno handlers receive an executor-owned signal and optional epoch-millisecond deadlineAt.
Worker.cancel(executionId) immediately requests cancellation of a local active execution and
returns false for absent or already-cancelled work. Timeout records timeout; shutdown and explicit
cancel record cancelled, release the claim and cannot record success after abort. The standalone
InProcessJobRunner also defaults to five seconds of abort cleanup. Cancellation is cooperative;
in-process JavaScript and CPU-blocking work cannot be physically terminated by the runner. Forward
context.signal to cancellable operations. This Worker owns and stops supplied queue and
taskQueue ports as it does defaults.
Declare retry: { maxRetries, backoff, delayMs, maxDelayMs } on a discovered schema-backed handler
with Object.assign(handler, { id, retry }), or use the core builder's .retry(...) method.
Handler fields override definition defaults, and both registry generators preserve the policy.
In-process and polyglot jobs share one bounded retry executor: maxRetries counts retries after the
initial attempt, and zero still runs once. Timeout is terminal; shutdown requeues without a job dead
letter.
Exhausted jobs are stored through Worker.deadLetters before the transport acknowledges them. Its
default is a KV-backed specialization of the queue's canonical dead-letter port in a separate
namespace. WorkerOptions.deadLetters accepts a replacement; the port supports listing, depth and
controlled reprocessing. Worker.readHealthStatus() and the generated background health route read
durable depth, and a storage outage makes dependency readiness unavailable. Job retries retain one
delivery, so size visibility and active-claim leases for the full retry budget and make effects
idempotent across transport redelivery.
See Tune the worker runtime for handler examples, backoff formulas, precedence, replay, and time-budget sizing.
From the root of a NetScript project:
netscript plugin install worker --name workersThe plugin owns its setup — the CLI ships no embedded templates. The scaffolder wires the Workers
API service, background processors, stream topics, database schema, and Aspire resources into your
workspace, then pins the matching @netscript/* versions. Workers require Deno KV and optionally
Postgres; the install records both from the manifest so netscript db and Aspire provision them for
you.
To consume the plugin programmatically (custom hosts, tests, tooling), add it as a library:
deno add jsr:@netscript/plugin-workers@<version>The standalone plugin CLI is also directly runnable:
deno x -A jsr:@netscript/plugin-workers@<version>/cli list-jobsPin <version> to match your installed CLI; bare jsr:@netscript/* specifiers do not resolve on
the pre-release line.
Use the inert @netscript/plugin-workers/service entrypoint for a caller-owned listener. The
factory requires a native auth policy, accepts native CORS and serve options, and returns the
native running handle. The host supplies its database port, KV adapter and credential; the
following recipe shows that composition boundary without opening a host database here:
import '@netscript/kv/redis'; // Register only the adapter selected by the host.
import { getKv } from '@netscript/kv';
import { createPluginServiceContext } from '@netscript/plugin/sdk';
import { createStaticCredentialAuthenticator } from '@netscript/service/auth';
import { startWorkersService } from '@netscript/plugin-workers/service';
declare const openDatabaseClient: () => Promise<unknown>;
declare const workerCredential: string; // Obtain from the host's secret authority.
const context = await createPluginServiceContext('workers', {
getDatabaseClient: () => openDatabaseClient(),
getKv: () => getKv(),
});
const authenticator = createStaticCredentialAuthenticator({
credentials: {
[workerCredential]: {
subject: 'service:workers', scopes: ['workers:read'], roles: ['service'],
},
},
});
const controller = new AbortController();
const running = await startWorkersService(context, {
auth: { authn: { authenticator } },
serve: { hostname: '127.0.0.1', port: 0, signal: controller.signal },
});
try {
console.log(running.addr.port); // Publish the allocated endpoint through the host.
} finally {
await running.stop(); // Drains requests; the host still owns and closes its KV.
}serve.port, including zero, takes precedence over context PORT, then process PORT.
Supply auth: { public: true, reason: '...' } only for an intentional public service. The factory
forwards policy and listener options without changing them. Importing the singular entrypoint
does not register Redis, read environment values or start a listener; the explicit host import
above performs adapter registration.
Grant runtime permissions for the chosen listener, KV backend and host configuration: network
access to those endpoints, environment access to the configuration the host reads, and read/write
access to the selected local KV path if used. Enable Deno KV with --unstable-kv. Restrict these
permissions to the host's resources; the recipe's type check does not qualify a deployed host.
The plural @netscript/plugin-workers/services default remains the legacy context delegate. It
registers Redis, preserves the existing public policy reason, reads context PORT before process
PORT, and uses native wildcard-host and signal defaults. Its direct-run bootstrap still requires
NETSCRIPT_PLUGIN_SERVICE_BOOTSTRAP_MODULE. Choose the singular factory to own auth and binding.
netscript generate plugins discovers worker modules from the installed plugin's
scaffold.runtime.json, then reads project policy from netscript.config.ts. The manifest owns
discovery metadata only; it is never a second policy store. Validation and defaults remain owned by
@netscript/plugin-workers-core/config, and the registry generator consumes that normalized result.
Policy binds to a discovered module by its normalized project-relative entrypoint. The configured source must agree with discovery, and a plugin-owned handler's exported id must agree with its configured id; unmatched or conflicting paths, ids, and sources stop generation with a diagnostic. Discovered jobs with no config entry retain the generic registry defaults for compatibility.
Grouped jobs are canonical. When workers.groups[].jobs[] and legacy workers.jobs[] declare the
same id and entrypoint, the grouped definition wholly shadows the flat definition, including its
group topic and every policy field. A partial collision—same id at another path or the same path
with another id—is an error rather than an order-dependent merge.
export default {
name: 'orders',
databases: { config: [] },
workers: {
jobsDir: './workers/jobs',
groups: [{
topic: 'billing',
jobs: [{
id: 'reconcile-invoices',
name: 'Reconcile invoices',
entrypoint: './reconcile-invoices.ts',
priority: 80,
maxConcurrency: 2,
}],
}],
},
};Install the plugin, then list the jobs it manages:
$ netscript plugin install worker --name workers
Installed worker plugin "workers" on port <allocated-port>.
Created 4 plugin files.
Regenerated 12 Aspire helper files.
$ deno x -A jsr:@netscript/plugin-workers@<version>/cli list-jobs
Found 0 worker jobs.
{
"jobs": []
}As a library, the manifest is inspectable data:
import { workersPlugin } from '@netscript/plugin-workers';
console.log(workersPlugin.name); // "@netscript/plugin-workers"
// Declared service contributions (the Workers API port is allocated by Aspire).
const service = workersPlugin.contributions.services?.[0];
console.log(service?.name); // "workers-api"
console.log(service?.port); // undefined unless explicitly pinned| Entry | What it gives you |
|---|---|
. |
workersPlugin — the typed PluginManifest with every contribution axis |
./cli |
The workers command group (add-job, run, list-jobs, executions, …) |
./worker |
The Worker consumer and cron Scheduler runtime classes |
./services |
The Workers API service composition (workers-api, Aspire-allocated port) |
./streams |
Durable-stream factory for execution and job entities |
./aspire |
The workers Aspire contribution for the AppHost |
./contracts |
The versioned workers API contract generated registries bind against |
./scaffold |
The plugin-owned scaffolder netscript plugin install worker executes |
The always-current symbol list is
deno doc jsr:@netscript/plugin-workers@<version>
(pin <version> on the pre-release line, as above).
- Workers reference — commands, service, and contract: rickylabs.github.io/netscript/reference/workers/
- Background Processing — jobs, tasks, and workflows end to end: rickylabs.github.io/netscript/background-processing/
- How-to — roll out runtime overrides: rickylabs.github.io/netscript/how-to/roll-out-runtime-overrides/
- API docs on JSR: jsr.io/@netscript/plugin-workers/doc
The worker runtime, CLI, and Workers API service require Deno 2.9+ (they use Deno.* and Deno KV
APIs). The manifest itself is plain data and can be imported anywhere TypeScript runs. Task runtimes
additionally shell out to PowerShell, Python, or a POSIX shell when a definition targets them, so
those interpreters must be present on the machine that runs such tasks.
Apache-2.0 — see LICENSE. Published to JSR with cryptographically verified provenance.
Each job's retention.kvRetentionDays (default 3) bounds terminal execution records under
workers/executions. The execution state store and the registry execution-write port both set KV
expireIn from completedAt; rewriting a result or progress record does not extend that deadline.
Pending, queued, and running executions remain durable while they are open. An explicit
archiveToDb: false never causes a database archive in these KV write or cleanup paths.
DELETE /cleanup now deletes expired terminal execution records instead of returning a stubbed
success. It inspects at most 1000 entries per call and advances to the next page on later calls.
Atomic version checks prevent it from deleting a record updated after inspection. Fresh terminal
records and open executions are preserved. This endpoint also removes legacy execution records
written without TTL; normal new-record expiry is performed by the KV backend and needs no app
session. Job/task definitions remain configuration rather than expiring execution history.
For native policy/CORS/listener composition and explicit SDK credentials, see Host workers and streams services.