Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ The sample Actors (`samples/actor_*`) crawl the live web, so running them needs

The behavioural spec lives in `requirements/*.md`: `system.md`, `api.md`, `storage.md`,
`actor-driver.md`, `cli.md`, `console.md` and `test.md`. `unsupported.md` lists the platform behaviour
the runtime deliberately leaves out.
the runtime deliberately leaves out. The one exception is the runtime-only `/actor-runtime/*` namespace,
whose contract is the OpenAPI document `src/api/openapi/actor-runtime.json`; `api.md` only points at it.

User-facing documentation lives in `skills/actor-runtime/SKILL.md`, which ships inside the image. When
a change alters what the runtime does for its users, update it in the same commit. Contribution
Expand Down
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,16 @@ Your Actor's own output is passed through unchanged.
- **Test pay-per-event pricing for free.** See what a run charges and costs, without spending money.
- **Relay missing calls to the platform.** Send calls the runtime cannot answer to the real Apify API.

Most of these are driven by endpoints under `/actor-runtime/*`, which the real Apify API does not have.
The runtime describes that namespace in an OpenAPI document and serves it, so you never have to guess
what a given runtime supports - and, since the platform answers `404` there, the same call also tells you
whether you are pointed at a runtime at all:

```bash
apify api GET /actor-runtime # every runtime-specific endpoint, its body and its responses
curl -s http://localhost:3333/actor-runtime/openapi.json | jq .paths # same document, for tooling
```

### Edit without rebuilding

`apify push` registers the pushed folder as the Actor's **dev folder**. Every later run mounts it over
Expand Down
223 changes: 33 additions & 190 deletions requirements/api.md

Large diffs are not rendered by default.

15 changes: 14 additions & 1 deletion requirements/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,19 @@
With neither source, `/users/me`'s `proxy` field is omitted and no `APIFY_PROXY_PASSWORD` is set
on run containers - never a placeholder.

## Enumerating the runtime's own endpoints

- The runtime-specific endpoints (`/actor-runtime/*`, `api.md`) have no counterpart on the Apify
platform, so a client cannot learn them from the platform's own OpenAPI specification. The runtime
therefore serves its own specification for that namespace, and the CLI reads it with a single stock
call: **`apify api GET /actor-runtime`** - one response enumerating every runtime-specific endpoint,
its request body, and its responses.
- No token is needed for that call, and the real Apify platform answers `404` on the same path, so it
doubles as "is the CLI pointed at a local Actor runtime, and which one?" - the document's
`info.version` is the runtime's own version.
- A client that asks for something in that namespace which the specification does not describe is told
so in those terms: `404` naming `GET /actor-runtime`, or `405` with an `Allow` header (`api.md`).

## Supported commands (POC)

- `apify push` - creates the Actor and Actor version from local source and triggers
Expand Down Expand Up @@ -71,7 +84,7 @@
`apify-cli` fetches its actor-templates manifest from the internet. Every later
push/call/log-stream/storage-access, and every build of an already-pulled base image, needs no
outbound network access (see `system.md`'s offline-after-first-build note) - unless the opt-in
upstream API fallback is enabled (`api.md`, "Upstream fallback"), in which case an eligible local
upstream API fallback is enabled (`api.md`, and the `api-fallback` operation in the runtime API specification it references), in which case an eligible local
miss makes one outbound request to the configured upstream instead of failing offline.
- The bundled sample Actors crawl the live web (`https://crawlee.dev/` by default), so an `apify call`
that runs one of them needs outbound network access from the Actor container even though the
Expand Down
2 changes: 1 addition & 1 deletion requirements/console.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@
## Settings page

- Every page's header navigation includes a link to `/settings`, the one page for the upstream API
fallback toggles (`api.md`'s "Upstream fallback" section); the link itself shows each toggle's current
fallback toggles (specified by the runtime API specification `api.md` references); the link itself shows each toggle's current
value.
- `/settings` shows `fallbackUnimplementedEnabled`, `fallbackNotFoundEnabled`, and `upstreamBaseUrl`
(the same values the API's toggle endpoint reports), plus a warning that enabling either toggle
Expand Down
5 changes: 5 additions & 0 deletions skills/actor-runtime/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,11 @@ a later step misses. Only a call naming an Actor this runtime does not know is r
`apify api GET v2/datasets/~my-results/items`. Names match case-insensitively. A bare name without
`~` is an id, except for an Actor, where it is also tried as a name. Another user's resource is not
found here - the runtime only ever shows your own.
- `apify api GET /actor-runtime` prints this runtime's own OpenAPI document - every `/actor-runtime/*`
endpoint above, and what it adds to the platform endpoints it extends. No token needed, and the real
Apify platform has no such endpoint, so a `404` there means you are talking to the platform rather
than to a runtime. `http://localhost:3333/actor-runtime/openapi.json` serves it unenveloped, for
OpenAPI tooling.
- Or unauthenticated by URL: `http://localhost:3333/v2/datasets?token=TOKEN`.
- The `runs/last` shortcuts address an Actor's newest run without knowing its id:
`apify api GET v2/actors/<actorId>/runs/last`, and the same under `/log`, `/dataset/items`,
Expand Down
115 changes: 115 additions & 0 deletions src/api/actor-runtime-spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
/**
* The `/actor-runtime/*` namespace's own OpenAPI document (`openapi/actor-runtime.json`) and the
* lookups the server drives from it.
*
* The document - not this module, and not `requirements/api.md` - is the single source of truth for
* which runtime-specific endpoints exist: `routes/actor-runtime-spec.ts` serves it verbatim so a client
* can enumerate the namespace, and the namespace's terminal handler decides `404` vs `405` vs `426`
* from these lookups. Adding an endpoint therefore means adding it to the document; a test
* (`test/integration/actor-runtime-spec.test.ts`) fails if a documented HTTP operation has no route
* behind it, so the two cannot drift apart silently.
*
* This is the namespace-local counterpart of `spec-table.ts`, which does the same 501-vs-404 job for
* the emulated Apify `/v2` surface from a vendored snapshot of the platform's own spec.
*/
import rawDocument from './openapi/actor-runtime.json' with { type: 'json' };

/** Only the parts of OpenAPI this runtime actually reads - not a general-purpose OpenAPI model. */
interface OperationObject {
operationId?: string;
summary?: string;
/** `"websocket"` marks an operation served by the HTTP upgrade handler rather than an Express route. */
'x-actor-runtime-transport'?: string;
}

interface OpenApiDocument {
openapi: string;
info: { title: string; version: string; description?: string };
paths: Record<string, Record<string, unknown>>;
}

/** Lower-case, as OpenAPI spells them; anything else under a path item (`parameters`, `summary`, ...)
* is not an operation. */
const HTTP_METHODS = ['get', 'put', 'post', 'delete', 'options', 'head', 'patch', 'trace'] as const;

export const ACTOR_RUNTIME_OPENAPI = rawDocument as unknown as OpenApiDocument;

export interface ActorRuntimeOperation {
/** Upper-case, matching `req.method`. */
method: string;
/** The OpenAPI path template, e.g. `/actor-runtime/debug/{actorId}`. */
path: string;
operationId: string;
summary: string;
/** `websocket` operations are upgraded by `events-ws.ts`, so no Express route ever matches them. */
transport: 'http' | 'websocket';
}

function readOperations(document: OpenApiDocument): ActorRuntimeOperation[] {
const operations: ActorRuntimeOperation[] = [];
for (const [path, pathItem] of Object.entries(document.paths)) {
for (const method of HTTP_METHODS) {
const operation = pathItem[method] as OperationObject | undefined;
if (!operation) continue;
operations.push({
method: method.toUpperCase(),
path,
operationId: operation.operationId ?? `${method}${path}`,
summary: operation.summary ?? '',
transport: operation['x-actor-runtime-transport'] === 'websocket' ? 'websocket' : 'http',
});
}
}
return operations;
}

export const ACTOR_RUNTIME_OPERATIONS: readonly ActorRuntimeOperation[] = readOperations(ACTOR_RUNTIME_OPENAPI);

/** `/actor-runtime/events/` and `/actor-runtime/events` are the same path here; the empty trailing
* segment is not a segment. A bare `/actor-runtime` normalizes to no segments beyond the namespace. */
function segmentsOf(path: string): string[] {
return path.split('/').filter(Boolean);
}

/** Structural match, same rule as `spec-table.ts`: equal segment count, literal segments equal,
* `{param}` segments wildcard. */
function pathMatches(template: string, requestPath: string): boolean {
const templateSegments = segmentsOf(template);
const requestSegments = segmentsOf(requestPath);
if (templateSegments.length !== requestSegments.length) return false;
return templateSegments.every(
(segment, i) => (segment.startsWith('{') && segment.endsWith('}')) || segment === requestSegments[i],
);
}

/**
* Every operation the document describes on the path template `requestPath` matches - empty when the
* document describes no such path at all. The namespace's terminal handler branches on this: empty is
* a `404` (off-spec path), non-empty with no matching method is a `405` (the `Allow` header is built
* from exactly these).
*
* `requestPath` is the full namespace path (`/actor-runtime/...`), with or without a trailing slash,
* and without a query string.
*/
export function actorRuntimeOperationsAtPath(requestPath: string): ActorRuntimeOperation[] {
return ACTOR_RUNTIME_OPERATIONS.filter((operation) => pathMatches(operation.path, requestPath));
}

/**
* The router-relative Express path for a documented operation - `/actor-runtime/events/{runId}` ->
* `/events/:runId` - so a route can be registered from the document instead of from a second,
* hand-maintained copy of the same path.
*/
export function routerPathOf(operation: ActorRuntimeOperation): string {
const relative = segmentsOf(operation.path)
.slice(1)
.map((segment) => (segment.startsWith('{') && segment.endsWith('}') ? `:${segment.slice(1, -1)}` : segment))
.join('/');
return `/${relative}`;
}

/** The one operation the document describes for this method and path, if any. */
export function matchActorRuntimeOperation(method: string, requestPath: string): ActorRuntimeOperation | undefined {
const upperMethod = method.toUpperCase();
return actorRuntimeOperationsAtPath(requestPath).find((operation) => operation.method === upperMethod);
}
Loading
Loading