Skip to content

Add priority queue - #93

Open
donaldgray wants to merge 3 commits into
mainfrom
feature/priority
Open

Add priority queue#93
donaldgray wants to merge 3 commits into
mainfrom
feature/priority

Conversation

@donaldgray

@donaldgray donaldgray commented Jul 31, 2026

Copy link
Copy Markdown
Member

Adds a new endpoint: POST /customers/{customer}/queue/priority (mimics POST /customers/{customer}/queue but skips queue and ends up in priority DLCS queue).

A priority submission is expedited both in terms of processing and ingest:

  • Process: tasks are enqueued to a dedicated queue/cluster (named by PRIORITY_QUEUE_NAME) with its own small worker pool, so they are never blocked by a standard-queue backlog. Each engine container runs a second qcluster process for it (entrypoint-worker.sh);. This means no new instances or services running as I expect use to be rare and want to avoid infra overhead.
  • Ingest: batches for priority collections are POSTed to the DLCS /customers/{customer}/queue/priority endpoint, so they also skip any DLCS-side backlog.

If PRIORITY_QUEUE_NAME is unset the behaviour is as before

Commits

  1. django-q==1.3.9django-q2==1.10.0, the maintained drop-in fork. Prerequisite for multi-queue routing (ALT_CLUSTERS), also seems like sensible update regardless.
  2. Add priority queue for expedited submissions - the feature itself: new route, Collection.priority field + migration, cluster routing in the view, DLCS priority ingest, worker entrypoint. Also repaired the integration test stack, which referenced an old Dockerfile.CompositeHandler left over from when this was part of the protagonist.

Configuration

New configuration:

Variable Default Notes
PRIORITY_QUEUE_NAME unset (disabled) With the SQS broker, must equal the name of a real SQS queue
PRIORITY_WORKER_COUNT 1 Priority workers per engine instance

Infrastructure

New priority SQS queue (same settings as the standard one) and set the above env vars on the API and engine services.

Migrations

New migration to add priority field.

donaldgray and others added 2 commits July 31, 2026 16:13
django-q 1.3.9 has been unmaintained since 2021; django-q2 is the
maintained drop-in fork (same django_q import path and INSTALLED_APPS
entry). Ships additional django_q migrations, applied by the existing
'manage.py migrate' step. Prerequisite for routing tasks to multiple
queues via ALT_CLUSTERS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
POST /customers/{customer}/queue/priority accepts the same payload as
the standard endpoint but routes processing through a VIP lane at both
levels, mirroring the DLCS API's own priority queue:

- locally, tasks are enqueued to a dedicated queue/cluster (named by
  PRIORITY_QUEUE_NAME) with its own worker pool, so they are not
  blocked by a standard-queue backlog. Each engine container runs a
  second qcluster process for it; no new instances are required.
- at ingest, batches for priority collections are POSTed to the DLCS
  /queue/priority endpoint.

If PRIORITY_QUEUE_NAME is unset the feature is inert: priority
submissions use the standard local queue but still ingest via the DLCS
priority endpoint. See docs/priority-queue.md for the full design.

Also repairs the integration test stack, which referenced a
Dockerfile.CompositeHandler left over from the protagonist monorepo
import and was unrunnable: builds the root Dockerfile instead, supplies
the required env vars, gates services on a postgres healthcheck, and
bumps pytest-docker for Docker Compose v2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@donaldgray
donaldgray requested a review from fmcc July 31, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant