Pool Classifier

Pool Classifier is a Cloud Run service and Flask dashboard for monitoring Taskcluster worker pools. It periodically classifies recent task results, matches task logs against failure patterns, and surfaces pool health, alerting workers, success rates, and unclassified failures.

The Python package remains named worker_health for compatibility after the repository extraction. Those package-path references are vestigial.

Quick start

uv sync --group dev
./pc_db.sh init
./pc_start.sh

The local dashboard is available at http://localhost:8080/. A Taskcluster token is required at ~/.tc_token; see the development guide for its format and the complete local workflow.

To launch the production-style Gunicorn server directly, use:

uv run pool-classifier
PORT=8090 uv run pool-classifier

It binds to 0.0.0.0:8080 by default. PORT, POOL_CLASSIFIER_HOST, GUNICORN_WORKERS, GUNICORN_THREADS, and GUNICORN_TIMEOUT configure the same defaults as the Cloud Run container; command-line options take precedence.

Preview a task classification

Use the read-only task-classifier command to compare the current working-tree patterns with patterns at a Git ref for one Taskcluster task:

uv run task-classifier TASK_ID
uv run task-classifier TASK_ID --run 0 --base-ref origin/main

TASK_ID is required. --run selects a terminal run (the newest terminal run is the default); --base-ref selects the baseline pattern revision (default: HEAD). --provisioner, --worker-type, and --poll-interval use the same defaults as the polling classifier, and --no-color disables terminal colors. The command only fetches task data and prints a comparison; it does not update classification storage or patterns.

To inspect current detail-page code without waiting for a classifier to replace a cached dashboard snapshot, start the port-8081 debug listener with snapshots disabled. pc_start.sh enables Flask debug mode automatically on port 8081; the default port 8080 remains the stable target for pc_fetch_data.sh:

PC_PORT=8081 POOL_CLASSIFIER_DISABLE_DASHBOARD_SNAPSHOTS=1 ./pc_start.sh

Documentation

Development

• Local setup, configuration, tests, and local tools

Run the PostgreSQL-backed suite with a terminal coverage report locally:

scripts/run_local_postgres_tests.sh --cov=worker_health --cov-report=term-missing

CI enforces 65% overall coverage, publishes coverage.xml as a workflow artifact, and reports coverage to Codecov.

Operations

• Cloud Run runbook
• Release and ad-hoc deployment procedure
• Production database maintenance
• Backfill observed start-lag metadata
• Backfill recent job-source metadata

API reference

• Public API
• Utilization API
• Observed scheduled-to-start lag API

Design and history

• Utilization and queue lag for pool sizing
• Fitness and Pool Classifier comparison
• Cloud Run migration history
• Dashboard query refactor
• 2026-07-30 database migration-lock incident

Repository layout

worker_health/
  pool_classifier.py                  # Taskcluster polling and classification
  pool_classifier_web/                # Flask app, storage, migrations, and Terraform
tests/                                # pytest suite
scripts/                              # test, build, benchmark, and maintenance helpers