apify-aws-actor

An Apify actor that scrapes a sanctioned public source, enriches each item with an LLM via OpenRouter, and ships as a single Docker image deployable to AWS Fargate/Lambda — with scheduling, batching, parallelization, failure monitoring, and alerting.

What it does

This repository is a production-shaped Apify actor. It scrapes a sanctioned public source (books.toscrape.com, a site built expressly for scraping practice), splits work into batches that run with bounded concurrency, optionally enriches each item through an OpenRouter LLM call, pushes results to the Apify dataset, and emits structured health metrics. The exact same Docker image is the artifact deployed to AWS (Fargate task or Lambda container), scheduled via EventBridge or GitHub Actions cron, with failure monitoring (schema-drift detection) and alerting (webhook/SNS).

Skills demonstrated

Job bullet / capability	Where it is proven (file → function/symbol)
Apify actor SDK lifecycle	src/apify_aws_actor/main.py → main() (async with Actor), run_actor()
OpenRouter LLM enrichment	src/apify_aws_actor/enrich_llm.py → OpenRouterClient.enrich()
Sanctioned-source scrape + parse	src/apify_aws_actor/scraper.py → scrape_url(), parse_catalogue()
Selector stability vs. site changes	src/apify_aws_actor/selectors.py → CATALOGUE, SELECTOR_VERSION
Batching + parallelization at scale	src/apify_aws_actor/batching.py → run_batched(), gather_bounded()
Failure monitoring + schema-drift	src/apify_aws_actor/monitoring.py → RunMetrics, detect_schema_drift()
Alerting with dedup + dry-run	src/apify_aws_actor/alerting.py → AlertDispatcher.dispatch()
Config + ToS-safe allowlist	src/apify_aws_actor/config.py → Settings.assert_allowed()
Containerization (one image, many targets)	Dockerfile (multi-stage)
AWS infra (Fargate/EventBridge/CloudWatch)	infra/*.json, infra/README.md
Scheduling	infra/eventbridge-schedule.json, .github/workflows/scheduled-run.yml
CI: lint + offline tests + docker build	.github/workflows/ci.yml
Offline test suite (mocked network/LLM)	tests/ + tests/conftest.py

Architecture

Apify run  /  AWS Fargate task  (same Docker image)
        │
        ▼
config.py ── validates input, enforces ALLOWED_SOURCES allowlist
        │
        ▼
batching.py ── split catalogue URLs into batches, bounded asyncio.Semaphore
        │
        ▼
scraper.py ── httpx fetch + selectolax parse (selectors.py is the single source of truth)
        │
        ▼
enrich_llm.py ── OpenRouter /chat/completions per item (best-effort, degrades gracefully)
        │
        ▼
Actor.push_data ──▶ Apify dataset (dataset_schema.json view)
        │
        ▼
monitoring.py ── RunMetrics + schema-drift, emits structured JSON log line
        │
        ▼
alerting.py ──▶ webhook / (AWS) CloudWatch metric filter ▶ alarm ▶ SNS

Data source & legal/ToS note

The actor targets books.toscrape.com (and optionally quotes.toscrape.com) — sites published specifically as legal sandboxes for scraping practice. The allowlist in config.py (ALLOWED_SOURCES) makes any non-sanctioned host a hard PermissionError, so the actor cannot be pointed at auth-walled or rate-abused targets. Concurrency defaults are deliberately conservative to stay polite. No credentials are used and no robots/ToS are circumvented.

Quickstart

git clone https://github.com/RaphaelFakhri/apify-aws-actor.git
cd apify-aws-actor

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

cp .env.example .env          # fill in keys only if you enable enrichment/alerts

# Run locally (no Apify platform needed; uses the same orchestration core):
python -m apify_aws_actor.main

# Or via the Apify CLI:
#   npm i -g apify-cli && apify run

Configuration

All env vars are prefixed ACTOR_ (see config.py).

Env var	Default	Meaning
ACTOR_SOURCE_KEY	books	Sanctioned source key
ACTOR_MAX_ITEMS	40	Upper bound on items per run
ACTOR_BATCH_SIZE	10	Catalogue pages per batch
ACTOR_CONCURRENCY	4	Max concurrent requests per batch
ACTOR_ALLOWED_SOURCES	books.toscrape.com,quotes.toscrape.com	Host allowlist
ACTOR_ENABLE_ENRICHMENT	false	Toggle OpenRouter enrichment
ACTOR_OPENROUTER_API_KEY	(none)	OpenRouter key (env only, never committed)
ACTOR_OPENROUTER_MODEL	openai/gpt-4o-mini	Model slug
ACTOR_ALERT_WEBHOOK_URL	(none)	Generic alert webhook
ACTOR_DRY_RUN_ALERTS	true	When true, alerts are logged not sent

Actor input fields (validated by Apify) are declared in .actor/input_schema.json: source_key, max_items, batch_size, concurrency, enable_enrichment.

Running tests offline

pytest -q

The whole suite runs with no network. tests/conftest.py serves committed HTML (tests/fixtures/catalogue_page.html) and a canned OpenRouter response (tests/fixtures/openrouter_response.json) through an httpx.MockTransport, and push_data is a local collector — so scraping, enrichment, batching, monitoring, alerting, and a full end-to-end run are all exercised deterministically.

Docker

docker build -t apify-aws-actor:latest .
docker run --rm -e ACTOR_MAX_ITEMS=20 apify-aws-actor:latest

This image is the AWS artifact — build once, deploy to Fargate or Lambda.

Deploying to AWS

See infra/README.md. Summary:

1. Build/push the image to ECR.
2. Register infra/fargate-task-def.json.
3. Create SNS topic + subscription; create the metric filter + alarm (infra/cloudwatch-alarm.json).
4. Create the schedule (infra/eventbridge-schedule.json).
5. Lambda container alternative: infra/lambda-container.md.

Scheduling & batching tuning

• Scheduling: pick one of EventBridge (infra/eventbridge-schedule.json) or GitHub Actions cron (.github/workflows/scheduled-run.yml) as the source of truth to avoid duplicate runs.
• Batching/parallelism: raise ACTOR_BATCH_SIZE/ACTOR_CONCURRENCY for throughput, but keep concurrency polite to respect the source.

Monitoring, alerting & stability against site changes

• monitoring.py emits a single-line JSON run_summary metric (success rate, counts, schema-drift fields) for CloudWatch Logs Insights / metric filters.
• detect_schema_drift() flags expected fields that vanished across all items — the canonical symptom of a site-structure change — and trips a critical alert.
• selectors.py localizes every CSS selector behind a SELECTOR_VERSION stamp that is recorded on every record and metric, so a drift is traceable to a selector revision.
• alerting.py dispatches to a webhook (and surfaces an email), deduplicates identical alerts within a run, and honors DRY_RUN.

Project layout

.actor/        Apify manifest, input + dataset schemas
src/apify_aws_actor/   package (main, scraper, batching, enrich_llm, monitoring, alerting, config, selectors)
infra/         AWS infra-as-notes (Fargate, EventBridge, CloudWatch, Lambda notes)
tests/         offline pytest suite + fixtures
Dockerfile     multi-stage image (the AWS artifact)
.github/workflows/   ci.yml, scheduled-run.yml

Contributing & lint/format

ruff check .
ruff format .
pytest -q

License & disclaimer

MIT — see LICENSE. This project scrapes only sanctioned practice sites; do not repoint it at sources whose ToS or robots policy disallow scraping. The infra/ JSON is illustrative and uses placeholder account IDs/ARNs.