A local-first data warehouse built around DuckDB. Databox ingests public data with dlt, coordinates concurrent writes through Quack, transforms it with SQLMesh, validates it with Soda, and orchestrates the workflow with Dagster—without always-on infrastructure.
Rufous is the user-facing birding product built on Databox: a React/TypeScript interface for interactive Arizona encounter maps, personal collections, explicit source refresh, watched-bird alerts, and evidence-grounded trip planning.
React/TypeScript -> typed FastAPI APIs -> DuckDB warehouse -> bounded Google ADK workflow and strict-schema model inference
The full Rufous warehouse experience is local-first and loopback-only; DuckDB and model credentials stay behind the local typed API. A separate browser-only public export is deployed at rufous.loughondata.com with static, privacy-reviewed data and no database or model credentials.
Rufous does not estimate encounter probability. Its recently reported group contains species with distinct eBird submissions in the configured eBird lookback (30 days back by default, with both boundary dates included); its GBIF occurrence-context group contains species without qualifying eBird submissions in that lookback. The first group sorts by eligible submission count, newest report, then species name; the second sorts by distinct occurrence count, newest occurrence date or year, then species name. Each source record counts once. The planner records the exact date range in its trace; all eBird and GBIF evidence used for ranking is within the enforced 50 km radius.
task full-refresh # populate routine sources after the one-time bootstrap below
task app:dev # FastAPI :8000 + Vite :5173 with hot reload
task app:check # typecheck + tests + build + configured bundle audit
task app:audit-bundle # audit an existing build
task app # build and serve at http://127.0.0.1:8000task verify is a bounded smoke refresh for pipeline verification; it is not the
data-population step for Rufous.
See the Rufous operations guide for local setup and operator-only delivery procedures.
flowchart LR
sources[Public sources] --> dlt[dlt]
dlt -->|writes through Quack| duckdb[(DuckDB)]
duckdb --> sqlmesh[SQLMesh]
soda[Soda] -. validates .-> duckdb
dagster[Dagster] -. orchestrates .-> dlt
dagster -. orchestrates .-> sqlmesh
dagster -. asset checks .-> soda
New dlt sources move through a reviewable, agent-guided modeling workflow:
flowchart LR
schema[dlt schema] --> annotate["annotate-sources<br/>annotations + taxonomy"]
annotate --> ontology[create-ontology]
ontology --> cdm["generate-cdm<br/>Kimball CDM"]
cdm --> transform["create-transformation<br/>SQLMesh models"]
The project skills—annotate-sources,
create-ontology,
generate-cdm, and
create-transformation—turn raw
schemas into business-aware warehouse models before transformation SQL is
written. See the workflow.
Prerequisites: Python 3.12+, uv, and Task. Node.js 22+ and npm are only needed for Rufous.
git clone https://github.com/Doctacon/databox.git
cd databox
task install # creates .env from .env.example when absent
task ciAfter the initial dependency install, source tests replay recorded responses, so
task ci needs neither provider credentials nor a populated warehouse.
After task install, configure EBIRD_API_TOKEN, NOAA_API_TOKEN, and
XENO_CANTO_API_KEY in .env. Live trip-plan synthesis also requires
CF_WORKERS_AI_API_KEY and CF_WORKERS_AI_ACCOUNT_ID; keep the example's
allowlisted model selector. For a new database, bootstrap the pinned AVONET
snapshot once, then refresh the routine sources:
$EDITOR .env
mkdir -p data .dagster
DAGSTER_HOME="$PWD/.dagster" PYTHONPATH="$PWD" \
uv run dg launch --target-path packages/databox --job avonet_ingest
task full-refresh # ingest and transform into data/databox.duckdb
task app # build and serve Rufous at http://127.0.0.1:8000AVONET is intentionally excluded from routine refreshes. See the operations runbook.