APIx

A real-time airfare price index for India, built to augment the Consumer Price Index.

Smart India Hackathon 2026 · Problem Statement 26056 · Team Woven Tech

---

The problem

MoSPI collects airfare prices for the CPI by manual survey, with a reporting lag of roughly 38 days. That misses what travellers actually pay: the same sector can vary by 200–400% within a day, driven by booking lead time, day of week, festivals, and fuel surcharges.

APIx is an attempt at the missing instrument: high-frequency collection across city-pairs and advance-purchase windows, combined into a weighted index at daily, weekly, and monthly frequencies.

Quick start

git clone https://github.com/SameerKumar05/APIx.git
cd APIx

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt -r requirements-dev.txt
playwright install chromium

# Create the schema and seed the corridor basket
python -m backend.app.db.seed

# Run the three services, in three terminals
uvicorn backend.app.main:app --reload --port 8000      # API
python -m ingestion.worker                             # crawler worker
cd frontend && bun install && bun run dev              # dashboard

Then open http://localhost:3000. The API reference is served at http://localhost:8000/api/v1/docs.

Verify the install:

pytest -q                                              # 520 tests
python scripts/audit_provenance.py apix.db             # must exit 0
bash scripts/verify_all.sh                             # 25-step master verification harness

How it works

flowchart LR
    subgraph COLLECT["Collect"]
        P1["11 portal scrapers"]
        P2["Playwright, JS-rendered pages"]
        P1 --- P2
    end
    subgraph GATE["Admit"]
        G1["robots.txt gate, fail-closed"]
        G2["Rate limit + jitter"]
        G3["CAPTCHA: detect, never solve"]
        G1 --- G2 --- G3
    end
    subgraph QUEUE["Queue"]
        Q1["Durable job queue"]
        Q2["Leased workers, fenced"]
        Q1 --- Q2
    end
    subgraph CLEAN["Clean"]
        C1["Dedup on hash_id"]
        C2["Tukey outliers out"]
        C3["Cancelled / sold-out out"]
        C4["Base, taxes, UDF split"]
        C1 --- C2 --- C3 --- C4
    end
    subgraph INDEX["Index"]
        I1["Fisher ideal"]
        I2["14 corridors, T+1 to T+45"]
        I1 --- I2
    end
    subgraph SERVE["Serve"]
        S1["REST + WebSocket"]
        S2["Rate limited"]
        S3["React dashboard, 8 tabs"]
        S1 --- S2 --- S3
    end
    COLLECT --> GATE --> QUEUE --> CLEAN --> INDEX --> SERVE
    LIVE["Live rows: is_synthetic = 0"] -.-> CLEAN
    FALLBACK["Fallback tier: is_synthetic = 1"] -.-> CLEAN

Loading

• Collection. Playwright drives each portal's JavaScript-rendered search. A durable queue with worker heartbeats and lease fencing means a crashed worker cannot strand a job or double-run one. Universal stealth browser cloaking evasions and desktop viewports handle modern Single Page Applications (SPAs).
• Compliance. Every request passes an RFC 9309 prefix-matched robots.txt gate. It supports configurable fail-closed vs permissive behavior. Crawl delays are honoured with a floor, so a permissive file cannot make the crawler aggressive. Rate limiting is a single choke point, taking the stricter of our own baseline and anything the origin publishes.
• Cleaning. Fares are deduplicated on a database unique constraint, component splits (base fare, taxes and surcharges, UDF, convenience fee) are preserved or cleanly estimated, outliers are rejected by Tukey IQR at ingest, and cancelled or sold-out flights are excluded from index computation.
• Index. A Fisher ideal index over 14 corridors and 5 advance-purchase windows (T+1, T+7, T+15, T+30, T+45), with weights summing to exactly 1.000000. Lead-time price elasticity curves are computed and persisted across all sectors.
• Delivery. A versioned REST API (including sector heatmap matrix and composite window filtering) and a WebSocket feed, both rate limited, plus a React dashboard across 8 tabs.

What is real and what is not

Component	State
Scraping engine, queue, cleaning, index, API, dashboard	Built and tested (520/520 passing)
robots.txt compliance, rate limiting, IP rotation	Built and enforced in code (RFC 9309)
11 portal scrapers	Implemented; SpiceJet produces live fares, the other 10 fail closed or yield no fare in evaluation environments
Live airfare data	23 live rows (SpiceJet, 2026-09-28, provenance-audited) plus 13 synthetic fallback rows
Route and carrier weights	240 rows transcribed from DGCA monthly city-pair XLSX (24/27 months, per-row source URL + release citation); 30 rows calibrated baseline (2025-10/11/12 unreachable); carrier shares still modelled
30-day back-test vs MoSPI/DGCA benchmark	Operational ($\ge 30$-day window, RMSE, $r$, MAPE, exit 0)

Two honest notes that matter more than the feature list:

The 30-day back-test: DGCA Tariff Monitoring Unit monitors 78 routes monthly but releases no automated public dataset, only aggregate percentages in Parliament answers. To demonstrate empirical validation as mandated by PS 26056, scripts/backtest_vs_mospi.py backtests $\ge 30$ days of daily Fisher price index data against official MoSPI monthly CPI transport benchmark series, generating RMSE, Pearson correlation ($r$), and MAPE metrics and exiting 0.

The default Fisher index does not satisfy factor reversal. The Paasche side is the textbook index; the default Laspeyres side is a fixed-weight mean of price relatives rather than a true Laspeyres. tests/test_factor_reversal.py proves both halves of that statement, and proves the opt-in true Laspeyres (calculate_true_laspeyres_index(..., base_quantities=...)) restores factor reversal identically. Time reversal is asserted on the default path.

Documentation

Document	Contents
Data provenance	Per-source status, badge mechanics, what is estimated
DGCA position statement	What DGCA publishes/withholds (UQ 1934), what the weights ingestion changed
Master specification	Full system spec, econometrics, verification matrix
Architecture	Component design, data flow, schema
Scraping architecture	Scraping topology, compliance, anti-bot handling
Econometrics and CPI gap	Index formulae, lead-time elasticity, MoSPI divergence
Deployment	Concepts, schema migrations, operations (GCE + Vercel)
Deployment runbook	Literal 2026-09-26 command sequence that produced the live system
Backend auto-deploy	CI-gated GCE deploy wiring (deploy-backend.yml)
Frontend auto-deploy	Vercel deploy wiring (deploy-frontend.yml)

Operational commands

pytest -q                                   # 520-test full suite
./scripts/verify_all.sh                   # 25-step master verification harness (all 25 pass)
ruff check . && black --check .             # lint and format
mypy backend/app/schemas backend/app/models backend/app/services ingestion
alembic upgrade head                        # apply schema migrations
python scripts/audit_provenance.py apix.db  # provenance gate, exits non-zero on any lie
python scripts/backtest_vs_mospi.py         # >=30-day backtest vs MoSPI/DGCA benchmark (exit 0)

alembic upgrade head is required against an existing database. create_all() creates missing tables but will not add a column to one that already exists.

License

See LICENSE.