A self-contained lab for running distributed PySpark analytics over the full NYC Yellow Taxi dataset (2016–2025, ~655M rides) on a local Kubernetes cluster, plus a streaming tier that replays those trips as Kafka events and processes them with Apache Flink. Everything - the dev environment, the cluster, the Spark standalone cluster, Kafka, Flink, and the data - is provisioned reproducibly from code, so the whole stack comes up with a single devcontainer build.
┌──────────────────────── VS Code Devcontainer (Python 3.11) ────────────────────────────┐
│ │
│ Jupyter / spark_benchmark.py ──► Spark Driver ─┐ │
│ (taxi_data_platform) │ spark://…:7077 │
│ ▼ │
│ ┌──────────────── kind cluster (Kubernetes, provisioned by Terraform) ───────────┐ │
│ │ │ │
│ │ spark-master (Deployment + Service) │ │
│ │ ▲ │ │
│ │ │ registers │ │
│ │ spark-worker × N (custom Python 3.11 image, 1 core / 8G each) │ │
│ │ │ │ │
│ │ └── mounts taxi-pvc (hostPath PV) ◄── seeded parquet (2016–2025) │ │
│ │ │ │ │
│ │ ┌─────────────────────┼──────────────── streaming tier ──────────────────┐ │ │
│ │ │ taxi-producer Job ─┘ reads PV, time-ordered replay │ │ │
│ │ │ │ JSON events │ │ │
│ │ │ ▼ │ │ │
│ │ │ Kafka (KRaft, 1 broker) topic: taxi-trips │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ Flink session cluster (JobManager + TaskManagers, PyFlink) │ │ │
│ │ │ │ tumbling event-time windows │ │ │
│ │ │ ▼ │ │ │
│ │ │ Kafka topic: taxi-metrics ──► console consumer │ │ │
│ │ └────────────────────────────────────────────────────────────────────────┘ │ │
│ └────────────────────────────────────────────────────────────────────────────────┘ │
│ │
│ Tilt orchestrates the k8s manifests + port-forwards Spark (:8080) & Flink (:8081) │
└────────────────────────────────────────────────────────────────────────────────────────┘
The entire environment is defined under .devcontainer/ and builds from .devcontainer/Dockerfile. Open the repo in VS Code and "Reopen in Container" — no host tooling required beyond Docker.
What the image provides:
- Python 3.11 (
python:3.11-bookworm) with uv for dependency management - OpenJDK 17 (required by the Spark client)
- Infra CLIs baked in:
kubectl(v1.33),terraform,kind(v0.32),tilt - Docker-in-Docker so the container can build images and run the kind cluster
Bring-up sequence (run automatically on container create/start):
| Hook | Script | Does |
|---|---|---|
postCreateCommand |
post-create-commands.sh | uv sync deps → create kind cluster + seed data → build & load the custom Spark worker image |
postStartCommand |
post-start-commands.sh | Wait for Docker, recreate the kind cluster if it didn't survive a restart, refresh kubeconfig |
Forwarded ports & UIs (available after make up-everything):
| Service | URL | Notes |
|---|---|---|
| Tilt UI | http://localhost:10350 | Resource status, logs, and controls for all k8s workloads |
| Spark UI | http://localhost:8080 | Jobs, stages, executors, and storage for the Spark standalone cluster |
| Flink UI | http://localhost:8081 | Running jobs, TaskManagers, and checkpoints for the Flink session cluster |
All cluster resources are declarative. The Makefile wires the steps together.
| Tool | What it does |
|---|---|
| Docker | Container runtime underpinning everything. Runs the devcontainer and, via Docker-in-Docker, hosts the kind node container and builds the custom Spark worker image. |
| Terraform | Infrastructure-as-code. Declaratively provisions (and destroys) the local kind cluster through the tehcyx/kind provider, so the cluster is reproducible from terraform/main.tf. |
| kind | Kubernetes-in-Docker — runs a full Kubernetes cluster inside a single Docker container, no cloud or VM required. Hosts the Spark master/workers and the taxi-data volume. |
| Kubernetes | Container orchestrator that kind runs. Schedules the Spark master/worker Pods, exposes the master via a Service, persists data with a PV/PVC, and handles scaling/self-healing of the worker Deployment. |
| Tilt | Dev orchestrator. Applies the Spark manifests and port-forwards the master (7077) and UI (8080), so one command (tilt up) brings the cluster workloads up with a live dashboard. |
terraform/main.tf provisions a local Kubernetes cluster via
the tehcyx/kind provider — a single-node kind cluster named terraform-kind
(kindest/node:v1.29.2). make create-k8-cluster runs terraform apply, exports
the kubeconfig, and seeds the taxi data.
The taxi parquet files are downloaded to the host with
datasets/NYC_Yellow_Taxi_Trip/download.sh
(120 monthly files, 2016–2025). make seed-data then:
- Applies a 15Gi
ReadWriteManyhostPath PV/PVC (taxi-pv.yaml, taxi-pvc.yaml) - Spins up a busybox seeder pod (taxi-seeder.yaml)
kubectl cps each year's parquet into the volume, then tears the pod down
Spark runs in standalone mode inside the cluster (kubernetes/spark/):
| Manifest | Resource |
|---|---|
| spark-rbac.yaml | spark ServiceAccount + RBAC |
| spark-master.yaml | Master Deployment + spark-master-svc (ports 7077 / 8080) |
| spark-workers.yaml | Worker Deployment — N replicas, 1 core / 8G each |
| spark-worker.Dockerfile | Custom worker image with Python 3.11 |
A single-broker KRaft Kafka (no ZooKeeper) defined in raw manifests (kubernetes/kafka/):
| Manifest | Resource |
|---|---|
| kafka.yaml | Kafka StatefulSet (KRaft combined broker+controller) + headless kafka-svc |
| kafka-init.yaml | Job that creates the taxi-trips (6 partitions) and taxi-metrics topics |
| taxi-producer.yaml | Job that runs the producer — reads from the shared PV, replays trips in event-time order |
| taxi-producer.Dockerfile | Producer image (Python 3.11 + kafka-python) |
A standalone session cluster mirroring the Spark master/workers layout (kubernetes/flink/):
| Manifest | Resource |
|---|---|
| flink-jobmanager.yaml | JobManager Deployment + Service (REST/UI 8081) |
| flink-taskmanager.yaml | TaskManager Deployment (N replicas, 4 slots each) |
| flink.Dockerfile | Custom image: Flink 2.3 + Kafka SQL connector |
Tiltfile applies all manifests (Spark, Kafka, Flink) and port-forwards
the Spark master (7077 / 8080), Kafka (9092, external 9094), and Flink
(8081). Bring the whole stack up with:
make up-everything # create-k8-cluster + tilt upBoth tiers run on the public NYC Yellow Taxi Trip Records (NYC Taxi &
Limousine Commission), 2016–2025: 120 monthly parquet files, ~655 million
rides (~15 GB). Rows are filtered to the valid year range before processing,
and the Spark jobs read only the columns they need (tpep_pickup_datetime,
fare_amount) so unused columns are pruned at the parquet level. Download via
download.sh.
A PySpark analytics application that answers four questions about New York City taxi traffic over a decade of trip records. The Spark driver runs inside the devcontainer while the computing is distributed across the Spark workers.
- spark/session.py —
SparkSessionfactory +spark_sessioncontext manager. Resolves a driver host IP the executor pods can route back to (the devcontainer is the kind bridge gateway, not a peer) and pinspython3.11on the executors so UDFs work. - spark/jobs/ — one module per analysis. Each exposes a pure
DataFrame → DataFramefunction (imported by both the notebook and the benchmark) plus a standalonemain()for one-off CLI runs. _common.py holds the shared multi-year loader. - spark/taxi_data_analysis.ipynb — loads the dataset once (cached,
MEMORY_AND_DISK), runs all four jobs, and renders + exports the charts at 300 DPI to charts/. - spark_benchmark.py — runs every job x times at y workers, uncached, reporting avg/std (see §4).
All charts were produced on the full dataset (~655M rides) using a 10-worker Spark cluster (1 core / 8 GB each).
| Chart | Question |
|---|---|
 |
How many rides per month? — rides_per_month.pygroupBy(year, month).count() over the full dataset. The COVID-19 lockdown collapse in March 2020 dominates the trend. |
 |
Which month is busiest on average? — avg_rides_per_month.py Per-year monthly totals collapsed into an average per calendar month, drawn as parallel coordinates with the average overlaid. |
 |
Month-over-month fare change — mom_fare_change.py Spark SQL with a LAG(...) OVER (ORDER BY year, month) window function to compute the percentage change in average fare versus the prior month. |
 |
Rides by time of day — rides_by_time_of_day.py A vectorized Pandas UDF classifies each pickup hour into a time-of-day bucket (Night / Morning Rush / … / Evening), then aggregates. |
The same parquet data also feeds a streaming pipeline: a producer replays trips in pickup-time order as Kafka events, and Flink computes windowed metrics that it writes back to Kafka. All components are managed by the same Tilt/Makefile workflow as Spark.
flink/kafka/producer.py reads the
parquet from the shared PV, prunes out-of-range timestamps (matching the Spark
loader), sorts each file by pickup time, and publishes one JSON event per
trip. A --speedup factor maps event-time gaps to wall-clock sleeps, so the
historical data plays back as a (compressed) live stream with event-time
semantics. It runs as a Job from a small
image (Python 3.11 + kafka-python).
A standalone session cluster. Jobs are built and submitted straight from Python —
flink/session.py is the Flink analog of
create_spark_session: it returns a remote TableEnvironment pointed at the
JobManager REST endpoint (localhost:8081), so there is no flink CLI, kubectl,
or ConfigMap involved. Each job in flink/jobs/
exposes a run(t_env) function (shared Kafka source DDL with an event-time
watermark lives in _common.py):
- rides_per_window.py — ride count per 1-day tumbling window
- rolling_avg_fare.py — average fare per 1-day tumbling window
Both emit append-only rows (one per closed window) to the taxi-metrics topic.
flink/taxi_streaming_analysis.ipynb
is the streaming analog of the Spark notebook: it submits both Flink jobs (via flink.session), replays trips into Kafka, reads the windowed metrics back from taxi-metrics, and renders + exports charts (300 DPI)
to charts/:
| Chart | Question |
|---|---|
 |
Rides per day — the live demand curve |
 |
Average fare per day |
 |
Demand vs. fare on a shared timeline |
Run with spark_benchmark.py: It reads the full dataset fresh on every run (no caching, identical work per run) so timings are comparable across worker counts. Below: average seconds ± std over 5 runs, by number of workers (1 core each).
| Job | 1 worker | 5 workers | 10 workers | 20 workers |
|---|---|---|---|---|
rides_per_month |
76.64 ± 2.71 | 17.86 ± 2.04 | 10.68 ± 2.42 | 8.74 ± 3.88 |
avg_rides_per_month |
72.05 ± 0.56 | 16.34 ± 0.33 | 9.25 ± 0.47 | 6.79 ± 0.46 |
mom_fare_change |
87.18 ± 0.56 | 20.26 ± 0.35 | 11.30 ± 0.64 | 8.27 ± 1.01 |
rides_by_time_of_day |
146.98 ± 1.13 | 35.17 ± 1.65 | 22.02 ± 2.14 | 20.42 ± 2.42 |
| Total | 382.85 | 89.63 | 53.25 | 44.22 |
Scaling from 1 → 20 workers cuts the end-to-end suite from ~383s to ~44s, roughly an 8.7× speedup.
Run with flink_benchmark.py. It loads 1,000,000 trips (January 2025) into Kafka once, then submits the bounded job at increasing parallelism, timing each run. Below: average seconds ± std over 5 runs, by Flink parallelism (= workers, 1 task slot each).
| Job | 1 worker | 5 workers | 10 workers | 20 workers |
|---|---|---|---|---|
rides_per_window |
9.20 ± 0.78 | 3.19 ± 0.77 | 3.25 ± 0.75 | 3.44 ± 1.91 |
avg_fare |
8.88 ± 0.03 | 2.85 ± 0.03 | 4.93 ± 4.52 | 2.42 ± 0.73 |
| Total | 18.08 | 6.03 | 8.18 | 5.86 |
# Full stack (cluster + data + Spark + Kafka + Flink + Tilt UI)
make up-everything
# Spark batch benchmark
make spark-benchmark runs=5 workers=10
# open notebook
make notebook
# Flink benchmark (run for each worker count)
make flink-benchmark runs=5 workers=10
# Quality gates
make run-tests check-linting check-types