Current implemented scope: Phase 4 — artist metadata enrichment. The repository
contains a local Bronze → Silver → Gold pipeline plus separate MusicBrainz and
Last.fm artist enrichment. See docs/PROJECT_WALKTHROUGH.md
for the technical walkthrough.
Personal data engineering project for collecting my Last.fm listening history.
Current status: Phase 4 — Artist metadata enrichment
Last.fm API
↓
Python ingestion
↓
Bronze / Raw JSON
↓
Silver transformation
↓
Silver / Typed Parquet
↓
Gold analytical marts
↘
Separate artist enrichment: MusicBrainz + Last.fm tags
Bronze stores immutable, auditable user.getRecentTracks responses. Silver extracts completed scrobbles, normalizes optional fields, converts timestamps to UTC, deduplicates overlapping events, preserves Bronze lineage, and writes typed Parquet. Gold builds local analytical marts for activity, artists, tracks, discovery, temporal patterns, streaks, concentration, and diversity.
The available history begins in 2016, although tracking was intermittent before 2020. Consistent Last.fm usage begins approximately in 2020. Low or absent pre-2020 scrobble counts must not be interpreted as low music consumption.
git clone <repository>
cd lastfm-data-platform
python -m venv .venvActivate the virtual environment using the command for your operating system, then install dependencies:
pip install -r requirements.txt
cp .env.example .envSet these values in .env:
LASTFM_API_KEY
LASTFM_SHARED_SECRET
LASTFM_USERNAME
The shared secret is retained for configuration completeness but is not needed for this read-only endpoint.
Run a small development ingestion:
python -m src.ingestion.ingest_scrobbles --backfill --max-pages 2Run the complete historical backfill:
python -m src.ingestion.ingest_scrobbles --backfillIf a backfill stops after a transient failure, resume the latest incomplete metadata-backed run:
python -m src.ingestion.ingest_scrobbles --backfill --resumeEach execution has one run ID shared by all of its pages and writes run_metadata.json with running, failed, or completed state. Older Bronze directories created before run metadata was introduced are preserved and are not automatically migrated or resumed.
Transform all completed metadata-backed Bronze runs into Silver:
python -m src.transformation.transform_scrobbles --fullProcess only completed Bronze runs not already recorded in the Silver manifest:
python -m src.transformation.transform_scrobblesSilver output is local and ignored by Git. See docs/phase2-decisions.md for the event identity, run selection, timestamp, null handling, storage, and incremental processing decisions.
Build Gold analytical marts from Silver:
python -m src.analytics.build_gold --fullThe Gold build is a deterministic full rebuild because the local Silver dataset is small enough for inexpensive aggregation. Outputs remain local under data/gold/ and are ignored by Git. See docs/phase3-decisions.md for grains, coverage eras, UTC handling, discovery definitions, streaks, concentration, and limitations.
Run incremental ingestion using the latest completed scrobble found in Bronze as its watermark:
python -m src.ingestion.ingest_scrobblesEach execution writes to a unique run directory under data/bronze/lastfm/recent_tracks/. Overlapping raw responses between runs are intentional; event-level deduplication belongs in a future Silver layer. Now-playing tracks are preserved in raw payloads but are excluded from watermark calculation.
Build the Phase 5 enriched analytical product from local Silver and Phase 4 outputs:
python -m src.analytics.build_enriched --profile
streamlit run dashboard/app.pyPhase 5 writes ignored local outputs under data/gold_enriched/ and consumes
them in a Streamlit dashboard without reading raw personal data. It validates
the artist bridge, keeps tags as a one-to-many association, profiles genre,
geography, lifecycle, diversity, concentration and discovery, and labels the
incomplete 2026 calendar year. Gold Enriched also derives historical local-time
analytics from explicit IANA timezone periods while retaining UTC as canonical.
See docs/phase5-decisions.md
for the actual grains, weighting policy, coverage and findings.
Tests use mocked HTTP responses and never call Last.fm:
pytestPhase 4 artist enrichment is implemented separately from Bronze and Silver:
python -m src.enrichment.enrich --profile --resume
python -m src.enrichment.enrich --from-cache --profile --resumeThe second command rebuilds normalized outputs from the local raw cache without network calls. The real local snapshot contains 116,871 Silver events and 4,956 artist entities. Tracking before 2020 is incomplete, and calendar year 2026 is incomplete; recorded counts must not be interpreted as complete consumption.
See docs/phase4-decisions.md and the project
walkthrough for grains, coverage and limitations.
Phase 4 artist metadata enrichment is implemented separately from Bronze and Silver using cache-first MusicBrainz identity resolution and Last.fm artist tags.
- Phase 1 — Bronze ingestion
- Phase 2 — Silver normalization, data quality and incremental processing
- Phase 3 — Gold analytical marts
- Phase 4 — Artist metadata enrichment
- Phase 5 — Enriched analytics and local portfolio dashboard
- Phase 6 — Future architecture and productionization (not implemented)