End-to-end Azure data engineering pipeline using Spotify dummy data with Medallion Architecture, Databricks, Delta Live Tables, and incremental loading.
Configuration-driven ETL framework in Python — YAML-defined pipelines, plugin architecture, incremental loading, data quality checks, DuckDB warehouse.
A metadata-driven, CDC-based Medallion architecture pipeline that ingests a Spotify-style streaming warehouse from Azure SQL, processes it through Bronze → Silver → Gold using Databricks Autoloader and Delta Live Tables, and models it as a Star Schema in Unity Catalog — with SCD Type 2 history, Git-based CI/CD, and automated failure alerting.
This project demonstrates an end-to-end ETL pipeline built using Snowflake modern layered architecture. pipeline ingests retail data from CSV files, validates , transforms the data, loads clean records into curated tables, stores invalid records in reject tables, supports incremental loading using MERGE, and maintains execution logs for monitoring.
Add a description, image, and links to the incrementalloading topic page so that developers can more easily learn about it.
To associate your repository with the incrementalloading topic, visit your repo's landing page and select "manage topics."