modern-data-engineering

Extract Load Transform (ELT) framework is a metadata based batch orchestration framework for modern data platforms. Implemented using Azure PaaS data services. Common ingestion and transformation patterns available out of box. Reusable code can be easily extended to cater to custom patterns.

In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider.

This curated list brings together powerful open-source tools, frameworks, and resources for data engineering 🛠️ and data science 📈. It started with inspiration from pracdata's awesome-open-source-data-engineering list 🌟 and has grown into a resource to organize and explore the ever-evolving data landscape 🌐.