This project orchestrates the end-to-end data integration of Europe COVID-19 datasets using Azure Data Factory. It automates the ingestion, validation, transformation, and loading of COVID-19 case, death, testing, and population data into a structured format.
Directory structure:
βββ README.md
βββ publish_config.json
βββ dataflow/ -- # Contains all data flow definitions
βββ dataset/ -- # Contains dataset definition
βββ factory/ -- # Factory configuration
βββ linkedService/ -- # Linked service connections
βββ pipeline/ -- # Pipeline definitions
βββ trigger/ -- # Pipeline triggers
Europe COVID-19 datasets were collected from the European Centre for Disease Prevention and Control (ECDC).
- Azure Data Factory
Pipelines
π Pipeline: pl_ingest_ecdc
Automates the ingestion of raw case and death data from ECDC stored in ADLS Gen2. It retrieves a list of files and iterates over them using a ForEach activity, copying each into the destination for downstream processing.

π Pipeline: pl_ingest_population
Validates and ingests population data files. It checks for file existence, verifies schema integrity using metadata, and conditionally copies valid files into the data lake. Invalid files can be logged or flagged for review.

π Pipeline: pl_process_casesDeaths
Processes raw case and death data using the df_transform_casesDeaths data flow, which cleanses and reshapes the data for analytics.

Data flows
π Data Flow: df_transform_hospitalAdmissions
Transforms raw hospital admissions data by joining with lookup tables, aggregating by date and country, and creating weekly and daily outputs.

π Data Flow: df_transform_casesDeaths
Filters for European countries, pivots COVID-19 case/death indicators, enriches with metadata, and exports structured results.

π Data Flow: df_transform_testing
Processes raw testing data by integrating date and country dimensions, performing weekly aggregation, and exporting the results.

- Databricks
The pipeline pl_process_population executes a Databricks notebook to transform raw population data. It filters to 2019, cleans percentage values, pivots by age group, enriches with country metadata, and saves the processed output to ADLS.
- Azure Data Lake Gen 2
Used for lookup, raw and processed data storage.

- Azure SQL DB
Azure SQL DB is used as a final destination for curated COVID-19 datasets. Data from processed pipelines (e.g., testing or hospital admissions) can be written to SQL tables for easy querying, reporting, or BI consumption.




