Skip to content

About

🦠 ADF pipelines to ingest, process, and transform COVID-19 data from the ECDC

Resources

Stars

0 stars

Watchers

1 watching

Forks

Latest commit

Β 

History

19 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

eu-covid19-adf

Overview

This project orchestrates the end-to-end data integration of Europe COVID-19 datasets using Azure Data Factory. It automates the ingestion, validation, transformation, and loading of COVID-19 case, death, testing, and population data into a structured format.

Table of Contents

  1. Overview
  2. Architecture
  3. Project Structure
  4. Data Source

Architecture

image

Project Structure

Directory structure:

    β”œβ”€β”€ README.md
    β”œβ”€β”€ publish_config.json
    β”œβ”€β”€ dataflow/        --  # Contains all data flow definitions
    β”œβ”€β”€ dataset/         --  # Contains dataset definition
    β”œβ”€β”€ factory/         --  # Factory configuration
    β”œβ”€β”€ linkedService/   --  # Linked service connections
    β”œβ”€β”€ pipeline/        --  # Pipeline definitions
    └── trigger/         --  # Pipeline triggers

Data Source

Europe COVID-19 datasets were collected from the European Centre for Disease Prevention and Control (ECDC).

Technology Stack

  1. Azure Data Factory

Pipelines

πŸ“„ Pipeline: pl_ingest_ecdc Automates the ingestion of raw case and death data from ECDC stored in ADLS Gen2. It retrieves a list of files and iterates over them using a ForEach activity, copying each into the destination for downstream processing. image

πŸ“„ Pipeline: pl_ingest_population Validates and ingests population data files. It checks for file existence, verifies schema integrity using metadata, and conditionally copies valid files into the data lake. Invalid files can be logged or flagged for review. image

πŸ“„ Pipeline: pl_process_casesDeaths Processes raw case and death data using the df_transform_casesDeaths data flow, which cleanses and reshapes the data for analytics. image

Data flows

πŸ”„ Data Flow: df_transform_hospitalAdmissions Transforms raw hospital admissions data by joining with lookup tables, aggregating by date and country, and creating weekly and daily outputs. image

πŸ”„ Data Flow: df_transform_casesDeaths Filters for European countries, pivots COVID-19 case/death indicators, enriches with metadata, and exports structured results. image

πŸ”„ Data Flow: df_transform_testing Processes raw testing data by integrating date and country dimensions, performing weekly aggregation, and exporting the results. image

  1. Databricks

The pipeline pl_process_population executes a Databricks notebook to transform raw population data. It filters to 2019, cleans percentage values, pivots by age group, enriches with country metadata, and saves the processed output to ADLS.

image

image

  1. Azure Data Lake Gen 2

Used for lookup, raw and processed data storage. image

  1. Azure SQL DB

Azure SQL DB is used as a final destination for curated COVID-19 datasets. Data from processed pipelines (e.g., testing or hospital admissions) can be written to SQL tables for easy querying, reporting, or BI consumption.

image

image

image

About

🦠 ADF pipelines to ingest, process, and transform COVID-19 data from the ECDC

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors