This repository hosts externally sourced datasets used for dashboarding and analysis. It is designed for personal use, experimentation, and learning.
- Store publicly available data from external organizations
- Maintain cleaned, analysis-ready datasets
- Provide stable data sources for Power BI dashboards
Open the repository in GitHub Codespaces or in VS Code using "Reopen in Container".
Dependencies will be installed automatically as part of the container setup.
Create a virtual environment and install dependencies:
python -m venv .venv
source .venv/bin/activate # or .venv\Scripts\activate on Windows
pip install -r requirements.txtEach dataset has its own folder containing:
raw/: original downloaded filesprocessed/: cleaned and standardized filesscripts/: dataset-specific processing scriptsprocess_latest_raw.py: process latest raw file
README.md: source-specific readme file
Cross-cutting documentation:
- data catalog
- update processes
- conventions
Power BI should connect to: sources/<dataset-name>/processed/current.csv
These files are stable and updated as new data becomes available.
Data is updated manually or via scripts:
- Download latest data →
raw/ - Clean/transform →
processed/ - Update
current.csv - Archive dated snapshot
- Folders: lowercase-with-hyphens
- Files: ISO date prefix where applicable (
YYYY-MM-DD) - Columns: lowercase_with_underscores
This is a lightweight data pipeline intended for:
- prototyping
- public dashboards
- exploratory analysis
Future enhancements may include:
- automated updates
- curated output layer
- validation scripts