Skip to content

feat: add Windmill PBF export script to clip OSM data per sandbox project and upload to S3 - #7275

Open
prabinoid wants to merge 1 commit into
developfrom
feat/sandbox-data-export
Open

feat: add Windmill PBF export script to clip OSM data per sandbox project and upload to S3#7275
prabinoid wants to merge 1 commit into
developfrom
feat/sandbox-data-export

Conversation

@prabinoid

@prabinoid prabinoid commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this? (check all applicable)

  • 🍕 Feature
  • 🐛 Bug Fix
  • 📝 Documentation
  • 🧑‍💻 Refactor
  • ✅ Test
  • 🤖 Build or CI
  • ❓ Other (please specify)

Related Issue

Related to #7238

Describe this PR
Adds a self-contained Windmill Python script (scripts/export/pbf_data_export.py) that automates OSM data exports for active Tasking Manager projects.

On each run it:

  • Fetches project boundaries from the TM active-projects API (or a static GeoJSON)
  • Downloads a dated .osm.pbf from S3
  • Builds OSM geometry tables (points/lines/polygons) in an in-memory DuckDB instance
  • Clips each project to its boundary, applies tag filters per category (Buildings, Roads, Waterways, Landuse, Seamarks), and writes GeoJSON/SHP/KML outputs
  • Zips and uploads results to S3 (or writes locally for testing)
  • Supports two extraction engines (pure via ST_ReadOSM and quackosm) and adapts automatically to DuckDB MAP accessor semantics across versions. The script is designed to be dropped directly into Windmill with all parameters exposed as a typed form.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant