Skip to content

data.md

Latest commit

 

History

History
247 lines (208 loc) · 10.8 KB

data.md

File metadata and controls

247 lines (208 loc) · 10.8 KB
Raw

External data sources

RivMapper does not bundle any of the public datasets it uses; you download them once into a per-machine working_directory and every river you process points at the same shared trees.

This document lists the datasets, where to get them, the on-disk layout each RivMapper module expects, and the citation to use in any paper that builds on the analysis.

At-a-glance checklist

Dataset Used by On-disk root
HydroRIVERS v1.0 (HydroSHEDS) extract_hydrorivers <wd>/HydroATLAS/HydroRIVERS/HydroRIVERS_v10_<zone>_shp/
Nyberg global channel belts channel_belt <wd>/ChannelBelts/{tifs/, GRM_index.shp}
Per-river channel-belt outline (user-drawn) channel_belt <wd>/ChannelBelts/Extracted_ChannelBelts/<river>/<river>_outline.shp
GRWL vector tiles + IMWtiles index create_masks <wd>/Allen_Pavelsky_2018/GRWL_vector_V01.01/, shapefile_data/IMWtiles.shp
GEE: Landsat 5/7/8/9 Collection 2 create_masks (server-side, no local files)
GEE: NASADEM / ArcticDEM hydraulic_geometry (server-side, no local files)
GQBF v0.1 geopackages hydraulic_geometry, transit_length <wd>/GQBF/GQBFv0.1_reaches_<continent>_EPSG4326.gpkg
WBMsed Annual Averages extract_wbmsed <wd>/WBMsed/Averages/*.asc
BASED XGBoost depth model hydraulic_geometry <wd>/Gearon_etal_2024/based-api/*.ubj

<wd> is the value of the working_directory column for the river you're processing. RivMapper writes its outputs into the same tree, so a typical working directory ends up looking like:

<wd>/
  HydroATLAS/...                     <- input
  ChannelBelts/...                   <- input + output
  Allen_Pavelsky_2018/...            <- input
  GQBF/...                           <- input
  WBMsed/...                         <- input + output
  Gearon_etal_2024/...               <- input
  RiverMapping/                       <- output
    Reaches/<river>/
    Channels/<river>/
    HydraulicGeometry/<river>/
    RiverMasks/<river>/
    Mobility/<river>/
    Visualization/<river>/

Dataset details

HydroRIVERS v1.0

Continental shapefiles of every river segment ≥ 10 km² drainage area. RivMapper uses the HYRIV_ID, NEXT_DOWN, LENGTH_KM, and UPLAND_SKM attributes for downstream tracing and reach-length scaling.

  • Source: HydroSHEDS, https://www.hydrosheds.org/products/hydrorivers

  • Citation: Lehner, B., & Grill, G. (2013). Global river hydrography and network routing: Baseline data and new approaches to study the world’s large river systems. Hydrological Processes, 27(15), 2171–2186. https://doi.org/10.1002/hyp.9740

  • Files needed: one zone per continent your study rivers fall in. Zone codes: af Africa, ar Arctic (incl. Alaska), as Asia, au Australasia, eu Europe, na North America, sa South America, si Siberia. Each download is a .zip containing a single HydroRIVERS_v10_<zone>.shp plus its sidecars.

  • Layout:

    <wd>/HydroATLAS/HydroRIVERS/HydroRIVERS_v10_<zone>_shp/
    HydroRIVERS_v10_<zone>.shp
    HydroRIVERS_v10_<zone>.shx
    HydroRIVERS_v10_<zone>.dbf
    HydroRIVERS_v10_<zone>.prj

  • Size: ~50 MB to ~1.3 GB per zone depending on continent.

  • Master CSV column: hydroatlas_zone (one of the codes above).

Nyberg global channel belts

Per-tile rasters classifying each pixel by channel-belt confidence (1 = no data, 2-6 = belt). RivMapper thresholds values 2-6 as "channel belt".

  • Source: Nyberg et al. 2023 supplemental data; download URLs in the paper's data availability statement.
  • Citation: Nyberg, B., Henstra, G., Gawthorpe, R.L., Ravnås, R., Ahokas, J., 2023. Global scale analysis on the extent of river channel belts. Nat Commun 14, 2163. https://doi.org/10.1038/s41467-023-37852-8
  • Files needed: the per-tile GeoTIFFs (tifs/) plus the GRM tile-index shapefile that maps tile filenames onto polygons.
  • Layout: 
    <wd>/ChannelBelts/
    tifs/<tile_id>.tif        ... many tiles ...
    GRM_index.shp             (with attribute "filename" matching tile_id)
  • Per-river inputs (user-drawn). For each river you want to process, draw a polygon outlining the channel-belt area you want extracted (in QGIS/ArcGIS) and save it as 
    <wd>/ChannelBelts/Extracted_ChannelBelts/<river>/<river>_outline.shp
    This polygon is what channel_belt clips to before thresholding.

GRWL + IMWtiles

The Global River Widths from Landsat (GRWL) vector dataset partitions the world into ~1° tiles. RivMapper uses GRWL to compute mean channel widths for clean_masks's GRWL-buffer cleaning step.

  • GRWL source: https://zenodo.org/record/1297434 (Allen & Pavelsky 2018).
  • Citation: Allen, G.H., Pavelsky, T.M., 2018. Global extent of rivers and streams. Science 361, 585-588. https://doi.org/10.1126/science.aat0636
  • Files needed: every .shp whose tile ID intersects any reach you process. The IMWtiles index ships in shapefile_data/IMWtiles.shp in this repository and is what create_masks queries to discover which GRWL tiles to load.
  • Layout: 
    <wd>/Allen_Pavelsky_2018/GRWL_vector_V01.01/GRWL_vector_V01.01/
    <TILE_ID>.shp           ... per intersecting tile ...

Google Earth Engine assets

create_masks and hydraulic_geometry pull data from Earth Engine at run time; nothing is stored locally on your machine.

  • Authentication: run earthengine authenticate once. Newer earthengine-api versions require a Cloud project; pass it through with the --ee-project CLI flag where exposed.
  • Image collections used:
    • LANDSAT/LT05/C02/T1_L2, LANDSAT/LE07/C02/T1_L2, LANDSAT/LC08/C02/T1_L2, LANDSAT/LC09/C02/T1_L2 (surface reflectance Collection 2 Tier 1, 1984-present)
    • NASA/NASADEM_HGT/001 (NASADEM, latitudes < 60°)
    • UMN/PGC/ArcticDEM/V3/2m_mosaic (ArcticDEM, latitudes ≥ 60°)
  • Citation: Gorelick, N., Hancher, M., Dixon, M., Ilyushchenko, S., Thau, D., & Moore, R. (2017). Google Earth Engine: Planetary-scale geospatial analysis for everyone. Remote Sensing of Environment, Big Remotely Sensed Data: Tools, Applications and Experiences, 202, 18–27. https://doi.org/10.1016/j.rse.2017.06.031

GQBF (Liu et al. 2024)

Per-reach bankfull-discharge polylines for every river ≥ 1 km² drainage area, distributed as one geopackage per continent.

  • Source: Zenodo, https://doi.org/10.5281/zenodo.13855371
  • Citation: Liu, Y., Wortmann, M., Slater, L., 2024. Global River BankFull Discharge (GQBF) v0.1.
  • Files needed: geopackage(s) for the continent(s) your rivers fall in. Continent codes af/as/au/eu/na/sa. RivMapper auto-maps HydroATLAS Arctic zone (ar) to GQBF North America (na).
  • Layout: 
    <wd>/GQBF/GQBFv0.1_reaches_<continent>_EPSG4326.gpkg
  • Used columns: reach_id, upstream_line_ids, downstream_line_ids, qbf (m³/s), length (m), grwl_width_median (m). RivMapper internally renames the long column names to fit shapefile constraints.

WBMsed Annual Averages (Cohen et al. 2022)

Global gridded sediment-flux model output. RivMapper takes the per-pixel maximum within each reach polygon (not the mean) to capture the mainstem channel pixel, then smooths the resulting per-reach series with a degree-2 polynomial fit to filter values inherited from adjacent rivers.

  • Source: Cohen et al. 2022 supplement
  • Citations:
    • Cohen, S., Kettner, A.J., Syvitski, J.P.M., Fekete, B.M., 2013. WBMsed, a distributed global-scale riverine sediment flux model: Model description and validation. Computers & Geosciences 53, 80-93. https://doi.org/10.1016/j.cageo.2011.08.011
    • Cohen, S., Syvitski, J., Ashley, T., Lammers, R., Fekete, B., Li, H.-Y., 2022. Spatial Trends and Drivers of Bedload and Suspended Sediment Fluxes in Global Rivers. WRR 58, e2021WR031583. https://doi.org/10.1029/2021WR031583
  • Files needed: annual-average rasters in ESRI ASCII grid format covering bedload flux, suspended-bed flux, washload flux, total flux (kg/s), discharge (m³/s), and mean particle size (m). The exact file name set varies by WBMsed export; extract_wbmsed detects which column is which by keyword (flux, suspend, wash, size, discharge) rather than hard-coding names.
  • Layout: 
    <wd>/WBMsed/Averages/*.asc
  • Master CSV column: sediment_bulkdensity_kg_m3 (used to convert mass flux to volumetric flux in transit_length); optional washload_proportion (default 1.0).

BASED depth model (Gearon et al. 2024)

XGBoost regression model for bankfull depth from (width, slope, discharge), trained on a global compilation of bankfull surveys.

  • Source: https://github.com/jameshgrn/based_api
  • Citation: Gearon, J. H., Martin, H. K., DeLisle, C., Barefoot, E. A., Mohrig, D., Paola, C., & Edmonds, D. A. (2024). Rules of river avulsion change downstream. Nature, 634(8032), 91–95. https://doi.org/10.1038/s41586-024-07964-2
  • File needed: the .ubj weights from the based-api repository.
  • Layout: 
    <wd>/Gearon_etal_2024/based-api/based_us_sans_trampush_early_stopping_combat_overfitting.ubj
  • Optional. If the model file is missing, hydraulic_geometry logs a warning per river and writes NaN into the BASED_depth_m column; every other metric still computes.

Per-river user inputs

Two RivMapper modules need a hand-drawn input from the user, alongside the public datasets above:

  1. channel_belt needs a polygon outlining the area to clip the Nyberg raster to, at:

    <wd>/ChannelBelts/Extracted_ChannelBelts/<river>/<river>_outline.shp

    Draw it generously — the script will keep only the largest connected channel-belt component inside the polygon.

  2. extract_hydrorivers needs the upstream and (optionally) downstream HYRIV_ID values that bracket your river of interest. Look them up in QGIS by clicking on the appropriate HydroRIVERS segments and reading the HYRIV_ID attribute. These go in the us_hyriv_id and ds_hyriv_id columns of the master CSV.

The reach polygons (<wd>/RiverMapping/Reaches/<river>/<river>.shp) and the per-year main_channel.shp files used by clean_masks are generated by RivMapper itself in steps 2 and 5 — you do not need to draw those by hand.

Reproducibility notes

When publishing results, record:

  • The HydroRIVERS, GRWL, GQBF, Nyberg, WBMsed, and BASED versions you used (the dataset names above are versioned where applicable).
  • The Earth Engine image collection IDs RivMapper resolved at run time (these are listed in the source above and reproduced in the Landsat composite section of create_masks.py).
  • Your Python environment, ideally captured as pip freeze or a conda environment file alongside your master CSV.

These together with the master CSV are sufficient to reproduce a RivMapper run.