Skip to content

index_footprints WKB screen: 1.8 GB shapely peak at clone scale #429

Description

@espg

Follow-up from #400 (question 10, flagged not chosen). index_footprints decodes the WKB geometry column with shapely purely to reproduce granule_records' empty/non-polygonal row screen. Wall is negligible (<1% of the cover) but it is the pass's memory peak: on the 555,867-row clone, 835 MB after the parquet read → 1,169 MB after to_numpy → 1,794 MB with shapely objects live (~960 MB for a type check; the all-True short-circuit already recovers half). Options from #400: (a) read the WKB type code out of the Arrow buffer directly — vectorized, no GEOS, but a second implementation of a screen that must not drift from granule_records; (b) feed the whole column to mortie first and fall back to the screen only on refusal. Decide and implement one.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions