Support for Matlab HDF5 files?

[lauralindzey]

I am writing a QGIS plugin, and am trying to switch from using h5py to (vendored) pyfive since I have found that the biggest obstacle to user adoption is just getting h5py installed correctly.

One of the datasets that I need to support provides their data in .mat files; for earlier versions, I use scipy.io for them, but for later versions, that library recommends using a HDF5 reader.

The data can be downloaded from:
https://data.cresis.ku.edu/data/rds/2022_Antarctica_BaslerMKB/CSARP_standard/20230106_01/

I tested with the Data_20230106_01_014.mat file

I then try to read it using:

import pyfive
filepath = "/path/to/datadir/Data_20230106_01_014.mat"
data = pyfive.File(filepath)

and I get the error:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Users/lindzey/Documents/QIceRadar/code/qiceradar_plugin/external/pyfive/pyfive/high_level.py", line 300, in __init__
    self._superblock = SuperBlock(self._fh, 0)
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lindzey/Documents/QIceRadar/code/qiceradar_plugin/external/pyfive/pyfive/misc_low_level.py", line 40, in __init__
    raise NotImplementedError(
NotImplementedError: unsupported superblock version: 46

By contrast, this works:

import h5py
data = h5py.File(filepath)

The issue seems to be that Matlab-generated HDF5 files have a header before the HDF5 header, but pyfive only looks for the header at offset 0.

I have a patch that is working for me, but I did not open a PR because it was generated by Claude and I think it's an ugly hack that's not how you'd actually want to solve this problem in your codebase, and I don't have the bandwidth to figure out the right/clean way and shepherd it through review.

[bnlawrence]

Thanks for this. You've uncovered a real HDF5 failure for us: I didn't know that HDF5 allows a user block in front of the superblock. The right answer for us is to look for superblocks at a bunch of standard offsets, not just at offset zero. We then need to rebase all offsets from that. This looks relatively straightforward to do. Will have this in the next release (which because a key person is on leave, is likely to be mid September).