ADIOS I/O

[BenWibking]

ORNL has optimized ADIOS2 for its systems and vice versa. In particular, Frontier is optimized for file-per-process I/O and ADIOS2 can do this with a single configuration variable change (with its BP5 backend: https://adios2.readthedocs.io/en/v2.9.1/engines/engines.html#bp5).

See also: https://www.exascaleproject.org/wp-content/uploads/2021/02/ADIOS_tutorial_ECP_AHM_Apr2021_full.pdf

With the BP5 backend, it is also significantly faster than either HDF5 and binary MPI I/O (e.g., https://www.scd.stfc.ac.uk/SiteAssets/Pages/CIUK-2021-Poster-Competition/POSTER_Shrey_Bhardwaj.pdf)

ADIOS also supports directly passing GPU buffers to its APIs, while HDF5 does not: https://adios2.readthedocs.io/en/v2.9.1/advanced/gpu_aware.html

[Yurlungur]

How portable is ADIOS as far as data analysis tooling? Is it easy to install, e.g., a Python reader?

[BenWibking]

I don't think it's as easy to install as pip install h5py, unfortunately. It does have a Spack package, and it seems to usually work.

On the other hand, it does come with its own Python API included: https://adios2.readthedocs.io/en/v2.9.1/api_high/api_high.html#python-high-level-api

[Yurlungur]

Hmm my experience with spack for python packages is very poor. If you use spack for python, you better throw away your entire other python stack and use spack for everything. If you let things mix, it's painful.

Though if the setup.py build/install process works ok it. might be fine.

[BenWibking]

And Paraview and VisIt support with an XML metadata file (not sure if it's identical to the XDMF metadata format, but it looks very similar): https://adios2.readthedocs.io/en/v2.9.1/ecosystem/visualization.html

[BenWibking]

Hmm my experience with spack for python packages is very poor. If you use spack for python, you better throw away your entire other python stack and use spack for everything. If you let things mix, it's painful.

Though if the setup.py build/install process works ok it. might be fine.

Yeah, I've experienced this...

For a different can of worms, there's a Conda package: https://adios2.readthedocs.io/en/v2.9.1/setting_up/setting_up.html#conda

[BenWibking]

The main motivation for this is the extremely poor parallel HDF5 performance we've experienced on Frontier (it just can't write compressed/chunked datasets at scale -- at least, without a ridiculous amount of tuning).

[BenWibking]

This is a nice wrapper around the ADIOS API: https://github.com/openPMD/openPMD-api
See examples: https://openpmd-api.readthedocs.io/en/latest/usage/firstwrite.html

[BenWibking]

All of the existing codes that use openPMD use 1 var per level and sparsely fill it (natively supported by ADIOS).

Will that work for Parthenon? We have to convert to a global index space, but I don't immediately see any reasons why we couldn't do that. Then we would have to add custom attributes to store the block metadata.

[BenWibking]

@pgrete It turns out that openPMD does not really support ghost cells (you can write them, but the analysis tools that read openPMD won't mask them out -- see openPMD/openPMD-api#1044).

Is there a way to allocate new MeshBlocks without ghost cells just for the I/O?

[forrestglines]

Is there a way to allocate new MeshBlocks without ghost cells just for the I/O?

I think creating a variable with metadata flags Independent and OneCopy is what you want. However, I'm not sure that keeping that data allocated all the time is what we need. I think your suggestion in another chat to just allocate another view would work.

Ideally, I think we'd like to pass in a Kokkos subview to openPMD that excludes ghost zones. I have no idea how openPMD would treat a subview though.