To go along with the ani1x and ani2x datasets, it would be good to also include aimnet2.
One problem is the size of aimnet2 when writing to our hdf5 format. Our format is designed to work best with the idea that a single record (i.e., system) would have many unique configurations associated with, rather than many unique systems, with fewer configurations each. In the latter case, writing files becomes prohibitively slow, as it basically writes many small arrays; it also creates an excessive amount of metadata for each being written (this goes back to a prior issue where compression numpy arrays inside of the hdf5 file was less effective, and sometimes worse than uncompressed...and compressing the entire hdf5 file itself was often better).
To be able to support aimnet2 (and speed up ani2x writing as well and reduce file size), we need to support grouping of records by number of atoms. This would result in fewer large arrays to write to the hdf5 file speeding up performance and reducing file size.
I think the easiest solution is to create a class (RecordGroup) that is a child of Record in the curate module. RecordGroup can essentially use all the same functionality to stack the arrays in the underlying properties (metadata would do this via lists, as it is more freeform). The new child class would need some additional book keeping for each group to know how to breakup the grouped records. We'd also need to add in another format for atomic numbers that is like atomic_numbers_grouped so we can properly validate the input to the pydantic class. Grouping records has some draw backs if there is a lot of additional metadata in the file (as some of it may not be able to be written easily to the hdf5 file), but that should be an issue for aimnet2 and ani datasets.
the HDF5Dataset module that reads in the datafiles would need to be reworked a little, however, it should be relatively straightforward to basically add in a loop where we extract individual records before we hit all the processing and validation.
To go along with the ani1x and ani2x datasets, it would be good to also include aimnet2.
One problem is the size of aimnet2 when writing to our hdf5 format. Our format is designed to work best with the idea that a single record (i.e., system) would have many unique configurations associated with, rather than many unique systems, with fewer configurations each. In the latter case, writing files becomes prohibitively slow, as it basically writes many small arrays; it also creates an excessive amount of metadata for each being written (this goes back to a prior issue where compression numpy arrays inside of the hdf5 file was less effective, and sometimes worse than uncompressed...and compressing the entire hdf5 file itself was often better).
To be able to support aimnet2 (and speed up ani2x writing as well and reduce file size), we need to support grouping of records by number of atoms. This would result in fewer large arrays to write to the hdf5 file speeding up performance and reducing file size.
I think the easiest solution is to create a class (
RecordGroup) that is a child ofRecordin the curate module.RecordGroupcan essentially use all the same functionality to stack the arrays in the underlying properties (metadata would do this via lists, as it is more freeform). The new child class would need some additional book keeping for each group to know how to breakup the grouped records. We'd also need to add in another format for atomic numbers that is likeatomic_numbers_groupedso we can properly validate the input to the pydantic class. Grouping records has some draw backs if there is a lot of additional metadata in the file (as some of it may not be able to be written easily to the hdf5 file), but that should be an issue for aimnet2 and ani datasets.the HDF5Dataset module that reads in the datafiles would need to be reworked a little, however, it should be relatively straightforward to basically add in a loop where we extract individual records before we hit all the processing and validation.