Missing metadata refactor

[colinvwood]

Refactors how we support alternative vocabularies for missing data. No longer encodes these in np.nans and instead keeps a "sidecar" series (or dataframe) to store the alternative representations of missing values.

[colinvwood]

Hey @ebolyen, is something like this what you had in mind?

[ebolyen]

Hey @colinvwood, yep this is the basic idea I think! Are you ready for a review? Another thing we can do is extend the pyproject toml to include pandas 3.x and UV should pick it up in its matrix.

[colinvwood]

Not quite, I want to add some tests that are aware of the missing masks and then it will be. Re pandas 3.x, I'm not sure how to do that--would that be a new dependency-groups entry?

[ebolyen]

Just change this line to be < 3.1 instead of 3.0: https://github.com/rachis-org/rachis/blob/dev/pyproject.toml#L17

[colinvwood]

Looks like there's some work to be done before pandas 3 becomes a reality

[colinvwood]

Should be ready for review now. Do you have time @ebolyen or should I bug Anthony?

[colinvwood]

@ebolyen I'm going to work through the pandas 3 errors now.

A big question from the get-go: do we want the canonical type of CategoricalMetadataColumns to become pd.StringDtype (str) instead of object going forward? This better matches what we say are the accepted values anyway--strings or missing values. The downside is that to_dataframe() and to_series() would then return a different type, not sure if this is public enough to be considered a problem.

Either way--whether we stick with object or switch to str--the decision we make is going to become more engrained because many tests will have to be updated to force a specific dtype for hand-written (i.e. expected) dataframes. We'll be spanning pandas 2 & 3 and pandas 3 uses the str type as default. This means that either way tests will complain under one of the supported versions.

[ebolyen]

There is also CategoricalDtype which may be the most correct of the new types.

I think we should probably switch away from object, it's inevitable that we'll need to do this at some point I think.

[colinvwood]

I chose to move forward with dtype="str" rather than pd.CategoricalDtype for a few reasons:

• pd.CategoricalDtype requires defined categories which would be completely new to the codebase and would require constant maintenance (e.g. after filtering what do we do with non-observed categories)
• series of pd.CategoricalDtype can only be mutated if the new values are in the categories, this could be a headache when using returns from to_series, to_dataframe
• pd.CategoricalDtype still requires typed categories, we would make these str anyway, and at that point pd.CategoricalDtype just feels like a headache wrapper around what we want

Also, I think I was confused earlier when I wrote about supporting pandas 2.x and 3.x simultaneously--this won't actually be the case right? We'll pin to >3.x in the next release and that'll be that? All tests pass for pandas 3.x.

[ebolyen]

It would be nice to support 2.3 and 3.0 (https://scientific-python.org/specs/spec-0000/) if it was possible (since it is a major version, I do wonder how sensible this even is). The actual release will only have one pandas version, but as a library Rachis can be a bit more flexible where it makes sense.

[colinvwood]

Okay it works with 2.3 as well now, I also made 2.3 the lowest supported version instead of 2.2.

[lizgehret]

Just chatted with @ebolyen about this, we'll get this reviewed shortly!