Skip to content

Preserve masked cells when merging multivariate fields - #167

Open
antonsoo wants to merge 1 commit into
ajdawson:mainfrom
antonsoo:fix/preserve-multivariate-masks
Open

antonsoo wants to merge 1 commit into
ajdawson:mainfrom
antonsoo:fix/preserve-multivariate-masks

Conversation

@antonsoo

Copy link
Copy Markdown

MultivariateEof documents support for masked arrays, but np.concatenate discards their masks when merging fields. Changing values underneath a mask then changes the eigenvalues and includes excluded cells in the returned EOFs:

import numpy as np
from eofs.multivariate.standard import MultivariateEof

x = np.ma.array(np.arange(36, dtype=float).reshape(12, 3), mask=False)
x.mask[:, 1] = True
y = np.sin(np.arange(48)).reshape(12, 4)
before = MultivariateEof([x, y]).eigenvalues()
x.data[:, 1] = 1000 * np.arange(12)**2
after = MultivariateEof([x, y]).eigenvalues()
np.testing.assert_allclose(before, after)

Use np.ma.concatenate when a field is masked so the existing solver excludes those cells. Promote masked integral merged arrays to float for the solver's NaN representation. This also preserves masks in projectField; ordinary array concatenation stays unchanged.

Seven regressions cover mixed/both masked fields, weighting, projections and integral inputs. The full suite passes on Python 3.12 with Iris/xarray/Dask and the core suite passes on 3.8. Independent valid-cell SciPy SVD checks on the bundled SST data cover eigenvalues, projections, reconstructions and masks through the standard and Iris interfaces. Previously correct complete outputs remain exact, including additional float32/float64 and Dask controls. A clean installed wheel repeats the results; composition with #166 preserves its covariance workloads.

NumPy concatenate discards masks from MaskedArray inputs, allowing hidden cells to alter the covariance modes and unmasking the returned spatial fields. Use masked concatenation when any field is masked. Promote integral masked fields so the downstream NaN representation can preserve their missing cells. Keep ordinary ndarray and Dask concatenation unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant