How do I encode NaN values in xarray / zarr with integer dtype?

Question

I have a large xarray DataArray containing NaNs and want to save it with zarr. I want to minimize the file size and am OK with losing a few bits of precision - 16 bits ought to be OK. I tried using numcodecs.FixedScaleOffset(astype='u2') filter but this stores all NaNs as zero. Since the data also contains zeros as valid values, this is not very helpful.

Answer 1

NumPy's u2 (a.k.a. uint16) does not support NaN values (please this SO answer). Zarr is merely reflecting NumPy's behavior.

Answer 2

It doesn't work with numcodecs.Quantize, but the xarray encoding parameters can specify _FillValue:

dataset.to_zarr(store, encoding={'<array-name>': {'dtype': 'uint16', '_FillValue': 65535}})

See https://xarray.pydata.org/en/stable/io.html#writing-encoded-data