Zarr Extensions

Appearance

Scale_Offset codec ​

Defines an array -> array codec that subtracts an offset, then multiplies by a scaling factor. This transformation is commonly used to map floating-point values into the support range of an integer data type, which achieves a lossy form of compression.

This codec is only defined for data types where multiplication, division, addition, and subtraction are well-defined. The following data types defined in this repository are supported:

  • Integer types: int2, int4, int8, int16, int32, int64
  • Unsigned integer types: uint2, uint4, uint8, uint16, uint32, uint64
  • Floating-point types: float4_e2m1fn, float6_e2m3fn, float6_e3m2fn, float8_e3m4, float8_e4m3, float8_e4m3b11fnuz, float8_e4m3fnuz, float8_e5m2, float8_e5m2fnuz, float8_e8m0fnu, bfloat16, float16, float32, float64

Codec metadata ​

This codec is declared in metadata as a JSON object with the following structure:

FieldTypeRequired
namestringyes
configurationobjectno

Name ​

The value of the name field MUST be the string "scale_offset".

Configuration ​

The value of the configuration field is a JSON object with the following structure:

FieldTypeRequired
offsetJSON-encoded scalarno
scaleJSON-encoded scalarno

Additional keys are reserved for future versions of this codec. Metadata with additional keys MUST be treated as invalid by readers.

If neither scale nor offset are specified, the entire configuration field may be omitted. In this case the codec is effectively a no-op.

offset ​

The value of the offset field is a scalar that defines the quantity subtracted from input values during encoding. If present, this value MUST be encoded to JSON using the Zarr V3 fill value encoding for the input array's data type. A missing offset field encodes the additive identity element for the data type, e.g., 0 for numeric data types.

scale ​

The value of the scale field is a scalar that defines the quantity multiplied with input values during encoding, after offset has been subtracted. If present, this value MUST be encoded to JSON using the Zarr V3 fill value encoding for the input array's data type. A missing scale field encodes the multiplicative identity element for the data type, e.g., 1 for numeric data types.

Algorithm ​

The encoding and decoding transformations MUST be performed using the arithmetic semantics of the input array's data type. If any intermediate or final value produced during encoding or decoding is not representable in that data type, implementations MUST treat this as an error.

Encoding ​

Given an input array in, an output array out is generated by the following routine:

out = (in - offset) * scale

The fill value of out is generated by applying the transformation above to the fill value of in.

Decoding ​

The decoding routine inverts the encoding routine, i.e.:

out = (in / scale) + offset

The fill value of out is generated by applying the transformation above to the fill value of in.

Examples ​

Float32 with scale and offset ​

The following snippet of array metadata demonstrates the metadata for the scale_offset codec applied to a float32 array.

json
{
    "data_type": "float32",
    "codecs": [
        {
            "name": "scale_offset",
            "configuration": {
                "offset": 5,
                "scale": 0.1
            }
        },
        "bytes"
    ]
}

Uint16 range reduction ​

In this example, a uint16 array with values in the range [1000, 1255] is shifted down by 1000 so that values fall in the range [0, 255], then cast to uint8 via the cast_value codec.

json
{
    "data_type": "uint16",
    "codecs": [
        {
            "name": "scale_offset",
            "configuration": {
                "offset": 1000
            }
        },
        {
            "name": "cast_value",
            "configuration": {
                "data_type": "uint8"
            }
        },
        "bytes"
    ]
}

Float64 to uint8 with NaN preservation ​

In this example, a float64 array with values in the range [0.0, 2540.0] and a fill value of NaN is stored as uint8. The scale_offset codec maps values from [0.0, 2540.0] to [1.0, 255.0] by applying (x - offset) * scale with offset = -10 and scale = 0.1, reserving 0 for NaN. The cast_value codec then casts to uint8, using scalar_map to explicitly map NaN to 0 on encode and 0 to NaN on decode. This ensures a lossless round-trip for the fill value. There is no such assurance for the array values, however.

json
{
    "data_type": "float64",
    "fill_value": "NaN",
    "codecs": [
        {
            "name": "scale_offset",
            "configuration": {
                "offset": -10,
                "scale": 0.1
            }
        },
        {
            "name": "cast_value",
            "configuration": {
                "data_type": "uint8",
                "rounding": "nearest-even",
                "scalar_map": {
                    "encode": [["NaN", 0]],
                    "decode": [[0, "NaN"]]
                }
            }
        },
        "bytes"
    ]
}

Implementation notes ​

Data type casting ​

To achieve data compression, this codec should be followed by an array->array codec that converts arrays to a narrower data type, e.g. the cast_value codec.

Fill value handling ​

Implementations MUST transform the fill value as it propagates through the codec chain, using the same semantics as the element transformation. This ensures that the fill value remains aligned with the data for subsequent fill-value-aware codecs, such as sharding_indexed.

Legacy numcodecs.fixedscaleoffset codec ​

The scale_offset codec described here is based on a Zarr V3 codec called numcodecs.fixedscaleoffset, which was defined operationally as a direct translation of the Zarr-V2-compatible fixedscaleoffset codec into Zarr V3. That fixedscaleoffset codec was itself defined operationally by an implementation (numcodecs.FixedScaleOffset) in the Numcodecs library.

In conjunction with the cast_value codec, the scale_offset codec supersedes the numcodecs.fixedscaleoffset codec. Authors of implementations of the scale_offset codec may wish to support the numcodecs.fixedscaleoffset codec, e.g. by treating it as a read-only alias for a [scale_offset, cast_value] codec sequence. The numcodecs.fixedscaleoffset codec will be defined here, with instructions for mapping the semantics of that codec to the scale_offset codec.

Codec metadata ​

FieldTypeRequired
namestringyes
configurationobjectyes

Name ​

The value of the name field MUST be the string "numcodecs.fixedscaleoffset".

Configuration ​

The value of the configuration field is a JSON object with the following structure:

FieldTypeRequired
offsetnumberyes
scalenumberyes
astypeZarr V2 data type metadatayes
dtypeZarr V2 data type metadatayes

offset ​

The value of the offset field is a scalar that defines the quantity subtracted from input values during encoding. This value MUST be a JSON number.

scale ​

The value of the scale field is a scalar that defines the quantity multiplied with input values during encoding. This value MUST be a JSON number.

astype ​

The value of the astype field defines the data type of the array returned by the encoding procedure, and the data type of the array provided as input for the decoding procedure. This value MUST be Zarr V2 data type metadata.

This field was removed from the scale_offset codec in favor of expressing data type conversion via a dedicated codec.

dtype ​

The value of the dtype field defines the data type of the array given as input to the encoding procedure, and the data type of the array returned by the decoding procedure. This value MUST be Zarr V2 data type metadata.

Note that in Zarr V3, a dtype field is not needed because the data type of the input to an array-array codec is determined by its location in the codecs metadata. Zarr V2 chunk encoding did not have such guarantees, and so the dtype field was necessary for the fixedscaleoffset codec implemented in Numcodecs.

Algorithm ​

This section records a pseudocode summary of the behavior of the numcodecs.FixedScaleOffset codec.

Encoding ​

Given an input array I, an output array O with data type astype is generated by the following procedure:

arr = I - offset
arr = arr * scale
arr = round(arr) # round to nearest integer
O = cast(arr, astype) # Cast using NumPy casting rules

Decoding ​

Given an input array I, an output array O with data type dtype is generated by the following procedure, which inverts the encoding procedure:

arr = cast(I, dtype) # Cast using NumPy casting rules
arr = arr / scale
O = arr + offset

Conversion to scale_offset ​

A numcodecs.fixedscaleoffset codec does not correspond to a single scale_offset codec, but rather a sequence of codecs that includes the scale_offset codec. To generate this sequence of codecs, take the following steps:

  • Configure a scale_offset codec using the scale and offset parameters from the numcodecs.fixedscaleoffset codec. This is the first codec in the sequence.
  • Use the astype field to configure a codec that performs data type casting, e.g., the cast_value codec. For integer data types, the numcodecs.fixedscaleoffset implementation handled numerical overflow by wrapping around the maximal value of the target data type, so ensure that the data type casting codec can express the same behavior. E.g., for the cast_value codec, set the out_of_range configuration field to "wrap". This codec should follow the scale_offset codec in the sequence of codecs.

Conversion example ​

This numcodecs.fixedscaleoffset codec:

json
{
    "name": "numcodecs.fixedscaleoffset", 
    "configuration": {
        "astype": "uint8",
        "dtype": "float32",
        "scale": 0.1,
        "offset": 10
        }
}

Would be transformed to the following codecs:

json
[
    {
        "name": "scale_offset",
        "configuration": {
            "scale": 0.1,
            "offset": 10
            }
    },
    {
        "name": "cast_value",
        "configuration": {
            "data_type": "uint8",
            "out_of_range": "wrap"
            }
    }
]

These codecs would transform a float32 array with a fill_value of 10.0 and values from the range [10.0, 100.0] into a uint8 array with a fill_value of 0 and values in the range [0, 10].

References ​

The scale_offset codec is based on the FixedScaleOffset class defined in the Numcodecs library. See https://numcodecs.readthedocs.io/en/stable/filter/fixedscaleoffset.html for more information.

Change log ​

No changes yet.

Current maintainers ​