Switch to libdeflate for performance

[kylebarron]

libdeflate is expected to be significantly faster than flate2

This will require a refactor internally so that the decoder can compute the output array size. The output array size should be able to be computed from tile height, width, bits per sample and number of samples.

cc @weiji14 , what compressions did you test when you tested async-tiff as slower than gdal/libtiff?

[weiji14]

I was benchmarking on LZW compression with predictor=2, and against LiberTIFF (I think libtiff is faster, but less thread-safe).

To look up SIMD code in GDAL, search for SSE2 or AVX2. It's a little tricky to find, but this commit seems the most useful I've found. It tested:

• Float32, LZW-compressed, Predictor=3
• Byte, ZSTD-compressed, Predictor=2

If benchmarking, you'll probably need to separate out the predictive codec part from the entropy codec (zstd, deflate, etc), See https://element84.com/software-engineering/beyond-the-default-a-modern-guide-to-raster-compression by @jkeifer for some really nice details.

[kylebarron]

I don't think we should add custom SIMD code into async-tiff just yet. I think there are lower hanging fruit, like choosing faster decompressors, etc.

[weiji14]

Yeah, I don't think we should add SIMD stuff yet either (I'm putting off implementing too much hardware specific stuff until https://wheelnext.dev/proposals/pep817_wheel_variant_support/ gains more traction). Mostly I'm worried about how to keep the Decoder trait future-proof.

[kylebarron]

The decoder is going to have to be modified anyways #203. And I think I may try to emulate something like https://github.com/developmentseed/deck.gl-raster/blob/93cb2ce32530d35325693e2ed7de72eadacb3b60/packages/geotiff/src/decode.ts, where

• The decoder can return typed data, not just a raw buffer
• We have a set of varied input tags as input. Here I was considering passing all of Tile into the decoder. Or maybe Tile should be defined as holding just x, y, buffers, and DecoderMetadata, and then all of DecoderMetadata gets passed to decode, for each of buffers