Current section
Files
Jump to
Current section
Files
docs/codec-review.md
# ExCodecs Codec Review
A detailed technical review of each compression codec available in ExCodecs,
covering history, design goals, performance characteristics, ecosystem
adoption, licensing, Rust ecosystem maturity, and tradeoffs.
## Table of Contents
- [Zstandard (Zstd)](#zstandard-zstd)
- [LZ4](#lz4)
- [Snappy](#snappy)
- [Bzip2](#bzip2)
- [Blosc2](#blosc2)
- [Comparison Table](#comparison-table)
- [Decision Guide](#decision-guide)
---
## Zstandard (Zstd)
### History
Zstandard was created by Yann Collet at Facebook (now Meta) and first released
in 2016. Collet is also the creator of LZ4. Zstd was developed to provide a
compression algorithm that offered both high ratios and high speeds, filling
the gap between LZ4 (fast but lower ratio) and zlib/bzip2 (high ratio but
slow). Facebook deployed Zstd at scale across their infrastructure, and it has
since become one of the most widely adopted modern compression algorithms.
### Design Goals
Zstd was designed around four principles:
1. **High ratio at reasonable speed**: Zstd should compress better than zlib
at comparable or faster speeds, and decompress significantly faster than
zlib across all levels.
2. **Configurable tradeoff**: The 22 compression levels allow users to choose
any point on the speed/ratio curve, from near-LZ4 speeds to near-bzip2
ratios.
3. **Fast decompression**: Regardless of compression level, decompression
speed remains consistently high. This is because decompression is a
simpler, more predictable operation that benefits from optimized rouiting.
4. **Dictionary compression**: Zstd supports training dictionaries for
compressing many small, structurally similar payloads (e.g., JSON API
responses) where traditional compression performs poorly.
### Compression Ratio
Zstd achieves strong compression ratios across its level range:
| Level | Ratio (Silesia corpus) | Compression Speed | Decompression Speed |
|-------|------------------------|--------------------|---------------------|
| 1 | ~2.8x | ~520 MB/s | ~1300 MB/s |
| 3 | ~3.0x (default) | ~430 MB/s | ~1300 MB/s |
| 9 | ~3.4x | ~140 MB/s | ~1300 MB/s |
| 19 | ~3.7x | ~14 MB/s | ~1300 MB/s |
| 22 | ~3.8x | ~4 MB/s | ~1300 MB/s |
The decompression speed is nearly constant regardless of compression level.
At level 22, the compressed output is smaller but decompresses just as fast
as level 1 output.
### Speed Characteristics
- **Compression**: Highly variable by level. Level 1 approaches LZ4 speeds.
Level 19+ enters the territory of slow but thorough compression suitable
for archival.
- **Decompression**: Extremely fast and consistent. Typical decompression
speeds exceed 1 GB/s, making Zstd an excellent choice for read-heavy
workloads.
- **Memory**: Zstd uses a sliding window for compression (controlled by
`window_log`). Larger windows improve ratio but require more memory.
### Ecosystem Adoption
Zstd has seen massive adoption since its release:
- **Linux kernel**: Zstd is used for kernel and initramfs compression since
Linux 4.14 (2017).
- **File systems**: Btrfs, SquashFS, and F2FS support Zstd compression.
- **Web**: Chrome supports Zstd content encoding (Accept-Encoding:
zstd). CloudFlare and Facebook serve Zstd-compressed responses.
- **Archiving**: tar, rsync, and various backup tools support Zstd.
- **Data formats**: Zstd is a registered compression format in Apache
Parquet, Zarr, and several database systems.
- **GitHub**: Uses Zstd for Git object compression.
- **Docker**: Container image layers can be compressed with Zstd.
### Licensing
Zstd is dual-licensed under:
- **BSD 3-Clause** (patent grant included) -- for general use
- **GPLv2** -- for GPL-compatible projects
The dual licensing ensures broad compatibility. The BSD license is the
default and covers the vast majority of use cases. Meta has also committed
not to assert patents against Zstd users.
### Rust Ecosystem Maturity
The `zstd` crate (used by ExCodecs) is well-maintained and wraps the
reference C implementation via the `zstd-sys` crate.
| Aspect | Status |
|---------------|--------------------------------------------------|
| Crate | `zstd` v0.13 |
| Backend | C FFI to libzstd (reference implementation) |
| Streaming | Supported (encode/decode all, bulk, streaming) |
| Dictionary | Supported via `zstd::dict` |
| Maintenance | Actively maintained, follows zstd releases |
| Safety | FFI boundary is unsafe; crate provides safe API |
The use of the C reference implementation via FFI is standard practice in the
Rust ecosystem. Pure Rust Zstd implementations exist but are not mature enough
for production use. The `zstd` crate abstracts the FFI boundary with a safe
API, and Rustler further wraps it in a safe NIF interface.
### Tradeoffs
| Pro | Con |
|------------------------------------------------|------------------------------------------------|
| Excellent ratio/speed tradeoff | C dependency via zstd-sys (build complexity) |
| Fast decompression at all levels | Higher memory usage at high levels |
| Wide ecosystem adoption and tooling | NIF binary size larger than pure-Rust codecs |
| 22 compression levels for fine control | Dictionary training requires separate tooling |
| Strong IP position (BSD + patent grant) | Not the fastest compress (LZ4 is faster) |
| Active development and optimization | |
---
## LZ4
### History
LZ4 was created by Yann Collet in 2011 as an evolution of his earlier
LZ4Ultra and FastLZ algorithms. Collet designed LZ4 to prioritize compression
and decompression speed above all else. Its design is rooted in the LZ77
family of algorithms, with aggressive optimizations for modern CPU
architectures. LZ4 rapidly gained adoption in real-time systems, log
processing, and anywhere that throughput matters more than ratio.
### Design Goals
1. **Speed above all**: LZ4 targets compression at over 1 GB/s and
decompression at multi-GB/s speeds. It achieves this through a minimal
match format and branch-free decompression loops.
2. **Simple format**: The LZ4 block format is deliberately minimal -- a
sequence of token bytes, literal lengths, match lengths, and offsets.
This makes implementations easy to verify and fast to parse.
3. **Low memory footprint**: LZ4 uses a small hash table for compression
(typically 16 KB) and requires no additional memory for decompression.
4. **Deterministic performance**: LZ4's speed is consistent and predictable,
with minimal variance across input types. There are no pathological inputs
that cause dramatic slowdowns.
### Compression Ratio
LZ4 prioritizes speed over ratio. Typical ratios on the Silesia corpus:
| Variant | Ratio | Compression Speed | Decompression Speed |
|-----------|--------|--------------------|---------------------|
| LZ4 | ~2.1x | ~700 MB/s | ~3000 MB/s |
| LZ4 HC | ~2.5x | ~40 MB/s | ~3000 MB/s |
These ratios are lower than Zstd or Bzip2, but the speed advantage is
enormous -- LZ4 decompresses 3-4x faster than Zstd and 10-30x faster than
Bzip2.
### Speed Characteristics
- **Compression**: LZ4 block compression is among the fastest general-purpose
algorithms available. LZ4 HC trades compression speed for better ratio
but remains fast to decompress.
- **Decompression**: LZ4 decompression is exceptionally fast (3+ GB/s on
modern hardware) because the format is designed for branch-free, SIMD-
friendly decompression loops.
- **Memory**: Minimal. Compression uses a configurable hash table (default
16 KB). Decompression requires only the input and output buffers.
### Ecosystem Adoption
- **Linux kernel**: LZ4 has been used for kernel and zram compression since
Linux 3.15 (2014).
- **File systems**: Btrfs, SquashFS, and F2FS support LZ4.
- **Databases**: Redis uses LZ4 for list compression. MongoDB supports LZ4
for document compression.
- **Messaging**: Apache Kafka supports LZ4 compression for topic data.
- **Networking**: OpenVPN and various VPN products use LZ4 for in-flight
compression.
- **Logging**: Various log aggregation tools use LZ4 for real-time
compression of log streams.
### Licensing
LZ4 is licensed under **BSD 2-Clause** (simplified BSD license). This is a
permissive license with no patent concerns.
### Rust Ecosystem Maturity
ExCodecs uses `lz4_flex`, a pure Rust LZ4 implementation:
| Aspect | Status |
|---------------|--------------------------------------------------|
| Crate | `lz4_flex` v0.11 |
| Backend | Pure Rust (no C FFI) |
| Format | LZ4 block and frame format |
| Safety | 100% safe Rust |
| Maintenance | Actively maintained |
The choice of `lz4_flex` over the C-based `lz4` crate was deliberate:
1. **No C dependency**: Pure Rust avoids build complexity and cross-
compilation issues. This is especially important for `rustler_precompiled`
targets.
2. **Safety**: No unsafe FFI boundary. The entire compression and
decompression path is safe Rust.
3. **Simplicity**: The pure Rust crate has fewer build dependencies and
produces smaller binaries.
4. **Performance**: `lz4_flex` achieves comparable speeds to the C reference
implementation on modern hardware.
### Tradeoffs
| Pro | Con |
|------------------------------------------------|------------------------------------------------|
| Extremely fast compression and decompression | Lower compression ratio |
| Pure Rust implementation (lz4_flex) | Not suitable for archival or storage |
| Minimal memory usage | Large inputs can produce marginal ratios |
| Deterministic, predictable speed | LZ4 HC (higher ratio) is much slower |
| Simple, well-understood format | No streaming support in ExCodecs yet |
| No C dependencies for cross-compilation | Frame format not exposed in current API |
---
## Snappy
### History
Snappy was originally created by Google under the name "Zippy" and was
renamed and open-sourced as Snappy in 2011. It was designed for internal use
in Google's infrastructure -- compressing data for Bigtable, MapReduce, and
inter-process communication. Snappy's design philosophy centers on maximum
throughput with minimal CPU overhead, targeting the use case where data is
compressed for transient transport and decompressed immediately on the other
end.
### Design Goals
1. **Maximum throughput**: Snappy targets compression speeds exceeding
500 MB/s and decompression speeds exceeding 1.5 GB/s. It sacrifices
compression ratio to achieve this.
2. **Minimal overhead**: The Snappy format adds very little metadata.
The overhead for incompressible data is minimal (typically 5-6 bytes
per 32 KB block plus the literals).
3. **Stability**: Snappy's format and behavior are deterministic. The same
input always produces the same output. This makes it suitable for
content-addressable storage where bit-exact reproduction is required.
4. **Simplicity**: Snappy has no configuration options. There is one
compression strategy, one decompression path, and one output format.
This makes it easy to implement correctly and fast to verify.
### Compression Ratio
Snappy achieves the lowest compression ratios among the ExCodecs codecs,
which is the expected tradeoff for its speed:
| Metric | Typical Value (Silesia) |
|----------------------------|-------------------------|
| Compression ratio | ~2.0x |
| Compression speed | ~500-600 MB/s |
| Decompression speed | ~1500-2000 MB/s |
For structured data (JSON, protocol buffers), Snappy typically achieves
2.0-2.5x compression. For random binary data, it may produce output larger
than the input (though the format handles this gracefully).
### Speed Characteristics
- **Compression**: Extremely fast and consistent. Snappy uses a simple hash
table and short match encoding. There are no expensive computations.
- **Decompression**: Among the fastest decompressors available. The format
is designed for SIMD-friendly, branch-predictable decompression.
- **Memory**: Very small. Compression uses a ~32 KB hash table.
Decompression is single-pass with no additional allocation beyond the
output buffer.
### Ecosystem Adoption
- **Google infrastructure**: Bigtable, MapReduce, Protocol Buffers (as an
option), and many internal Google systems.
- **Apache projects**: Hadoop, Cassandra, and various Apache databases
support Snappy compression.
- **Data formats**: The Snappy framing format is used in Parquet, ORC, and
other columnar storage formats.
- **Networking**: Various RPC frameworks support Snappy for compressing
request/response payloads.
### Licensing
Snappy is licensed under **BSD 3-Clause**. Google holds the copyright and
has made no patent claims related to Snappy.
### Rust Ecosystem Maturity
ExCodecs uses the `snap` crate:
| Aspect | Status |
|---------------|--------------------------------------------------|
| Crate | `snap` v1.1 |
| Backend | Pure Rust |
| Format | Raw and framing format supported |
| Safety | Mostly safe Rust; some unsafe for SIMD paths |
| Maintenance | Actively maintained |
The `snap` crate implements both the raw (block) format and the framing
format. ExCodecs uses only the raw format (`snap::raw::Encoder` and
`snap::raw::Decoder`), which is the simpler and faster of the two.
### Tradeoffs
| Pro | Con |
|------------------------------------------------|------------------------------------------------|
| Extremely fast compression and decompression | Lowest compression ratio in the set |
| No configuration needed (one mode) | No tunable parameters at all |
| Simple, well-tested format | Not suitable for archival or storage |
| Deterministic output (same input = same output)| May expand random/incompressible data |
| Pure Rust implementation available | Less flexible than LZ4 (no HC variant) |
| Very small memory footprint | Deprecated at Google in favor of Zstd |
**Note on Snappy's future**: Google has largely moved to Zstd for internal use.
Snappy remains widely deployed and supported, but it is effectively in
maintenance mode. New projects should consider Zstd or LZ4 instead, unless
they need Snappy for compatibility with existing data formats.
---
## Bzip2
### History
Bzip2 was created by Julian Seward in 1996 and released as open source. It
was one of the first widely available compression algorithms to use the
Burrows-Wheeler Transform (BWT), a technique discovered by Michael Burrows
and David Wheeler in 1994. Seward's insight was that BWT followed by
move-to-front coding and Huffman coding could achieve compression ratios
competitive with PPM (Prediction by Partial Matching) algorithms at much
higher speeds. Bzip2 quickly became a standard for software distribution and
archival on Unix systems.
### Design Goals
1. **High compression ratio**: Bzip2 targets compression ratios competitive
with the best available algorithms. It consistently achieves among the
highest ratios of any lossless general-purpose compressor.
2. **Recoverability**: Bzip2's block-based structure allows partial
decompression. If a compressed file is damaged, the blocks before the
damage can still be recovered.
3. **Simplicity of interface**: Bzip2 has a simple API with one primary
parameter -- block size (1-9). This makes it easy to integrate and
difficult to misconfigure.
4. **Stability**: The bzip2 format has been stable since 2000. Compressed
data from 2000 can still be decompressed today.
### Compression Ratio
Bzip2 achieves the highest compression ratios among ExCodecs' general-purpose
codecs:
| Block Size | Ratio (Silesia) | Compression Speed | Decompression Speed |
|------------|------------------|--------------------|---------------------|
| 1 | ~2.9x | ~15 MB/s | ~30 MB/s |
| 5 | ~3.2x | ~10 MB/s | ~28 MB/s |
| 9 (default)| ~3.3x | ~8 MB/s | ~25 MB/s |
These ratios are higher than Zstd at level 9-19 but come at a significant
speed cost. Bzip2 is roughly 50-100x slower at decompression than LZ4.
### Speed Characteristics
- **Compression**: Slow. Bzip2's BWT and Huffman coding passes are
computationally expensive. At block size 9, compression speeds are under
10 MB/s.
- **Decompression**: Also slow compared to modern algorithms. The BWT
inverse transform requires significant computation per block.
- **Memory**: Increases with block size. At block size 9, compression
requires approximately 8 MB of memory for the BWT workspace. This is
significantly more than LZ4 (16 KB) or Zstd (varies by level).
- **Blocking**: Bzip2 processes data in 100 KB - 900 KB blocks (controlled
by block size). This provides natural boundaries for parallel processing
(though ExCodecs does not currently expose parallel decompression).
### Ecosystem Adoption
- **Software distribution**: Many Linux distributions distribute source
tarballs as `.tar.bz2`. The kernel was historically distributed as
`.tar.bz2` before moving to Zstd.
- **Archival**: Bzip2 is widely used for long-term storage where ratio
matters more than speed.
- **Unix standard**: `bzip2` has been a standard Unix tool for decades.
Every major Linux distribution includes it.
- **Data exchange**: Some scientific data formats use bzip2 for compressing
large datasets.
### Licensing
Bzip2 uses a **BSD-like license** (based on the BSD 4-Clause license with an
additional advertising clause). The license is permissive and similar in
spirit to BSD, though the advertising clause is somewhat unusual.
### Rust Ecosystem Maturity
ExCodecs uses the `bzip2` crate:
| Aspect | Status |
|---------------|--------------------------------------------------|
| Crate | `bzip2` v0.4 |
| Backend | C FFI to libbz2 (reference implementation) |
| Streaming | Supported (BzEncoder, BzDecoder) |
| Safety | Unsafe FFI boundary; safe Rust API wrapper |
| Maintenance | Maintained, follows libbz2 releases |
The `bzip2` crate wraps the reference C implementation (`libbz2`) via FFI.
There is no mature pure Rust bzip2 implementation; the BWT and Huffman coding
are complex enough that a from-scratch Rust implementation would require
significant engineering and verification effort.
ExCodecs uses the streaming API (`bzip2::write::BzEncoder` and
`bzip2::read::BzDecoder`) even for one-shot compression/decompression.
This is because the streaming API handles the block-based nature of bzip2
correctly and allows the same code path to be extended for streaming support
in the future.
### Tradeoffs
| Pro | Con |
|------------------------------------------------|------------------------------------------------|
| Highest compression ratio (general purpose) | Very slow compression and decompression |
| Stable format (unchanged since 2000) | High memory usage at large block sizes |
| Block-based (partial recovery on corruption) | C dependency via bzip2-sys |
| Simple API (one parameter: block size) | No streaming support in ExCodecs yet |
| Widely available utility (bzip2 command) | Not suitable for real-time applications |
| Strong data integrity checks | Single-threaded (no parallel compression) |
---
## Blosc2
### History
Blosc2 was created by Francesc Alted starting in 2021, building on the
original Blosc (created in 2010). Blosc was designed as a meta-compressor
for numerical data in scientific computing, particularly for the PyTables
and bcolz projects. The "Blosc" name comes from "Blocking and Shuffling
Optimized Compression." Blosc2 extended the original format with a new
header structure, support for more internal compressors, and improved
multithreading.
Blosc2's key insight is that numerical arrays (float64, int32, etc.) compress
much better when the bytes are rearranged before compression. By transposing
an array so that similar-valued bytes are adjacent, standard compressors
like LZ4 and Zstd achieve dramatically better ratios on structured data.
### Design Goals
1. **Meta-compression**: Blosc2 is not a compression algorithm itself. It is
a framework that applies byte/bit shuffling followed by an internal
compressor (LZ4, Zstd, Snappy, BloscLZ, or zlib). The caller chooses
both the compressor and the shuffle strategy.
2. **Array-optimized**: Blosc2 is designed for data whose length is a
multiple of `typesize` -- the size of each element in the array. When
`typesize` is set correctly and shuffle is enabled, compression ratios
on numerical arrays can improve by 2-10x.
3. **Zero-overhead passthrough**: When compression produces output larger
than the input (common with small or random data), Blosc2 stores the
data uncompressed with minimal overhead -- just the 16-byte header.
4. **Multithreading**: The C-Blosc2 library supports multi-threaded
compression and decompression via a thread pool. (Note: ExCodecs'
pure Rust implementation currently uses single-threaded mode.)
5. **Self-describing format**: The Blosc2 header includes the compressor
type, compression level, shuffle mode, and typesize, allowing
decompression without external metadata.
### Compression Ratio
Blosc2's ratio depends heavily on the data type and shuffle setting:
| Data Type | Shuffle | Internal | Ratio (typical) |
|------------------------|-----------|----------|-----------------|
| Float64 array | byte | LZ4 | 4-10x |
| Float64 array | byte | Zstd | 6-15x |
| Float64 array | none | LZ4 | 1.5-3x |
| Float64 array | none | Zstd | 2-5x |
| Random binary | none | any | ~1.0x (passthrough) |
| JSON text | none | Zstd | 2-3x |
The shuffle step is what makes Blosc2 exceptional for typed arrays. Consider
an array of 64-bit floats: `[1.0, 2.0, 3.0, ...]`. In memory, this looks like:
```
Byte layout (8 bytes per float, no shuffle):
[3f f0 00 00 00 00 00 00] [40 00 00 00 00 00 00 00] [40 08 00 00 00 00 00 00]
^-- float 1.0 ----------- ^-- float 2.0 ----------- ^-- float 3.0 ----------
Byte layout (after byte shuffle, grouping bytes by position):
[3f 40 40 ...] [f0 00 08 ...] [00 00 00 ...] [00 00 00 ...] [00 00 00 ...] [00 00 00 ...] [00 00 00 ...] [00 00 00 ...]
^-- byte 0 ^-- byte 1 ^-- byte 2 ... (repeated zeros = excellent compression)
```
After shuffle, runs of similar bytes become adjacent, and compressors like
LZ4 and Zstd achieve dramatically better ratios on the run-length-encoded
zeros.
### Speed Characteristics
- **Compression**: Fast when using LZ4 or BloscLZ as the internal compressor.
The shuffle step adds some overhead but is typically cheap relative to the
compression itself.
- **Decompression**: Fast. The unshuffle step is cheap (O(n) byte
transposition), and the internal decompressor is fast.
- **Memory**: Blosc2 processes data in blocks (configurable via
`blocksize`). Smaller blocks reduce memory usage but may reduce ratio.
- **Threading**: The C-Blosc2 library supports multi-threaded compression
and decompression. ExCodecs' pure Rust implementation is currently
single-threaded, though the `numthreads` parameter is accepted for forward
compatibility.
### Ecosystem Adoption
- **Scientific computing**: Blosc2 is the primary compression format for
PyTables, bcolz, and the newer python-blosc2 library.
- **Data formats**: The HDF5 library can use Blosc2 as a compression filter.
Zarr supports Blosc2 as a compressor.
- **NumPy**: The blosc2 Python package provides NumPy-aware compression.
- **C-Blosc2**: The reference C library is used across the scientific Python
ecosystem.
### Licensing
Blosc2 is licensed under **BSD 3-Clause**.
### Rust Ecosystem
ExCodecs uses a **pure Rust implementation** of the Blosc2 format rather than
binding to the C-Blosc2 library. This was a deliberate design decision.
| Aspect | Status |
|---------------|--------------------------------------------------|
| Implementation | Pure Rust in blosc2_codec.rs |
| Internal codecs| LZ4 (lz4_flex), Zstd (zstd), Snappy (snap) |
| Shuffle | Byte shuffle/unshuffle, bit shuffle/unshuffle |
| Threading | Single-threaded (numthreads parameter accepted) |
| Coverage | Core compress/decompress, all major options |
### Why pure Rust instead of C-Blosc2 FFI?
1. **No C dependency**: C-Blosc2 depends on multiple internal libraries
(libzstd, liblz4, etc.) and has a complex build system. Binding to it
via FFI would add significant complexity to the NIF build process and
complicate `rustler_precompiled` distribution.
2. **Code reuse**: The internal compress/decompress functions already exist
in the NIF (LZ4 via `lz4_flex`, Zstd via `zstd`, Snappy via `snap`).
Blosc2's meta-compression pattern just calls these with shuffled data.
Reusing the existing Rust crates avoids code duplication.
3. **Format simplicity**: The Blosc2 header is only 16 bytes with a
well-documented structure. Parsing and constructing the header in Rust
takes approximately 100 lines, far less effort than binding C-Blosc2.
4. **Build reproducibility**: A pure Rust implementation compiles
deterministically with the same toolchain. No system libraries, no
pkg-config, no cmake.
5. **The shuffle is the value**: The primary benefit of Blosc2 is the byte
and bit shuffle. The header format and passthrough behavior are trivial
to implement. The shuffle routines are approximately 30-50 lines of Rust
each.
**Limitations of the pure Rust approach**:
- No multi-threaded compression (the `numthreads` parameter is accepted but
ignored). This can be added in a future release using Rayon or a similar
Rust parallelism library.
- The bit shuffle implementation is a simplified version that handles common
cases correctly but may differ from C-Blosc2 for edge cases.
- Some advanced C-Blosc2 features (lazy decompression, filters beyond
shuffle, frames) are not yet supported.
### Tradeoffs
| Pro | Con |
|------------------------------------------------|------------------------------------------------|
| Dramatically better ratio on typed arrays | Poor ratio on random/non-structured data |
| Self-describing format (header includes meta) | Pure Rust impl lacks multithreading |
| Supports multiple internal compressors | Not a drop-in replacement for C-Blosc2 data |
| Zero-overhead passthrough on incompressible | Bit shuffle implementation simplified |
| Pure Rust implementation (no C deps) | numthreads parameter currently ignored |
| Reuses existing Rust codec crates | Some C-Blosc2 features not yet supported |
| Excellent for numerical/array data | Requires typesize knowledge for best results |
---
## Comparison Table
| Property | Zstd | LZ4 | Snappy | Bzip2 | Blosc2 |
|-------------------|-----------------|-----------------|-----------------|-----------------|----------------------|
| **Category** | General-purpose | General-purpose | General-purpose | General-purpose | Meta-compressor |
| **Ratio** | High (2.8-3.8x) | Low (2.1x) | Low (2.0x) | Very high (3.3x)| Variable (1-15x) |
| **Compress Speed**| Fast-slow* | Very fast | Very fast | Slow | Fast (depends on internal) |
| **Decomp Speed** | Very fast | Extremely fast | Extremely fast | Slow | Fast (depends on internal) |
| **Memory (comp)** | 1-128 MB | 16 KB | 32 KB | 1-8 MB | Block-size dependent |
| **Configurability**| 22 levels | 1-16 (HC variant)| None | Block size 1-9 | cname, clevel, shuffle, typesize, blocksize |
| **Streaming?** | Yes | No (block only) | No (raw only) | Yes (block-based)| Yes (block-based) |
| **Rust Backend** | zstd (C FFI) | lz4_flex (pure) | snap (pure) | bzip2 (C FFI) | Pure Rust w/ existing crates |
| **Best For** | Balanced use | Real-time/lowest latency | Short-lived data | Archival/storage | Numerical arrays |
| **License** | BSD/GPL dual | BSD 2-Clause | BSD 3-Clause | BSD-like | BSD 3-Clause |
| **ExCodecs Default**| Level 3 | Level 1 | N/A | Block 9 / WF 30 | LZ4, level 5, byte shuffle |
*Zstd compression speed varies from very fast (level 1, ~520 MB/s) to slow
(level 22, ~4 MB/s). Decompression is consistently fast (~1.3 GB/s).
### Speed Comparison (Approximate, Silesia Corpus)
```
Compression Speed (MB/s, higher is better)
LZ4 ████████████████████████████████████████ ~700
Snappy ██████████████████████████████████ ~550
Zstd-1 ███████████████████████████ ~520
Zstd-3 █████████████████████████ ~430
Blosc2 ██████████████████████ ~350*
Bzip2-9 ██ ~8
Decompression Speed (MB/s, higher is better)
LZ4 ████████████████████████████████████████ ~3000
Snappy ████████████████████████████ ~1700
Zstd ██████████████████████ ~1300
Blosc2 ██████████████████ ~1100*
Bzip2-9 █ ~25
Compression Ratio (higher is better)
Bzip2-9 █████████████████████████████████ ~3.3x
Zstd-22 ████████████████████████████████ ~3.8x
Zstd-3 ██████████████████████████ ~3.0x
Zstd-1 ████████████████████████ ~2.8x
LZ4 ████████████████████ ~2.1x
Snappy ██████████████████ ~2.0x
* Blosc2 speeds depend heavily on internal compressor and data typesize.
Numbers shown use LZ4 as internal compressor with byte shuffle on float64.
```
### When to Use Each Codec
| Use Case | Recommended Codec | Reason |
|---------------------------------------|--------------------|----------------------------------------|
| General-purpose compression | Zstd level 3 | Best ratio/speed tradeoff |
| Real-time/low-latency compression | LZ4 | Fastest compression and decompression |
| Short-lived data (RPC, caching) | Snappy | Minimal overhead, deterministic output |
| Archival/storage | Bzip2 | Highest ratio (or Zstd level 19-22) |
| Numerical arrays (float64, int32) | Blosc2 | Shuffle dramatically improves ratio |
| Content-addressable storage | Zstd | Deterministic output at same level |
| Maximum decompression speed | LZ4 | 3+ GB/s decompression |
| Minimum memory usage | LZ4 or Snappy | 16-32 KB compression buffer |
| Streaming compression | Zstd | Native streaming API |
| Binary data with known element size | Blosc2 | Shuffle exploit element structure |
---
## Decision Guide
```
What are you compressing?
|
+------------+-----------+
| |
Numerical arrays? General binary data?
(float64, int32, etc.) (text, JSON, arbitrary)
| |
Use Blosc2 What matters more?
with byte shuffle |
+---------+---------+
| |
Speed? Ratio?
| |
Use LZ4 How much speed
(1 GB/s+) are you willing
to sacrifice?
|
+-------+-------+
| |
Some A lot
| |
Use Zstd-3 Use Bzip2
(fast + good (best ratio,
ratio) slow speed)
```
For most use cases, **Zstd at level 3** is the recommended default. It
provides a strong compression ratio at high speed, and its decompression
performance is excellent. Switch to LZ4 only when latency is critical and
the ratio penalty is acceptable. Switch to Bzip2 only for archival use
where decompression speed is irrelevant. Use Blosc2 when your data is
numerical arrays and you can specify the element size.