ex_codecs

# Choosing a Compression Codec

This guide helps you select the right compression codec for your use case. It includes a comparison table, decision criteria, and worked examples.

> ### Is the data spatial, or spatially representable?
>
> Compression codecs operate on arbitrary binaries. If records represent
> coordinates, points, normals, colors, oriented particles, or Gaussian splats,
> consider converting them to `ExCodecs.Spatial` types first—even when the
> source is currently a flat array, CSV rows, maps, or database records.
>
> Use:
>
> - `ExCodecs.Spatial.PointCloud` for XYZ, XYZRGB(A), normals, and scalar
>   per-point attributes.
> - `ExCodecs.Spatial.GaussianCloud` for position, scale, rotation, opacity,
>   color, and spherical harmonics.
> - `:ply` for interoperable spatial exchange.
> - `:spatial_binary` (EXCP) for compact point clouds.
> - `:gsplat` (GSPL) for compact Gaussian clouds.
>
> Spatial encoding and compression are complementary. Encode the structured
> data first, then optionally compress the resulting binary:
>
> ```elixir
> alias ExCodecs.Spatial.{Point, PointCloud}
>
> cloud =
>   rows
>   |> Enum.map(fn %{x: x, y: y, z: z} -> Point.new(x, y, z) end)
>   |> PointCloud.new()
>
> {:ok, spatial_binary} =
>   ExCodecs.Spatial.encode(cloud, format: :spatial_binary)
>
> {:ok, compressed} = ExCodecs.encode(:zstd, spatial_binary, level: 3)
> ```
>
> If the numbers have no geometric meaning and only need compact storage,
> Blosc2 may be a better fit. See
> [Understanding Spatial Codecs](understanding_spatial_codecs.md) and
> [Spatial Wire Formats](../docs/spatial_formats.md).

## Quick Comparison

| Codec    | Compression Ratio | Compression Speed | Decompression Speed | Memory Usage | Configurable | Streaming | Best For                          |
|----------|-------------------|-------------------|---------------------|--------------|--------------|-----------|-----------------------------------|
| LZ4      | Low               | Very Fast         | Very Fast           | Very Low     | No           | No        | Real-time, latency-sensitive      |
| Snappy   | Low               | Very Fast         | Very Fast           | Very Low     | No           | No        | Short-lived data, RPC payloads    |
| Zstd     | High              | Fast              | Very Fast           | Moderate     | Levels 1-22  | No        | General purpose, storage, network |
| Bzip2    | Very High         | Slow              | Slow                | Moderate     | Block 1-9    | No        | Archival, offline processing      |
| Blosc2   | High*             | Fast              | Very Fast           | Moderate     | Codec/level/shuffle/typesize | No | Numerical arrays, typed data |

\* Blosc2 ratio depends heavily on the internal codec (`cname`) and shuffle settings. With byte shuffle on typed data, ratios can exceed Zstd.

“Streaming” here means an incremental compression API. All current compression
codecs operate on complete input buffers.

## Decision Framework

Answer the following questions to narrow your choice:

### 1. What is your data type?

- **Coordinates, point samples, particles, normals, or Gaussian splats** --
  model them with `ExCodecs.Spatial`, even if they currently arrive as arrays,
  rows, maps, or CSV. Optionally compress the encoded PLY/EXCP/GSPL binary.
- **Text, JSON, logs, general-purpose binary** -- Zstd is the best default. Use level 3 for a balance of speed and ratio.
- **Typed numerical arrays without spatial semantics (floats, integers, matrices)** -- Blosc2 with appropriate `typesize` and `shuffle` settings.
- **Short-lived messages, RPC payloads** -- Snappy or LZ4 for minimal latency.
- **Archival storage** -- Bzip2 for maximum ratio, or Zstd at level 19-22 for high ratio with better decompression speed.

### 2. How fast does compression need to be?

- **Need >= ~1 GB/s** - LZ4 or Snappy.
- **Need >= ~300-500 MB/s** - Zstd level 1-3, Blosc2 with LZ4 inner codec.
- **No constraint** - Zstd high levels (9-22) or Bzip2.

### 3. How fast does decompression need to be?

- **Need multi-GB/s** - LZ4 or Snappy.
- **Need ~1 GB/s class** - Zstd any level, Blosc2.
- **No constraint** - Any codec. Bzip2 decompression is typically much slower.

### 4. How much memory can you spare?

- **Very constrained (embedded, large concurrent load)** -- LZ4 or Snappy.
- **Moderate** -- Zstd at levels 1-14.
- **Available** -- Zstd at levels 15-22 or Bzip2 at block sizes 7-9.

### 5. Is data read once or many times?

- **Read many times** -- Invest in higher compression. The one-time compression cost amortizes over many decompressions. Zstd level 9-14 or Bzip2.
- **Read once or rarely** -- Use fast compression. LZ4 or Snappy.
- **Never decompressed (checksum only)** -- Consider whether you need compression at all.

## Codec-Specific Selection Guides

### When to Use LZ4

```elixir
{:ok, compressed} = ExCodecs.encode(:lz4, data)
```

- ExCodecs exposes one fixed, fast LZ4 profile; it does not accept `:level`.
- Low-latency message queues
- Real-time data pipelines where throughput matters more than size
- Temporary data that will be decompressed quickly
- In-memory caches where the CPU cost of decompression must be negligible
- When you need deterministic compression and decompression speeds

### When to Use Snappy

```elixir
{:ok, compressed} = ExCodecs.encode(:snappy, data)
```

- RPC frameworks (Snappy is the default in many RPC systems)
- Data that is already partially compressed or has low entropy
- Situations where you want zero configuration
- When every microsecond counts and compression ratio is secondary

### When to Use Zstd

```elixir
{:ok, compressed} = ExCodecs.encode(:zstd, data, level: 3)
```

- General-purpose compression (Zstd is the best default choice)
- Databases, file storage, and network transmission
- Workloads where decompression speed matters (Zstd decompresses fast regardless of compression level)
- When you need a configurable tradeoff (22 levels from fast to maximum ratio)

ExCodecs does not currently expose Zstd dictionary compression.

### When to Use Bzip2

```elixir
{:ok, compressed} = ExCodecs.encode(:bzip2, data, block_size: 9)
```

- Archival storage where maximum ratio is the priority
- Offline batch processing where compression time is not constrained
- Data that will be stored for a long time and decompressed rarely
- Interoperability with the `.bz2` ecosystem

### When to Use Blosc2

```elixir
{:ok, compressed} = ExCodecs.encode(:blosc2, data, cname: :zstd, clevel: 5, shuffle: :byte, typesize: 8)
```

- Numerical arrays (float64, int32, etc.)
- Scientific data, time series, matrix storage
- Situations where shuffle filters provide a significant ratio improvement
- When you need fine-grained control over the compression pipeline

Each Blosc2 NIF call is single-threaded (`nthreads: 1`). Parallelize independent
buffers with BEAM processes.

## Worked Examples

### Example 1: API Response Cache

A web application caches JSON responses. Data is compressed once, read many times.

**Choice: Zstd at level 5-9**

```elixir
# Compression (one-time cost)
{:ok, compressed} = ExCodecs.encode(:zstd, json_binary, level: 7)

# Decompression (many reads)
{:ok, original} = ExCodecs.decode(:zstd, compressed)
```

Rationale: Zstd decompresses quickly regardless of compression level, so invest more in compression to get better ratios for the cache.

### Example 2: Real-Time Message Broker

Messages arrive at high volume and must be forwarded with minimal latency.

**Choice: LZ4**

```elixir
{:ok, compressed} = ExCodecs.encode(:lz4, message)
```

Rationale: Latency is the priority. ExCodecs' fixed fast LZ4 profile adds
minimal overhead to the pipeline.

### Example 3: Scientific Data Archive

A research pipeline archives float64 measurement arrays to cold storage.

**Choice: Blosc2 with Zstd inner codec and byte shuffle**

```elixir
{:ok, compressed} = ExCodecs.encode(:blosc2, float_array_binary,
  cname: :zstd,
  clevel: 9,
  shuffle: :byte,
  typesize: 8
)
```

Rationale: The byte shuffle reorders bytes within each 8-byte float, grouping high-order bytes (often similar) together. On typed arrays this often yields about a 2-4x ratio gain versus compressing the raw array (illustrative; see the Blosc2 guide table).

### Example 4: Log File Archival

Monthly log files are compressed and stored in object storage.

**Choice: Zstd at level 15-19 or Bzip2 at block size 9**

```elixir
{:ok, compressed} = ExCodecs.encode(:zstd, log_data, level: 17)
# or
{:ok, compressed} = ExCodecs.encode(:bzip2, log_data, block_size: 9)
```

Rationale: Compression is a one-time batch operation. Maximum ratio reduces storage costs over months. Zstd at high levels offers better decompression speed than Bzip2 if you need occasional access.

### Example 5: Short-Lived RPC Payload

An internal service sends compressed protobuf messages over the network.

**Choice: Snappy**

```elixir
{:ok, compressed} = ExCodecs.encode(:snappy, protobuf_binary)
```

Rationale: Protobuf already removes much redundancy. Snappy adds minimal overhead on both compression and decompression, and requires no configuration. The ratio improvement will be modest but the latency impact is negligible.

## Summary Table by Use Case

| Use Case                     | Recommended Codec | Configuration                       |
|------------------------------|-------------------|-------------------------------------|
| General-purpose default      | Zstd              | `level: 3`                          |
| Real-time / low-latency      | LZ4               | No codec-specific options            |
| Fastest with no config       | Snappy            | (none)                               |
| Maximum ratio / archival     | Bzip2 or Zstd     | `block_size: 9` or `level: 19-22`   |
| Numerical arrays             | Blosc2            | `cname: :zstd, shuffle: :byte`      |
| Spatially meaningful records | Spatial + optional compression | PLY/EXCP/GSPL, then optionally Zstd |
| Small repetitive payloads    | Zstd              | `level: 3` (dictionary API unavailable) |
| In-memory cache              | LZ4 or Snappy      | No codec-specific options            |
| Already slightly compressed  | Snappy or LZ4      | No codec-specific options            |