Packages

An extensible BEAM-native codec framework for Elixir — compression and spatial formats

Current section

Files

Jump to
ex_codecs lib ex_codecs.ex
Raw

lib/ex_codecs.ex

defmodule ExCodecs do
@moduledoc """
Extensible BEAM-native **codec framework** for Elixir.
## One framework, specialized category APIs
Binary→binary registry codecs use:
ExCodecs.encode(codec_atom, binary, opts \\\\ [])
ExCodecs.decode(codec_atom, binary, opts \\\\ [])
All implementations are registered in the shared `ExCodecs.CodecRegistry`
catalog. Category modules provide entry points suited to their data:
* `ExCodecs.Compression` — `compress/3` / `decompress/3` aliases of the
registry API, plus listing codecs in the compression category
* `ExCodecs.Spatial` — domain types and formats for point clouds /
Gaussians (struct↔format). Call `ExCodecs.Spatial.encode/2` and
`ExCodecs.Spatial.decode/2`
The APIs belong to one framework and share tagged results and
`%ExCodecs.Error{}` conventions. They are not overloaded because their input
shapes differ. Registry encoding and decoding keep the codec atom first:
{:ok, compressed} = ExCodecs.encode(:zstd, data)
{:ok, decoded} = ExCodecs.decode(:zstd, compressed)
## Registry codecs
| Codec | Notes |
|-------|--------|
| `:zstd` | Pure-Rust Zstd (`structured-zstd`) |
| `:lz4` | Size-prepended `lz4_flex` blocks |
| `:snappy` | Standalone Snappy codec |
| `:bzip2` | Pure-Rust bzip2 |
| `:blosc2` | **C-Blosc2 chunk** (not super-chunk / B2ND / `.b2frame`) |
### Snappy vs Blosc2 `cname: :snappy`
- `ExCodecs.encode(:snappy, data)`**supported** (standalone codec).
- `ExCodecs.encode(:blosc2, data, cname: :snappy)`**rejected** (`:invalid_options`).
Snappy is not a standard C-Blosc2 inner compressor in this build; use
`:lz4`, `:blosclz`, `:zstd`, `:lz4hc`, or `:zlib` inside Blosc2.
### Blosc2 “chunk only”
`:blosc2` compresses **one buffer → one Blosc2 chunk**. That matches normal
ExCodecs use (and python-blosc2 `compress`/`decompress` on a single buffer).
It does **not** open `.b2frame` files, append super-chunks, or slice B2ND
arrays. For large data, chunk yourself and store multiple blobs.
## Quick start
# Registry compression
{:ok, c} = ExCodecs.encode(:zstd, "hello")
{:ok, "hello"} = ExCodecs.decode(:zstd, c)
# Category alias
{:ok, c} = ExCodecs.Compression.compress(:lz4, data)
# Spatial category
alias ExCodecs.Spatial.{Point, PointCloud}
cloud = PointCloud.new([Point.new(0.0, 0.0, 0.0)])
{:ok, ply} = ExCodecs.Spatial.encode(cloud, format: :ply)
{:ok, cloud} = ExCodecs.Spatial.decode(ply, format: :ply)
ExCodecs.available_codecs()
#=> [:blosc2, :bzip2, :gsplat, :lz4, :ply, :snappy, :spatial_binary, :zstd]
## Error policy
Public encode/decode paths return `{:ok, _}` or `{:error, %ExCodecs.Error{}}`.
They do **not** raise for invalid codecs, bad options, or compression failure.
(NIF load failure is converted to `{:error, %Error{reason: :nif_not_loaded}}`.)
"""
alias ExCodecs.{Codec, CodecRegistry, Error}
@doc """
Encodes a **binary** with a **registered** codec.
Primary framework entry point. First argument is always a codec atom
(`:zstd`, `:lz4`, …). For spatial structs use `ExCodecs.Spatial.encode/2`.
## Arguments
* `codec` (`atom()`) — registry key, such as `:zstd`
* `data` (`binary()`) — unencoded bytes
* `opts` (`keyword()`) — codec-specific options; defaults to `[]`
## Returns
* `{:ok, binary()}` — encoded payload
* `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` — `codec` is not
registered
* `{:error, %ExCodecs.Error{reason: :codec_unavailable}}` — `codec` is
registered without an implementation module
* `{:error, %ExCodecs.Error{reason: :invalid_data}}` — `data` is not a
binary, arguments do not have the documented shape, or the codec rejects
the input
* `{:error, %ExCodecs.Error{reason: :invalid_options}}` — codec-specific
option validation failed
* `{:error, %ExCodecs.Error{reason: :compression_failed}}` — native encoder
failed
* `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` — native library is
unavailable
## Raises
Does not raise for the documented failure modes, including malformed public
arguments. It may raise `ArgumentError` if the registry has not been started,
or propagate an unexpected exception from a third-party registered codec.
## Examples
iex> {:ok, compressed} = ExCodecs.encode(:zstd, "hello world")
iex> is_binary(compressed)
true
iex> {:ok, compressed} = ExCodecs.encode(:zstd, "hello world", level: 3)
iex> is_binary(compressed)
true
iex> {:error, %ExCodecs.Error{reason: :unsupported_codec}} =
...> ExCodecs.encode(:not_a_codec, "x")
iex> true
true
"""
@spec encode(atom(), binary(), keyword()) :: {:ok, binary()} | {:error, Error.t()}
def encode(codec, data, opts \\ [])
def encode(codec, data, opts) when is_atom(codec) and is_binary(data) and is_list(opts) do
case CodecRegistry.lookup(codec) do
{:ok, {_module, _category, %{interface: interface}}}
when interface != :binary ->
interface_error(codec, :encode)
{:ok, {module, _category, info}} ->
case ensure_available(info, codec) do
:ok -> module.encode(data, opts)
{:error, %Error{} = error} -> {:error, error}
end
{:error, :unsupported_codec} ->
{:error, Error.new(:unsupported_codec, codec: codec)}
end
end
def encode(codec, _data, _opts) when is_atom(codec) do
{:error,
Error.new(:invalid_data,
codec: codec,
message: "ExCodecs.encode/3 expects a binary as the second argument"
)}
end
def encode(%_{} = struct, _data_or_opts, _opts) do
{:error,
Error.new(:invalid_data,
message:
"Structured spatial data is encoded via ExCodecs.Spatial.encode/2 " <>
"(got #{inspect(struct.__struct__)}). Registry encode/3 is binary codecs only."
)}
end
def encode(_codec, _data, _opts) do
{:error,
Error.new(:invalid_data,
message: "ExCodecs.encode/3 expects encode(codec_atom, binary, opts \\\\ [])"
)}
end
@doc """
Decodes a **binary** with a **registered** codec.
First argument is always a codec atom. For spatial formats use
`ExCodecs.Spatial.decode/2` with `format:`.
## Arguments
* `codec` (`atom()`) — registry key, such as `:zstd`
* `data` (`binary()`) — encoded payload
* `opts` (`keyword()`) — codec-specific decoding options; defaults to `[]`
## Returns
* `{:ok, binary()}` — decoded payload
* `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` — `codec` is not
registered
* `{:error, %ExCodecs.Error{reason: :codec_unavailable}}` — `codec` has no
implementation module
* `{:error, %ExCodecs.Error{reason: :invalid_data}}` — arguments have the
wrong shape or the codec rejects the input
* `{:error, %ExCodecs.Error{reason: :invalid_options}}` — options are
invalid, including the mistaken `decode(binary, format: format)` shape
* `{:error, %ExCodecs.Error{reason: :decompression_failed}}` — payload is
corrupt or the native decoder failed
* `{:error, %ExCodecs.Error{reason: :nif_not_loaded}}` — native library is
unavailable
## Raises
Does not raise for the documented failure modes. It may raise `ArgumentError`
if the registry has not been started, or propagate an unexpected exception
from a third-party registered codec.
## Examples
iex> {:ok, compressed} = ExCodecs.encode(:zstd, "hello world")
iex> {:ok, original} = ExCodecs.decode(:zstd, compressed)
iex> original
"hello world"
"""
@spec decode(atom(), binary(), keyword()) :: {:ok, binary()} | {:error, Error.t()}
def decode(codec, data, opts \\ [])
def decode(codec, data, opts) when is_atom(codec) and is_binary(data) and is_list(opts) do
case CodecRegistry.lookup(codec) do
{:ok, {_module, _category, %{interface: interface}}}
when interface != :binary ->
interface_error(codec, :decode)
{:ok, {module, _category, info}} ->
case ensure_available(info, codec) do
:ok -> module.decode(data, opts)
{:error, %Error{} = error} -> {:error, error}
end
{:error, :unsupported_codec} ->
{:error, Error.new(:unsupported_codec, codec: codec)}
end
end
def decode(codec, _data, _opts) when is_atom(codec) do
{:error,
Error.new(:invalid_data,
codec: codec,
message: "ExCodecs.decode/3 expects a binary as the second argument"
)}
end
def decode(data, opts, []) when is_binary(data) and is_list(opts) do
if Keyword.has_key?(opts, :format) do
{:error,
Error.new(:invalid_options,
message:
"Spatial formats use ExCodecs.Spatial.decode/2 " <>
"(e.g. ExCodecs.Spatial.decode(data, format: :ply)). " <>
"Registry decode/3 is decode(codec_atom, binary, opts)."
)}
else
{:error,
Error.new(:invalid_data,
message: "ExCodecs.decode/3 expects decode(codec_atom, binary, opts \\\\ [])"
)}
end
end
def decode(_codec, _data, _opts) do
{:error,
Error.new(:invalid_data,
message: "ExCodecs.decode/3 expects decode(codec_atom, binary, opts \\\\ [])"
)}
end
@doc """
Lazily enumerates **spatial** primitives from a file path or binary.
Delegates to `ExCodecs.Spatial.stream_decode/2`. **Not** used for registry
compression codecs. Today this materializes the payload then streams the list.
## Arguments
* `source` (`Path.t() | binary()`) — filesystem path or encoded binary;
use the `:source` option to disambiguate a binary that is also a path
* `opts` (`keyword()`) — requires `:format` (`:ply`, `:spatial_binary`, or
`:gsplat`); optionally accepts `:source` (`:auto`, `:file`, or `:binary`)
and format-specific options
## Returns
An `Enumerable.t()` yielding `%ExCodecs.Spatial.Point{}` or
`%ExCodecs.Spatial.Gaussian{}` values. On failure it yields exactly one
`{:error, %ExCodecs.Error{}}` element with one of these reasons:
* `:invalid_options` — required `:format` is absent
* `:unsupported_codec` — the format is not stream-decodable
* `:io_error` — a file source cannot be read
* `:invalid_data` — the encoded payload is malformed or truncated
## Raises
Missing or invalid format values and file read errors are yielded as error
tuples. Passing non-keyword `opts` raises `FunctionClauseError`; enumeration
may propagate unexpected exceptions from the source enumerable or runtime.
## Examples
iex> alias ExCodecs.Spatial.{Point, PointCloud}
iex> {:ok, bin} = ExCodecs.Spatial.encode(PointCloud.new([Point.new(1.0, 2.0, 3.0)]), format: :ply)
iex> [%Point{x: x} | _] = ExCodecs.stream_decode(bin, format: :ply) |> Enum.to_list()
iex> x == 1.0
true
"""
@spec stream_decode(Path.t() | binary(), keyword()) :: Enumerable.t()
def stream_decode(source, opts) when is_list(opts) do
ExCodecs.Spatial.stream_decode(source, opts)
end
@doc """
Encodes an enumerable of spatial `%Point{}` / `%Gaussian{}` values.
Delegates to `ExCodecs.Spatial.stream_encode/2`. Collects the enumerable
(format headers need a count). **Not** for registry compression codecs.
## Arguments
* `enumerable` (`Enumerable.t()`) — points or Gaussians, all of one type
* `opts` (`keyword()`) — requires `:format` (`:ply`, `:spatial_binary`, or
`:gsplat`) and may contain format-specific options
## Returns
* `{:ok, binary()}` — encoded spatial payload
* `{:error, %ExCodecs.Error{reason: :invalid_options}}` — required format
is absent or a format option is invalid
* `{:error, %ExCodecs.Error{reason: :unsupported_codec}}` — format is
unknown
* `{:error, %ExCodecs.Error{reason: :invalid_data}}` — items are mixed,
malformed, unsupported, or include an error tuple
## Raises
Returns error tuples for documented format, option, and item failures.
Passing non-keyword `opts` raises `FunctionClauseError`. A value that does not
implement `Enumerable` raises `Protocol.UndefinedError`, and exceptions raised
while enumerating propagate.
## Examples
iex> alias ExCodecs.Spatial.Point
iex> {:ok, bin} = ExCodecs.stream_encode([Point.new(0.0, 0.0, 0.0)], format: :ply)
iex> is_binary(bin)
true
"""
@spec stream_encode(Enumerable.t(), keyword()) :: {:ok, binary()} | {:error, Error.t()}
def stream_encode(enumerable, opts) when is_list(opts) do
ExCodecs.Spatial.stream_encode(enumerable, opts)
end
@doc """
Lists **registered** codec atoms that are available at runtime.
## Arguments
None.
## Returns
A sorted `[atom()]` containing every shared-catalog entry whose
implementation module is non-`nil`, including binary and spatial codecs.
## Raises
May raise `ArgumentError` if the registry ETS table has not been started.
## Examples
iex> :blosc2 in ExCodecs.available_codecs()
true
"""
@spec available_codecs() :: [atom()]
def available_codecs do
CodecRegistry.available_codecs()
end
@doc """
Lists available codec names in one category.
## Arguments
* `category` — category atom such as `:compression` or `:spatial`
## Returns
A sorted list of available names in that category.
## Raises
Raises `FunctionClauseError` for a non-atom category and may raise
`ArgumentError` if the shared catalog has not started.
## Examples
iex> ExCodecs.available_codecs(:spatial)
[:gsplat, :ply, :spatial_binary]
"""
@spec available_codecs(atom()) :: [atom()]
def available_codecs(category) when is_atom(category) do
CodecRegistry.available_codecs(category)
end
@doc """
Returns whether a shared-catalog codec is available.
## Arguments
* `codec` (`atom()`) — registry key to test
## Returns
`true` if `codec` is registered with a non-`nil` implementation module;
otherwise `false`.
## Raises
Raises `FunctionClauseError` if `codec` is not an atom. May raise
`ArgumentError` if the registry ETS table has not been started.
## Examples
iex> ExCodecs.supports?(:zstd)
true
iex> ExCodecs.supports?(:nonexistent)
false
"""
@spec supports?(atom()) :: boolean()
def supports?(codec) when is_atom(codec) do
CodecRegistry.supports?(codec)
end
@doc """
Returns metadata for a registered codec.
## Arguments
* `codec` (`atom()`) — registry key whose metadata is requested
## Returns
* `{:ok, ExCodecs.Codec.t()}` — name, category, interface, module,
capability flags, and backend version; unavailable codecs are returned
with `module: nil`
* `{:error, :unsupported_codec}` — not registered (note: bare atom, not `%Error{}`)
## Raises
Raises `FunctionClauseError` if `codec` is not an atom. May raise
`ArgumentError` if the registry ETS table has not been started.
## Examples
iex> {:ok, info} = ExCodecs.codec_info(:zstd)
iex> info.name
:zstd
iex> {:ok, info} = ExCodecs.codec_info(:zstd)
iex> info.category
:compression
"""
@spec codec_info(atom()) :: {:ok, Codec.t()} | {:error, :unsupported_codec}
def codec_info(codec) when is_atom(codec) do
CodecRegistry.codec_info(codec)
end
defp ensure_available(info, codec) do
if info.module != nil do
:ok
else
{:error, Error.new(:codec_unavailable, codec: codec)}
end
end
defp interface_error(codec, operation) do
{:error,
Error.new(:invalid_options,
codec: codec,
message:
"#{inspect(codec)} uses the spatial category API; call " <>
"ExCodecs.Spatial.#{operation}/2 with format: #{inspect(codec)}"
)}
end
end