Current section

Files

Jump to
kayrock README.md
Raw

README.md

# Kayrock

[![CI Checks](https://github.com/kafkaex/kayrock/actions/workflows/checks.yml/badge.svg?branch=master)](https://github.com/kafkaex/kayrock/actions/workflows/checks.yml)
[![CI Tests](https://github.com/kafkaex/kayrock/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/kafkaex/kayrock/actions/workflows/test.yml)
[![Module Version](https://img.shields.io/hexpm/v/kayrock.svg)](https://hex.pm/packages/kayrock)
[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/kayrock/)
[![Total Download](https://img.shields.io/hexpm/dt/kayrock.svg)](https://hex.pm/packages/kayrock)
[![License](https://img.shields.io/hexpm/l/kayrock.svg)](https://hex.pm/packages/kayrock)
[![Last Updated](https://img.shields.io/github/last-commit/kafkaex/kayrock.svg)](https://github.com/kafkaex/kayrock/commits/master)

Idiomatic Elixir interface to the Kafka protocol.

This library provides serialization and deserialization for all Kafka protocol messages,
based on [kafka_protocol](https://github.com/kafka4beam/kafka_protocol) for Erlang.
It is built to work with [KafkaEx](https://github.com/kafkaex/kafka_ex) though there
is no reason why it couldn't be used elsewhere.

## Installation

Add `kayrock` to your list of dependencies in `mix.exs`:

```elixir
def deps do
  [
    {:kayrock, "~> 1.1"}
  ]
end
```

## Requirements

- Elixir 1.14 or later
- Erlang/OTP 24.3 or later

> **Upgrading from 1.0.x:** 1.1.0 reports the Kafka 4.4.0 maximum versions from
> `KafkaSchemaMetadata.version_range/1`. If you use Kayrock through KafkaEx,
> upgrade to KafkaEx 1.1.3 or later first: it caps negotiation at the versions
> it implements. KafkaEx <= 1.1.2 would start negotiating versions it cannot
> build.

## Basic Architecture and Usage

Kayrock generates structs for every version of every message in the Kafka
protocol. It does this by converting the Erlang-based schema descriptions in
`:kpro_schema` from the `kafka_protocol` library and is therefore limited to
messages and versions included there.

Each request message has a serializer (that generates iodata), and each response
message has a deserializer. There is also a protocol, `Kayrock.Request`, that has
an implementation created for each request.

The generated structs are namespaced as follows:
`Kayrock.<API>.V<version>.[Request|Response]`

For example:
- `Kayrock.Fetch.V4.Request`
- `Kayrock.Produce.V1.Response`
- `Kayrock.Metadata.V1.Request`

### Example Usage

```elixir
alias Kayrock.Request

# Create a request
request = %Kayrock.ApiVersions.V1.Request{
  client_id: "my_client",
  correlation_id: 0
}

# Serialize to wire protocol (iodata)
wire_protocol = Request.serialize(request)

# ... send wire_protocol over your connection ...
# ... receive binary response ...

# Deserialize the response
response_deserializer = Request.response_deserializer(request)
{deserialized_resp, _rest} = response_deserializer.(binary_response)

# deserialized_resp is now a %Kayrock.ApiVersions.V1.Response{}
```

## Message Compression

Kayrock supports four compression formats:

| Format | Built-in | Dependency Required |
|--------|----------|---------------------|
| gzip | Yes | None |
| snappy | No | `{:snappyer, "~> 1.2"}` |
| lz4 | No | `{:lz4b, "~> 0.0.13"}` |
| zstd | OTP 28+ | `{:ezstd, "~> 1.0"}` (for OTP < 28) |

### Installing Compression Dependencies

Add the compression libraries you need to your `mix.exs`:

```elixir
def deps do
  [
    {:kayrock, "~> 1.1"},

    # Add compression libraries as needed:
    {:snappyer, "~> 1.2"},     # For Snappy compression
    {:lz4b, "~> 0.0.13"},      # For LZ4 compression
    {:ezstd, "~> 1.0"},        # For Zstandard (OTP < 28)
  ]
end
```

### Zstandard Support

Zstandard compression is available via:
1. **OTP 28+**: Native `:zstd` module (no dependency needed)
2. **OTP < 28**: Add `{:ezstd, "~> 1.0"}` to your dependencies

## Code Generation

This repo includes a mix task, `mix gen.kafka_protocol`, that produces the code
in `lib/generated`. The generated code is checked into the repository to
simplify usage. End users should not need to run the mix task unless they are
doing development on Kayrock itself.

Schemas come from two sources:

- **Apache Kafka's JSON message definitions**, vendored (unmodified, with their
  Apache License headers) under `priv/kafka_messages/<kafka-version>/` and
  translated by `Kayrock.Generate.JsonSpec`. The pinned release is
  Kafka 4.4.0; see `priv/kafka_messages/4.4.0/PROVENANCE.md`. Re-fetch (or fetch
  another tag) with `mix kayrock.fetch_kafka_messages <tag>`.
- **`kafka_protocol`'s `kpro_schema`** for every version Kayrock generated
  before the JSON-driven generator existed, so those structs and their wire
  encoding stay byte-for-byte identical. (Kafka 4.x also dropped many old
  versions from its JSON files, per KIP-896.)

In JSON-derived versions, KIP-482 tagged fields are ordinary struct fields: a
field left at its default is not sent, and a decoded response fills them in
(with their defaults when absent). Tags Kayrock doesn't know about are kept as
raw `{tag, binary}` pairs in `tagged_fields`, in both directions. Versions
sourced from `kpro_schema` keep every tagged field in `tagged_fields`, as
before.

Which APIs are generated, and up to which version, is decided by one table:
`Kayrock.Generate.ApiTable` (`lib/kayrock/generate/api_table.ex`). Supporting a
newer version of an API means raising its maximum there and running
`mix gen.kafka_protocol`. The task validates the table first and refuses
versions that are marked `latestVersionUnstable` in the JSON specs.
`mix gen.kafka_protocol --check` (run in CI) fails if regenerating would change
anything in `lib/generated`.

## Relationship to Other Libraries

Kayrock uses only the `kpro_schema` module from `kafka_protocol`, and only at
code-generation time, for the API versions it has always generated. Newer
versions are generated from Apache Kafka's own JSON message definitions.
`kafka_protocol` provides quite a lot of functionality (especially when used as
part of `brod`). This repo limits the integration surface because:

1. `kafka_protocol` produces Erlang records whereas Kayrock provides Elixir structs
2. `kafka_protocol` provides network-level implementation whereas Kayrock only
   provides serialization and deserialization
3. `kpro_schema` is itself generated from Kafka's JSON message definitions,
   which Kayrock now reads directly for newer versions

### Built-in Client (Development Only)

This repo includes a lightweight Kafka client implementation for development and
testing purposes.

> **Warning:** The built-in client (`Kayrock.Client`) is NOT production-ready.
> For production usage, please use:
> - [KafkaEx](https://github.com/kafkaex/kafka_ex)
> - [brod](https://github.com/kafka4beam/brod)

## Testing

Kayrock includes both unit tests and integration tests.

### Unit Tests

```bash
mix test
```

### Integration Tests

Integration tests require Docker. We use
[testcontainers](https://github.com/testcontainers/testcontainers-elixir)
to run a Kafka cluster.

```bash
# Requires Docker to be running
mix test.integration
```

### Kafka 4.x Tests

`mix test.kafka4` runs the `:kafka4` suite (`test/kafka4/`) against a
single-node KRaft `apache/kafka:4.4.0` broker (`Kayrock.Kafka4Container`).
The `:integration`, `:sanity` and `:chaos` suites keep using Confluent 7.4.
Kafka 4.x removed many old protocol versions (KIP-896), and the broker does
not answer requests at those versions. `test/kafka4/api_versions_smoke_test.exs`
lists which generated versions 4.4.0 no longer accepts.

```bash
# Requires Docker to be running
mix test.kafka4
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.

## License

MIT License - see [LICENSE.txt](LICENSE.txt) for details.