Packages

A thin wrapper around the rust scylla crate: https://crates.io/crates/scylla.

Current section

Files

Jump to
ex_scylla README.md
Raw

README.md

# ExScylla

An Elixir wrapper around the [scylla-rust-driver](https://rust-driver.docs.scylladb.com/) using Rustler.

* Minimal interface & abstraction
* Easy to maintain & to add new features as they arrive in the rust driver
* Users of ExScylla should be able to leverage the rust driver documentation

Documentation and API examples are available at: https://hexdocs.pm/ex_scylla
Technical architecture and design docs can be found in the [docs/](docs/README.md) folder.
Agent instructions and guidelines are located in [AGENTS.md](AGENTS.md).

## Prerequisites

* **mise**: Language runtime manager for Erlang, Elixir, and Rust (`mise install`).
* **Docker**: Required for running ScyllaDB test containers and benchmarks.
* **scylla_unstable flag**: Enabled automatically by `native/ex_scylla/build.rs` and `native/ex_scylla/.cargo/config.toml`.

## Architecture & Performance

### NIF/Rustler & Tokio Runtime
ExScylla uses Rustler to interface with the `scylla-rust-driver`. Because the Rust driver is heavily asynchronous and relies on the Tokio runtime, ExScylla spins up a dedicated Tokio runtime in the background to handle the driver's futures.

### Resource Reference Model & Cloning
`SessionBuilder`, `Session`, and statements (`Batch`, `Query`, `Prepared`) use `Arc` references to the underlying Rust objects. These references are passed to Elixir as NIF resources.

**Performance Note**: Statement setters (`Query`, `Prepared`, `Batch`) return a new reference to an updated copy (cloning the underlying Rust struct). As old instances are not immediately removed (waiting for Elixir GC), these setters should be avoided in the hot path. It is better to create and configure a statement once, then reuse it multiple times. Builders (`SessionBuilder`, `ExecutionProfileBuilder`, `DefaultPolicyBuilder`, `LatencyAwarenessBuilder`) are different: their setters mutate the builder in place and return the same reference.

### Async API & Timeouts
The library uses asynchronous NIFs for non-blocking operations to avoid starving the Erlang schedulers. An async NIF returns immediately and later sends `{opaque, result}` to the calling process. The `async_*` functions have no timeout. The synchronous wrappers (for example `Session.query/3`) wait for the reply with a default timeout of 5 seconds, set by an optional trailing `timeout_ms` argument.

**Many requests from one process**: replies are sent from several Tokio threads at once. A process that keeps many `async_*` requests in flight should call `Process.flag(:message_queue_data, :off_heap)`, which avoids mailbox lock contention (about 15–20% faster for 1000 concurrent inserts in our benchmarks).

### Dual API (Decoded vs Raw)
Both APIs call the same NIFs. Rust always returns the result rows as a raw binary, and the two APIs differ only in how Elixir decodes it:
* **Decoded API** (`query`, `execute`, `batch`): returns `%ExScylla.Types.QueryResult{}` whose rows are `%ExScylla.Types.Row{columns: [...]}` structs of type-tagged values such as `{:int, 42}`.
* **Raw API** (`query_raw`, `execute_raw`, `batch_raw`): returns `%ExScylla.Types.QueryResultRaw{}` whose rows are plain lists of untagged values. This skips building the type tags.

## Example

This example assumes a running ScyllaDB node and an existing `test.s_doc` table with columns `a text, b int, c double`.

```elixir
alias ExScylla.SessionBuilder
alias ExScylla.Session
alias ExScylla.Types.QueryResult

{:ok, session} = SessionBuilder.new()
                 |> SessionBuilder.known_node("127.0.0.1:9042")
                 |> SessionBuilder.build()
{:ok, ps} = Session.prepare(session, "INSERT INTO test.s_doc (a, b, c) VALUES (?, ?, ?)")
values = [{:text, "test"}, {:int, 2}, {:double, 1.0}]
{:ok, %QueryResult{}} = Session.execute(session, ps, values)

# Iterative queries (streams)
Session.query_stream(session, "SELECT * FROM test.s_doc", [])
|> Enum.each(fn row -> IO.inspect(row) end)
```

## Installation
https://hex.pm/packages/ex_scylla

```elixir
def deps do
  [
    {:ex_scylla, "~> 0.11.0"}
  ]
end
```

## Repository Blueprint

### System Overview

ExScylla has two layers that must stay in sync:

1. **Elixir API** (`lib/ex_scylla`): `SessionBuilder`, `Session`, statements, execution profiles, load-balancing builders, result types, and `CqlTypes` row decoding. Wrappers are generated by `ExScylla.Macros.Native` and call stubs in `ExScylla.Native`.
2. **Rust NIF crate** (`native/ex_scylla`): a Rustler `cdylib` loaded as `Elixir.ExScylla.Native`. It wraps the `scylla` 1.10.0 driver objects as `ResourceArc` resources and runs all network I/O on one shared Tokio runtime.

Request flow for an I/O call such as `Session.query/3`:

1. The Elixir wrapper calls `ExScylla.Native.s_query(opaque, ...)`. The default `opaque` is `{function_name, make_ref()}`.
2. The NIF decodes the arguments synchronously, spawns a future through `async_elixir!`, and returns `:ok` right away.
3. When the future finishes, the caller receives `{opaque, result}`. The synchronous wrapper waits up to `timeout_ms` (default 5000), then returns `{:error, :timeout}`.
4. Result rows reach Elixir as a raw, resource-backed binary (`ScyllaRawRowsResource`). `ExScylla.Types.QueryResult.decode/1` (decoded API) or `decode_raw/1` (raw API) parses it in Elixir. Both APIs use the same NIFs.

### Local Interface & Invariants

- `mix.exs` (`ExScylla.MixProject`): app `:ex_scylla`, version `0.11.0`.
  - `elixirc_paths/1` adds `test/utils` for `:test` and `:bench`.
  - Aliases:
    - `test` re-runs `mix test.wrapper` (which calls `Mix.Tasks.Test.run/1`) in a subprocess. It passes `MIX_ENV`, `RUSTFLAGS="-C instrument-coverage --cfg scylla_unstable"` and `--color`/`--no-color`.
    - `bench` (`run_bench/1`) runs `mix run bench/<name>.exs` with `MIX_ENV=bench`.
  - The Hex package (`package/0`) ships `lib`, `priv/native/.place_holder`, `native/ex_scylla/src`, `native/ex_scylla/Cargo.*`, `native/ex_scylla/build.rs`, `native/ex_scylla/.cargo/config.toml` (needed for `--cfg scylla_unstable`), `.formatter.exs`, `mix.exs` and `README*`.
- `.formatter.exs`: formats `{mix,.formatter}.exs` and `{config,lib,test}/**/*.{ex,exs}`.
- `mise.toml` defines all workflow tasks (`deps:get`, `compile`, `lint`, `format`, `build`, `test`, `test:coverage`, `install-test-deps`, `bench`, `scylla:*`). Add new commands there, not in a `Makefile`.
- The package version in `mix.exs` and the crate version in `native/ex_scylla/Cargo.toml` move together (both `0.11.0`).

### Subdirectories & System Boundaries

- lib: [./lib/README.md](./lib/README.md) - Elixir source root. [./lib/ex_scylla/README.md](./lib/ex_scylla/README.md) covers `ExScylla.Native` stubs, `SessionBuilder`, `Session` (decoded, raw and stream APIs) and `CqlTypes`. It has children for [execution](./lib/ex_scylla/execution/README.md), [load_balancing](./lib/ex_scylla/load_balancing/README.md), [macros](./lib/ex_scylla/macros/README.md), [statement](./lib/ex_scylla/statement/README.md) and [types](./lib/ex_scylla/types/README.md) (with [errors](./lib/ex_scylla/types/errors/README.md)).
- native: [./native/README.md](./native/README.md) - the `ex_scylla` Rust crate ([./native/ex_scylla/README.md](./native/ex_scylla/README.md)) and its sources ([./native/ex_scylla/src/README.md](./native/ex_scylla/src/README.md)). Sources hold NIF handlers per area plus resource and type modules in [batch](./native/ex_scylla/src/batch/README.md), [execution](./native/ex_scylla/src/execution/README.md), [prepared_statement](./native/ex_scylla/src/prepared_statement/README.md), [query](./native/ex_scylla/src/query/README.md), [session](./native/ex_scylla/src/session/README.md) and [session_builder](./native/ex_scylla/src/session_builder/README.md).
- bench: [./bench/README.md](./bench/README.md) - standalone Benchee scripts comparing ExScylla with `erlcass` against a Testcontainers ScyllaDB instance. Run with `mise run bench <name>`.
- config: [./config/README.md](./config/README.md) - Mix config entry point that imports `<env>.exs`. Sets no application keys.
- Not covered by contracts: `test/` (ExUnit suite and `test/utils/test_support.ex`), `docs/` (prose design docs, see [docs/README.md](docs/README.md)), and `priv/` (the compiled NIF library plus the tracked `priv/native/.place_holder`, which ships in the Hex package).

### Global Invariants

- **Dual-layer sync.** Every public function needs a Rust NIF in `native/ex_scylla/src/`, a stub with the same arity in `lib/ex_scylla/native.ex`, and a wrapper in `lib/ex_scylla/`. Every `NifStruct` `#[module = "ExScylla.Types.*"]` and its fields must match the Elixir struct.
- **No blocking in schedulers.** Network I/O goes through `async_elixir!` and replies with exactly one `{opaque, result}` message. Never use `block_on`.
- **Copy-on-write statements, in-place builders.** `Query`, `Prepared` and `Batch` setters clone and return a new reference. `SessionBuilder`, `ExecutionProfileBuilder`, `DefaultPolicyBuilder` and `LatencyAwarenessBuilder` setters mutate under a `Mutex` and return the same reference.
- **Rows are decoded in Elixir.** Rust never builds row terms. `ScyllaValue` (Rust, used for bound values) and `ExScylla.CqlTypes.decode_value/2` (Elixir, used for results) must agree on the CQL type mapping.
- **Unstable driver APIs.** The crate must build with `--cfg scylla_unstable`, which `native/ex_scylla/build.rs` and `native/ex_scylla/.cargo/config.toml` both set.

### Global Side Effects

- Loading the NIF starts a process-global multi-threaded Tokio runtime that lives as long as the VM.
- `SessionBuilder.build` opens TCP connections to ScyllaDB/Cassandra nodes, and the driver runs background metadata, keepalive and pool tasks.
- Async results are delivered to the calling process mailbox. A reply that arrives after a sync timeout stays in the mailbox.
- Tests and benchmarks start a `scylladb/scylla:2026.3` Docker container through Testcontainers. Test builds write LLVM coverage profiles (`*.profraw`).

### System Verification

- Run: `mise run test` (ExUnit suite; requires Docker)
- Run: `mise run lint` (format check, warnings as errors, `cargo check`, clippy)

## Contributing

We welcome contributions! Please see the [CONTRIBUTING.md](CONTRIBUTING.md) file for guidelines on how to contribute to this project.

## Changelog

For a detailed list of changes and updates, please refer to the [CHANGELOG.md](CHANGELOG.md) file.