slate

<img src="https://slate.tylerbutler.com/slate.webp" alt="slate logo" width="200">

# slate

[![Package Version](https://img.shields.io/hexpm/v/slate)](https://hex.pm/packages/slate)
[![Hex Docs](https://img.shields.io/badge/hex-docs-ffaff3)](https://hexdocs.pm/slate/)

Type-safe Gleam wrapper for Erlang [DETS](https://www.erlang.org/doc/apps/stdlib/dets.html) (Disk Erlang Term Storage).

DETS provides persistent key-value storage backed by files on disk. Tables survive process crashes and node restarts. DETS is built into OTP — no external database or dependency is needed.

> [!IMPORTANT]
> **Erlang target only** — DETS is a BEAM feature with no JavaScript target support.

## When to use DETS

| Approach | Complexity | Persistence | Query capability |
|----------|-----------|-------------|------------------|
| JSON file | Low | Yes | None |
| **DETS** | **Low** | **Yes** | **Key lookup, fold** |
| SQLite/Postgres | High | Yes | Full SQL |
| Mnesia | High | Yes | Transactions, distribution |

DETS fills the gap between "serialize to a file" and "add a database dependency."

## Installation

```sh
gleam add slate
```

## Usage

### Set tables (one value per key)

```gleam
import gleam/dynamic/decode
import slate/set

pub fn main() {
  // Open or create a table
  let assert Ok(users) = set.open("data/users.dets",
    key_decoder: decode.string, value_decoder: decode.int)

  // Insert key-value pairs
  let assert Ok(Nil) = set.insert(users, "alice", 42)
  let assert Ok(Nil) = set.insert(users, "bob", 37)

  // Look up values
  let assert Ok(age) = set.lookup(users, key: "alice")
  // age == 42

  // Check membership
  let assert Ok(True) = set.member(users, key: "alice")
  let assert Ok(False) = set.member(users, key: "charlie")

  // Always close when done
  let assert Ok(Nil) = set.close(users)
}
```

### Safe table lifecycle with `with_table`

```gleam
import gleam/dynamic/decode
import slate
import slate/set

pub fn main() {
  // Table is closed after the callback returns
  let assert Ok(Nil) = set.with_table("data/config.dets",
    repair: slate.AutoRepair, access: slate.ReadWrite,
    key_decoder: decode.string, value_decoder: decode.string,
    callback: fn(table) {
      set.insert(table, "theme", "dark")
    })
}
```

Use `with_table` for short-lived operations. Select the repair policy and access
mode for each call. The helper closes the table when the callback returns and
attempts cleanup if the callback raises. If the owning process is terminated
before cleanup runs, DETS may still need repair on the next open.

For read-only access without automatic repair:

```gleam
import gleam/dynamic/decode
import slate
import slate/set

use table <- set.with_table(
  path: "data/config.dets", repair: slate.NoRepair, access: slate.ReadOnly,
  key_decoder: decode.string, value_decoder: decode.string)
set.lookup(table, key: "theme")
```

`ReadOnly` requires an existing file. This helper is available in all three
table modules.

**Migrating from 1.x:** `with_table` now requires `repair` and `access`.
Add `repair: slate.AutoRepair` and `access: slate.ReadWrite` to keep the previous
behavior. Cleanup and error handling are unchanged.

### Bag tables (multiple values per key)

```gleam
import gleam/dynamic/decode
import slate/bag

pub fn main() {
  let assert Ok(tags) = bag.open("data/tags.dets",
    key_decoder: decode.string, value_decoder: decode.string)

  let assert Ok(Nil) = bag.insert(tags, "color", "red")
  let assert Ok(Nil) = bag.insert(tags, "color", "blue")

  let assert Ok(colors) = bag.lookup(tags, key: "color")
  // colors == ["red", "blue"]

  let assert Ok(Nil) = bag.close(tags)
}
```

### Duplicate bag tables

```gleam
import gleam/dynamic/decode
import slate/duplicate_bag

pub fn main() {
  let assert Ok(events) = duplicate_bag.open("data/events.dets",
    key_decoder: decode.string, value_decoder: decode.string)

  let assert Ok(Nil) = duplicate_bag.insert(events, "click", "button_a")
  let assert Ok(Nil) = duplicate_bag.insert(events, "click", "button_a")

  let assert Ok(clicks) = duplicate_bag.lookup(events, key: "click")
  // clicks == ["button_a", "button_a"]

  let assert Ok(Nil) = duplicate_bag.close(events)
}
```

### Data persists across restarts

```gleam
import gleam/dynamic/decode
import slate/set

pub fn write() {
  let assert Ok(table) = set.open("data/state.dets",
    key_decoder: decode.string, value_decoder: decode.int)
  let assert Ok(Nil) = set.insert(table, "counter", 42)
  let assert Ok(Nil) = set.close(table)
}

pub fn read() {
  let assert Ok(table) = set.open("data/state.dets",
    key_decoder: decode.string, value_decoder: decode.int)
  let assert Ok(42) = set.lookup(table, key: "counter")
  let assert Ok(Nil) = set.close(table)
}
```

### Error handling

Most public operations return `Result(_, slate.DetsError)`.

`slate/set.update_counter` returns `Result(_, set.UpdateCounterError)` so it can
add the operation-specific `set.CounterValueNotInteger` case without widening
the shared `slate.DetsError` contract for unrelated APIs.

Match on the specific variants you expect in normal flows, and use the helper
functions when you want a stable code or a user-facing message:

```gleam
import slate
import slate/set

case set.lookup(table, key: "missing") {
  Ok(value) -> Ok(value)
  Error(slate.NotFound) -> Ok(default_value)
  Error(error) -> {
    let code = slate.error_code(error)
    let message = slate.error_message(error)
    // log code/message here
    Error(error)
  }
}
```

`UnexpectedError(detail)` is intended for diagnostics only; the detail string is
not a stable API contract, and `error_message` intentionally returns a generic
message for that variant.

When opening existing files, `Error(slate.NotADetsFile(context))` means the path is
readable but not a DETS file, and `Error(slate.NeedsRepair(context))` means the file was
not closed cleanly and you opened it with `NoRepair`.

File-related errors carry `FileErrorContext(path: Option(String), reason: String)`.
The path comes from OTP; `None` means OTP supplied no single filename. The reason
retains lower-level diagnostics, including nested OTP reasons. Both fields are
for trusted diagnostics only and can contain sensitive data. `error_code` and
`error_message` do not include these details and their outputs are unchanged.

**Breaking migration:** `FileNotFound`, `AlreadyOpen`, `AccessDenied`, `TypeMismatch`,
`NeedsRepair`, `NotADetsFile`, and `FileSizeLimitExceeded` now require context.
Change patterns such as `Error(slate.AccessDenied)` to
`Error(slate.AccessDenied(_))`, or bind `context` to inspect it. See the
[migration guide](docs/file-error-context-migration.md) for before/after examples,
including code that constructs errors. No DETS file migration is needed.

For `set.update_counter`, match `Error(set.CounterValueNotInteger)` directly and
unwrap shared table failures as `Error(set.TableError(error))`.

## API Overview

The three table types (`set`, `bag`, `duplicate_bag`) share a common core API:

| Function | Description |
|----------|-------------|
| `open(path, key_decoder, value_decoder)` | Open or create a table |
| `open_with(path, repair, key_decoder, value_decoder)` | Open with repair policy |
| `open_with_access(path, repair, access, key_decoder, value_decoder)` | Open with repair and access mode |
| `close(table)` | Close and flush to disk |
| `sync(table)` | Flush without closing |
| `with_table(path, repair, access, key_decoder, value_decoder, fn)` | Auto-closing callback with repair and access options |
| `insert(table, key, value)` | Insert a key-value pair |
| `insert_list(table, entries)` | Batch insert |
| `lookup(table, key)` | Get value(s) for key |
| `member(table, key)` | Check if key exists |
| `delete_key(table, key)` | Remove by key |
| `delete_object(table, key, value)` | Remove a specific key-value pair (`duplicate_bag` removes all exact duplicates) |
| `delete_all(table)` | Clear all entries |
| `to_list(table)` | Get all entries |
| `fold(table, acc, fn)` | Fold over entries |
| `size(table)` | Count entries |
| `info(table)` | Get the file size, object count, and absolute file path |

Use `info(table)` to retrieve the file path from an open handle:

```gleam
let assert Ok(info) = set.info(table)
info.file_path
```

**Migrating from 1.x:** `TableInfo` now has a `file_path: String` field.
Supply it when constructing the record. Include the third field, or use `..`,
when matching the constructor. Existing `info.file_size` and
`info.object_count` access remains valid.

`slate/set` also provides:

| Function | Description |
|----------|-------------|
| `insert_new(table, key, value)` | Insert if key is absent |
| `update_counter(table, key, amount)` | Atomic counter increment |

`slate/bag` also provides:

| Function | Description |
|----------|-------------|
| `insert_new(table, key, value)` | Reject an exact duplicate key-value pair (best-effort under concurrent shared access) |

The top-level `slate` module also provides:

| Function | Description |
|----------|-------------|
| `is_dets_file(path)` | Check if a file is a valid DETS file |
| `error_code(error)` | Stable machine-readable error code |
| `error_message(error)` | User-facing error message |

## Limitations

- **2 GB maximum file size** per table — a hard limit in DETS
- **No `ordered_set`** — DETS only supports `set`, `bag`, and `duplicate_bag`
- **Disk I/O** on every operation — for high-frequency reads, load into ETS at startup
- **Must close properly** — `with_table` closes on callback return and attempts cleanup on callback failure, otherwise ensure `close` is called
- **Bounded table name pool** — slate uses an internal bounded set of DETS table names to avoid unbounded atom growth. Opening too many distinct tables at once can fail with `TableNamePoolExhausted`; close tables when no longer needed
- **Erlang only** — DETS is a BEAM feature, no JavaScript target support

## Stability

slate follows [Semantic Versioning](https://semver.org/). The **public API** covered by semver guarantees consists of four modules:

- `slate` — shared types (`DetsError`, `FileErrorContext`, `AccessMode`, `RepairPolicy`, `TableInfo`) and helpers
- `slate/set` — set tables
- `slate/bag` — bag tables
- `slate/duplicate_bag` — duplicate bag tables

The Erlang FFI files (`slate_dets_ffi.erl`, `slate_with_table_ffi.erl`) are internal implementation details and are **not** part of the public API. They may change in any release without notice.

**Versioning policy:** patch releases contain bug fixes only, minor releases add backward-compatible features, and major releases may include breaking changes. The `error_code()` strings returned by `slate.error_code` are stable across minor and patch releases and are safe for programmatic matching (e.g., in error-handling logic or logging). The `error_message()` strings are human-readable and may change in any release.

See [CHANGELOG.md](CHANGELOG.md) for release history and upgrade notes, and the [GitHub Releases](https://github.com/tylerbutler/slate/releases) page for tagged versions.

## Related projects

- **[bravo](https://github.com/Michael-Mark-Edu/bravo)** — Comprehensive ETS (in-memory) bindings for Gleam
- **[shelf](https://github.com/tylerbutler/shelf)** — Persistent ETS tables backed by DETS, combining fast in-memory reads with durable storage

For details on the underlying storage engine, see the [Erlang DETS documentation](https://www.erlang.org/doc/apps/stdlib/dets.html).

## Development

See [DEV.md](DEV.md) for setup instructions, build tasks, and contribution guidelines.

## License

MIT