Current section
Files
Jump to
Current section
Files
README.md
# LeanLmdb
[](https://hex.pm/packages/lean_lmdb)
[](https://hexdocs.pm/lean_lmdb)
[](https://github.com/mindreframer/lean_lmdb/actions/workflows/ci.yml)
[](https://github.com/mindreframer/lean_lmdb/blob/main/LICENSE)
Fast, safe, and ergonomic LMDB storage for Elixir, powered by Rust.
## Installation
Add the Hex package to `mix.exs`:
```elixir
def deps do
[{:lean_lmdb, "~> 0.2.0"}]
end
```
## Quickstart
All names, keys, values, CAS payloads, and continuation tokens are binaries;
there is no Elixir-term serialization.
```elixir
path =
Path.join(
System.tmp_dir!(),
"lean_lmdb_example_#{System.unique_integer([:positive, :monotonic])}"
)
{:ok, env} = LeanLmdb.open(path, map_size: 64 * 1024 * 1024)
{:ok, users} = LeanLmdb.create_database(env, "users")
{:ok, cache} = LeanLmdb.create_database(env, "cache")
:not_found = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.put(users, "user:1", <<0, 255>>)
{:ok, <<0, 255>>} = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.delete(users, "user:1")
:ok = LeanLmdb.write_batch(env, [
{:put, users, "user:1", "Ada"},
{:put, users, "user:2", "Grace"},
{:delete, cache, "stale"}
])
:ok = LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Lin"})
{:conflict, "Lin"} =
LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Other"})
{:ok, first, token} = LeanLmdb.scan(users, prefix: "user:", limit: 2)
{:ok, second, :done} =
LeanLmdb.scan(users, prefix: "user:", limit: 2, continuation: token)
true = first ++ second == [{"user:1", "Ada"}, {"user:2", "Grace"}, {"user:3", "Lin"}]
```
Use a unique path in real tests and remove it only after all handles are
unreachable. There is no force-close API.
## Public result contracts
| Function | Success / non-error result | Error result |
| --- | --- | --- |
| `native_version/0` | `{"lean_lmdb", version}` | none |
| `open/2` | `{:ok, environment}` | `{:error, reason}` |
| `create_database/2` | `{:ok, database}` | `{:error, reason}` |
| `open_database/2` | `{:ok, database}` | `{:error, reason}` |
| `get/2` | `{:ok, binary}` or `:not_found` | `{:error, reason}` |
| `put/3`, `delete/2`, `sync/1` | `:ok` | `{:error, reason}` |
| `write_batch/2` | `:ok` after one atomic commit | `{:error, reason}` |
| `compare_exchange/4` | `:ok`, `{:conflict, :missing}`, or `{:conflict, binary}` | `{:error, reason}` |
| `scan/2` | `{:ok, [{key, value}], :done \| token}` | `{:error, reason}` |
| `info/1` | environment/database information map | `{:error, :invalid_environment}` |
| `clear_stale_readers/1` | `{:ok, non_neg_integer}` | `{:error, reason}` |
| `registry_info/0` | `%{live: n, stale: n, total: n}` | none |
See the generated API documentation for each function's validation rules and
stable reasons. Native failures return atoms, never platform error prose.
## Lifecycle and transaction containment
Within one BEAM OS process, compatible opens of the same canonical path share
native environment state. Opaque `Environment` handles and immutable
`Database` handles may remain live across calls; a database handle retains the
shared environment state. No heed transaction or cursor is an Elixir resource.
Every transaction/cursor starts and ends in one dirty-I/O NIF call.
Reads copy values into BEAM-owned binaries before ending the transaction. Scans
also copy every key, value, and token. LeanLmdb is therefore intentionally not
an end-to-end zero-copy API. NIF hot upgrade while native handles exist is not
supported; restart the BEAM when replacing the native library.
## Environment defaults, map sizing, and durability
`open/2` defaults to a 64 MiB map, 16 named databases, 126 readers, writable
access, `create: true`, `durability: :sync`, and `fixed_map: false`. `map_size`
is fixed, page-size aligned, and limited to 1 MiB..1 TiB. `max_dbs` is 1..1024
and `max_readers` is 1..4096. LeanLmdb does not resize maps or retry failed
operations. Size the map and reader/database limits before opening.
Durability is selected per environment:
```elixir
# Fully synchronized LMDB commits (default).
{:ok, durable} = LeanLmdb.open(path, durability: :sync)
# Preserve ACI, but a system crash may undo the latest transaction.
{:ok, metadata_deferred} = LeanLmdb.open(path, durability: :no_meta_sync)
# Do not flush on each commit; periodically force a durable checkpoint.
{:ok, derived} = LeanLmdb.open(path, durability: :no_sync)
:ok = LeanLmdb.sync(derived)
```
`durability: :no_sync` can lose recent commits and can corrupt the database on a
system crash when the filesystem does not preserve write order. This mode is
appropriate only when the store is disposable and can be rebuilt from an
authoritative source. `sync/1` calls forced `mdb_env_sync`; applications can run
it from a supervised timer to implement delayed group durability.
`fixed_map: true` exposes experimental `MDB_FIXEDMAP`. The recorded virtual
address must be available in every process opening the environment; open can
fail under ASLR/address-space conflicts. `WRITE_MAP`, `MAP_ASYNC`, and `NO_LOCK`
remain unavailable. Common capacity/operational results include `:map_full`,
`:map_resized`, `:readers_full`, `:transaction_full`, `:out_of_memory`, and
`:io_error`.
## CRUD, atomic mutation, and single-writer behavior
Keys are non-empty and at most LMDB's runtime maximum (currently bounded by 511
bytes here). Values may be empty and can contain arbitrary bytes. `put/3`
overwrites; `delete/2` is idempotent. A failed write aborts its transaction.
Batches contain at most 10,000 operations and 64 MiB of aggregate key/value
input. All database handles must belong to the supplied environment. The full
batch is validated before one write transaction, applied in list order, and
committed once. CAS distinguishes `:missing` from `<<>>` and returns the exact
copied current value on conflict.
LMDB allows concurrent readers but exactly one active writer per environment,
including across OS processes. A dirty-I/O call can wait for that writer; short
batches improve useful work per serialized commit without changing durability.
## Bounded scans and tokens
`scan/2` is forward-only in raw binary-key order. Choose `prefix: binary` or a
range using `from`, `to`, `from_inclusive` (default `true`), and `to_inclusive`
(default `false`). Do not mix prefix and range options. `limit` defaults to 100
and is 1..10,000. `max_bytes` defaults to 1 MiB and is 1..64 MiB, counting key
plus value bytes. If the first eligible row is larger than the selected budget,
the call returns `{:error, :row_too_large}`; page output never exceeds the byte
or row bound. Because 64 MiB is the maximum budget, a key/value row larger than
64 MiB cannot be traversed by `scan/2`; use smaller/chunked values or a different
store design.
A token is an opaque, versioned, process-local binary with a keyed integrity
tag, not a cryptographic credential. It contains no pointer or cursor. It binds
to the native environment resource ID, database name, and original bounds;
`limit` and `max_bytes` may change. Modified/mismatched tokens return
`:invalid_continuation`; tokens from a released environment return
`:stale_continuation`. Tokens do not survive a BEAM restart.
Each page has a fresh read transaction, so pages do **not** share one snapshot.
Resume is exclusive after the last emitted key. Writes between pages can add,
change, or remove later rows; insertions at or before the last emitted key will
not appear. An unchanged database traverses exactly once without duplicates.
## Multi-process operation and stale readers
Each BEAM OS process has its own registry and heed handle. LMDB coordinates via
its data and lock files. Configure all processes with the same map/database/
reader limits; first-opener and process-local details otherwise can produce
`:map_resized` or incompatible behavior. A commit becomes visible to another
process when that process starts a later transaction.
An abrupt writer death cannot publish a partial LMDB transaction, but recovery
is not corruption repair. A killed reader can leave a stale lock-table slot,
causing file growth or eventually `:readers_full`. `clear_stale_readers/1` uses
LMDB's reader check to remove dead-process slots only and is safe to repeat.
Never delete the lock file while any process has the environment open.
The deterministic integration suite uses independent BEAM processes, explicit
file barriers, bounded polling, and Unix hard kills; it does not infer recovery
from sleeps.
## Filesystem and operational caveats
Use only a stable local filesystem. NFS, SMB, distributed/network mounts, and
moving, replacing, truncating, or externally editing open environment files are
unsupported. Available disk and virtual address space and LMDB/platform limits
still apply. Plan and test backups and recovery according to your durability
requirements. LeanLmdb has no built-in backup, replication, corruption repair,
distributed consensus, automatic resize, or native hot-upgrade protocol.
The API fits workloads where data is binary, bounded scans are sufficient, and
serialized short writes are acceptable. It is a poor fit when values need
streaming/chunking, remote storage is required, or sustained parallel write
transactions are required.
## Benchmarking
A comprehensive Benchee suite covers point reads, synchronized mutations,
atomic batch sizes, bounded and complete scans, mixed traffic, parallel
contention, fresh projection rebuilds, and multiple LMDB map/reader/database
configurations. It produces p50/p95/p99 statistics, raw JSON, a reproducibility
manifest, and interactive HTML charts.
```sh
mix deps.get
mix run bench/run.exs -- --profile quick
mix run bench/run.exs -- --configs all --workloads reads --parallel 8
```
See the [benchmarking guide](https://github.com/mindreframer/lean_lmdb/blob/main/BENCHMARKING.md)
for profiles, all options, workload units, chart interpretation, and warm/cold-cache methodology. Generated reports are
written to `bench/output/` and are not performance gates.
## Source development
Install the versions above, fetch both dependency sets once, then run the single
quality gate:
```sh
mix deps.get
cargo +1.91.0 fetch --manifest-path native/lean_lmdb_nif/Cargo.toml --locked
bin/qa_check.sh
```
After dependencies exist, the gate globally forces source fallback, forces Hex
and Cargo offline, builds docs with warnings as errors, runs Elixir tests twice,
runs all-feature Rust lint and tests, and performs a release build without
test-support. It also checks an optional checksum manifest is packaged unchanged,
compiles the unpacked package from its included Rust source, then runs its
package-built NIF through `native_version/0` and isolated CRUD. Every Mix/Cargo command has a portable
per-command deadline (900 seconds by default, configurable with
`QA_COMMAND_TIMEOUT_SECONDS`).
## Maintainer release flow
Precompiled publication is explicit and draft-first. The workflow never runs on
a tag push, so it cannot recurse. For a version already set in `mix.exs` and the
native `Cargo.toml`:
```sh
git push origin main
gh workflow run precompiled-release.yml --ref main -f publish=true
gh run list --workflow precompiled-release.yml --limit 1
gh run watch RUN_ID --exit-status
gh release view v0.2.0 --json tagName,targetCommitish,isDraft,isPrerelease,assets
mix rustler_precompiled.download LeanLmdb.Native --all --print
```
The matrix uses target-aware Cargo caches. Every archive is directly loaded on
a matching runtime before aggregation; Linux musl uses Alpine. The aggregator
accepts exactly seven expected archives, creates a draft from the dispatched
commit, verifies GitHub's SHA-256 asset digests, then publishes. A failure after
draft creation removes only the draft and tag created by that run.
The generated `checksum-Elixir.LeanLmdb.Native.exs` is mandatory for the normal
precompiled path. Review and commit it, rerun `bin/qa_check.sh`, then push and
wait for the CI precompiled-consumer matrix. Those jobs compile the unpacked Hex
package from clean downstream projects with `cargo` and `rustc` replaced by
failing shims. Use `gh run cancel RUN_ID` immediately for an obsolete or known-
bad run, inspect failures with `gh run view RUN_ID --log-failed`, and retry only
after committing the fix.