lean_lmdb

# LeanLmdb

A lean, binary-first Elixir wrapper for LMDB, built with Rustler 0.38,
RustlerPrecompiled 0.8, and [heed](https://github.com/meilisearch/heed) 0.22.1. It is intended for bounded,
rebuildable local derived stores—not as a source-of-truth event store.

## Installation

Add the Hex package to `mix.exs`:

```elixir
def deps do
  [{:lean_lmdb, "~> 0.1.0"}]
end
```

Installation downloads the matching NIF from the
[`v0.1.0` GitHub release](https://github.com/mindreframer/lean_lmdb/releases/tag/v0.1.0)
and verifies its SHA-256 digest against the packaged
`checksum-Elixir.LeanLmdb.Native.exs` manifest. The package retains locked Rust
source as an explicit fallback for unsupported targets and local source builds.

NIF 2.16 artifacts are built and raw-smoked for this exact matrix:

| OS / libc | Architecture | Rust target |
| --- | --- | --- |
| macOS | Apple Silicon | `aarch64-apple-darwin` |
| macOS | Intel 64-bit | `x86_64-apple-darwin` |
| Linux glibc | ARM64 | `aarch64-unknown-linux-gnu` |
| Linux musl | ARM64 | `aarch64-unknown-linux-musl` |
| Linux glibc | x86-64 | `x86_64-unknown-linux-gnu` |
| Linux musl | x86-64 | `x86_64-unknown-linux-musl` |
| Windows | x86-64 | `x86_64-pc-windows-msvc` |

Set `LEAN_LMDB_BUILD=1` (`true`, `yes`, and `on` are also accepted,
case-insensitively) at compile time to force the packaged-source fallback.
`MIX_ENV=test` always forces source and enables test-support NIFs. For global QA
across all RustlerPrecompiled dependencies, set
`RUSTLER_PRECOMPILED_FORCE_BUILD_ALL=true`. Consumers forcing fallback must also
make the package's optional build-only dependency available, for example
`{:rustler, "~> 0.38.0", runtime: false}`. Source builds require Elixir 1.20,
Erlang/OTP 29, Cargo/Rust 1.91, and the platform C compiler/linker. Linux musl
builds dynamically link the musl CRT and are validated in matching Alpine
Elixir containers. Runtime environments must use a local filesystem.

## Quickstart

All names, keys, values, CAS payloads, and continuation tokens are binaries;
there is no Elixir-term serialization.

```elixir
path =
  Path.join(
    System.tmp_dir!(),
    "lean_lmdb_example_#{System.unique_integer([:positive, :monotonic])}"
  )
{:ok, env} = LeanLmdb.open(path, map_size: 64 * 1024 * 1024)
{:ok, users} = LeanLmdb.create_database(env, "users")
{:ok, cache} = LeanLmdb.create_database(env, "cache")

:not_found = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.put(users, "user:1", <<0, 255>>)
{:ok, <<0, 255>>} = LeanLmdb.get(users, "user:1")
:ok = LeanLmdb.delete(users, "user:1")

:ok = LeanLmdb.write_batch(env, [
  {:put, users, "user:1", "Ada"},
  {:put, users, "user:2", "Grace"},
  {:delete, cache, "stale"}
])

:ok = LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Lin"})
{:conflict, "Lin"} =
  LeanLmdb.compare_exchange(users, "user:3", :missing, {:put, "Other"})

{:ok, first, token} = LeanLmdb.scan(users, prefix: "user:", limit: 2)
{:ok, second, :done} =
  LeanLmdb.scan(users, prefix: "user:", limit: 2, continuation: token)
true = first ++ second == [{"user:1", "Ada"}, {"user:2", "Grace"}, {"user:3", "Lin"}]
```

Use a unique path in real tests and remove it only after all handles are
unreachable. There is no force-close API.

## Public result contracts

| Function | Success / non-error result | Error result |
| --- | --- | --- |
| `native_version/0` | `{"lean_lmdb", version}` | none |
| `open/2` | `{:ok, environment}` | `{:error, reason}` |
| `create_database/2` | `{:ok, database}` | `{:error, reason}` |
| `open_database/2` | `{:ok, database}` | `{:error, reason}` |
| `get/2` | `{:ok, binary}` or `:not_found` | `{:error, reason}` |
| `put/3`, `delete/2` | `:ok` | `{:error, reason}` |
| `write_batch/2` | `:ok` after one atomic commit | `{:error, reason}` |
| `compare_exchange/4` | `:ok`, `{:conflict, :missing}`, or `{:conflict, binary}` | `{:error, reason}` |
| `scan/2` | `{:ok, [{key, value}], :done \| token}` | `{:error, reason}` |
| `info/1` | environment/database information map | `{:error, :invalid_environment}` |
| `clear_stale_readers/1` | `{:ok, non_neg_integer}` | `{:error, reason}` |
| `registry_info/0` | `%{live: n, stale: n, total: n}` | none |

See the generated API documentation for each function's validation rules and
stable reasons. Native failures return atoms, never platform error prose.

## Lifecycle and transaction containment

Within one BEAM OS process, compatible opens of the same canonical path share
native environment state. Opaque `Environment` handles and immutable
`Database` handles may remain live across calls; a database handle retains the
shared environment state. No heed transaction or cursor is an Elixir resource.
Every transaction/cursor starts and ends in one dirty-I/O NIF call.

Reads copy values into BEAM-owned binaries before ending the transaction. Scans
also copy every key, value, and token. LeanLmdb is therefore intentionally not
an end-to-end zero-copy API. NIF hot upgrade while native handles exist is not
supported; restart the BEAM when replacing the native library.

## Environment defaults, map sizing, and durability

`open/2` defaults to a 64 MiB map, 16 named databases, 126 readers, writable
access, and `create: true`. `map_size` is fixed, page-size aligned, and limited
to 1 MiB..1 TiB. `max_dbs` is 1..1024 and `max_readers` is 1..4096. LeanLmdb
does not resize maps, retry failed operations, or expose relaxed LMDB flags.
Size the map and reader/database limits for the whole workload before opening.

Writes use heed/LMDB's default synchronized durability flags: no `NO_SYNC`,
`NO_META_SYNC`, `WRITE_MAP`, or `NO_LOCK` mode is exposed. A successful write
means LMDB committed it under those defaults; storage hardware, OS, and
filesystem guarantees still apply. There is no separate flush API. Common
capacity/operational results include `:map_full`, `:map_resized`,
`:readers_full`, `:transaction_full`, `:out_of_memory`, and `:io_error`.

## CRUD, atomic mutation, and single-writer behavior

Keys are non-empty and at most LMDB's runtime maximum (currently bounded by 511
bytes here). Values may be empty and can contain arbitrary bytes. `put/3`
overwrites; `delete/2` is idempotent. A failed write aborts its transaction.

Batches contain at most 10,000 operations and 64 MiB of aggregate key/value
input. All database handles must belong to the supplied environment. The full
batch is validated before one write transaction, applied in list order, and
committed once. CAS distinguishes `:missing` from `<<>>` and returns the exact
copied current value on conflict.

LMDB allows concurrent readers but exactly one active writer per environment,
including across OS processes. A dirty-I/O call can wait for that writer; short
batches improve useful work per serialized commit without changing durability.

## Bounded scans and tokens

`scan/2` is forward-only in raw binary-key order. Choose `prefix: binary` or a
range using `from`, `to`, `from_inclusive` (default `true`), and `to_inclusive`
(default `false`). Do not mix prefix and range options. `limit` defaults to 100
and is 1..10,000. `max_bytes` defaults to 1 MiB and is 1..64 MiB, counting key
plus value bytes. If the first eligible row is larger than the selected budget,
the call returns `{:error, :row_too_large}`; page output never exceeds the byte
or row bound. Because 64 MiB is the maximum budget, a key/value row larger than
64 MiB cannot be traversed by `scan/2`; use smaller/chunked values or a different
store design.

A token is an opaque, versioned, process-local binary with a keyed integrity
tag, not a cryptographic credential. It contains no pointer or cursor. It binds
to the native environment resource ID, database name, and original bounds;
`limit` and `max_bytes` may change. Modified/mismatched tokens return
`:invalid_continuation`; tokens from a released environment return
`:stale_continuation`. Tokens do not survive a BEAM restart.

Each page has a fresh read transaction, so pages do **not** share one snapshot.
Resume is exclusive after the last emitted key. Writes between pages can add,
change, or remove later rows; insertions at or before the last emitted key will
not appear. An unchanged database traverses exactly once without duplicates.

## Multi-process operation and stale readers

Each BEAM OS process has its own registry and heed handle. LMDB coordinates via
its data and lock files. Configure all processes with the same map/database/
reader limits; first-opener and process-local details otherwise can produce
`:map_resized` or incompatible behavior. A commit becomes visible to another
process when that process starts a later transaction.

An abrupt writer death cannot publish a partial LMDB transaction, but recovery
is not corruption repair. A killed reader can leave a stale lock-table slot,
causing file growth or eventually `:readers_full`. `clear_stale_readers/1` uses
LMDB's reader check to remove dead-process slots only and is safe to repeat.
Never delete the lock file while any process has the environment open.

The deterministic integration suite uses independent BEAM processes, explicit
file barriers, bounded polling, and Unix hard kills; it does not infer recovery
from sleeps.

## Filesystem and operational caveats

Use only a stable local filesystem. NFS, SMB, distributed/network mounts, and
moving, replacing, truncating, or externally editing open environment files are
unsupported. Available disk and virtual address space and LMDB/platform limits
still apply. Keep backups and the authoritative event/source data separately;
LeanLmdb has no backup, replication, corruption repair, distributed consensus,
automatic resize, or native hot-upgrade protocol.

The API fits a derived store when data is binary, locally rebuildable, bounded
scans are sufficient, and serialized short writes are acceptable. It is a poor
fit when the database is authoritative, values need streaming/chunking, remote
storage is required, or sustained parallel write transactions are required.

## Source development

Install the versions above, fetch both dependency sets once, then run the single
quality gate:

```sh
mix deps.get
cargo +1.91.0 fetch --manifest-path native/lean_lmdb_nif/Cargo.toml --locked
bin/qa_check.sh
```

After dependencies exist, the gate globally forces source fallback, forces Hex
and Cargo offline, builds docs with warnings as errors, runs Elixir tests twice,
runs all-feature Rust lint and tests, and performs a release build without
test-support. It also checks an optional checksum manifest is packaged unchanged,
compiles the unpacked package from its included Rust source, then runs its
package-built NIF through `native_version/0` and isolated CRUD. Every Mix/Cargo command has a portable
per-command deadline (900 seconds by default, configurable with
`QA_COMMAND_TIMEOUT_SECONDS`).

## Maintainer release flow

Precompiled publication is explicit and draft-first. The workflow never runs on
a tag push, so it cannot recurse. For a version already set in `mix.exs` and the
native `Cargo.toml`:

```sh
git push origin main
gh workflow run precompiled-release.yml --ref main -f publish=true
gh run list --workflow precompiled-release.yml --limit 1
gh run watch RUN_ID --exit-status

gh release view v0.1.0 --json tagName,targetCommitish,isDraft,isPrerelease,assets
mix rustler_precompiled.download LeanLmdb.Native --all --print
```

The matrix uses target-aware Cargo caches. Every archive is directly loaded on
a matching runtime before aggregation; Linux musl uses Alpine. The aggregator
accepts exactly seven expected archives, creates a draft from the dispatched
commit, verifies GitHub's SHA-256 asset digests, then publishes. A failure after
draft creation removes only the draft and tag created by that run.

The generated `checksum-Elixir.LeanLmdb.Native.exs` is mandatory for the normal
precompiled path. Review and commit it, rerun `bin/qa_check.sh`, then push and
wait for the CI precompiled-consumer matrix. Those jobs compile the unpacked Hex
package from clean downstream projects with `cargo` and `rustc` replaced by
failing shims. Use `gh run cancel RUN_ID` immediately for an obsolete or known-
bad run, inspect failures with `gh run view RUN_ID --log-failed`, and retry only
after committing the fix.