Packages

A Rust-backed AshVault key cache and AES-256-GCM cipher that hold key material outside the BEAM heap, so eviction can actually zero it.

Current section

Files

Jump to
Raw

README.md

# ash_vault_rustler

A Rust NIF providing an AshVault key cache and an AES-256-GCM cipher that hold key
material **outside the BEAM heap**, so that eviction can actually zero it.

Optional. The parent `ash_vault` has no Rust dependency and never will.

```elixir
def deps do
  [
    {:ash_vault, "~> 0.2.0"},
    {:ash_vault_rustler, "~> 0.2.0"}
  ]
end
```

```elixir
use AshVault.Vault,
  key_provider: MyApp.Keys,
  cache: [backend: AshVaultRustler.KeyCache, ttl: :timer.seconds(30)]
```

---

## What this buys

### The BEAM cannot zero memory, and that makes crypto-erasure eventual

AshVault's whole reason to exist is one sentence: destroying a customer's keys makes
their historical ciphertext permanently undecryptable. A key cache quietly weakens that.
"Evicting" a key from ETS drops a *reference*; the bytes survive in however many process
heaps read them, for however long the garbage collector takes, and there is no BEAM API
that overwrites them. Every BEAM crypto library shares this. Cloak does not attempt it —
zero hits for zero/wipe/scrub across its `lib/`, and it keeps every key in a named
`:protected` ETS table for the life of the process.

`AshVaultRustler.KeyCache` fixes exactly that, and only that:

| | `AshVault.KeyCaches.ETS` | `AshVaultRustler.KeyCache` |
|---|---|---|
| Eviction is synchronous | yes | yes |
| Eviction is observable (no later read can see it) | yes | yes |
| Generation fence against a racing repopulate | yes | yes |
| **Eviction overwrites the bytes** | **no** | **yes** |
| Key pages `mlock`ed out of swap | no | yes (best effort) |
| Needs a Rust toolchain or a precompiled artifact | no | yes |

Each cached key lives in a page-aligned allocation the crate owns, `mlock`ed into RAM,
and `Drop`ped with a volatile overwrite. `evict_scope/1` returns only after those writes
have happened.

### Opaque key handles

`AshVaultRustler.KeyProviders.Opaque` wraps any provider so it returns
`%AshVault.Key{}` handles instead of binaries, and `AshVaultRustler.Cipher` runs the AEAD
against a handle. The key never becomes an Elixir term during an encrypt or a decrypt.

### A cipher you can drop in without stranding a row

`AshVaultRustler.Cipher` uses the same id (`:aes_256_gcm_v1`), the same 12-byte nonce,
the same 16-byte tag and the same AAD as `AshVault.Ciphers.AES.GCM`. A value written by
either decrypts with the other; the test suite asserts both directions across sizes from
0 bytes to 200 KB.

---

## What this does **not** buy

Be precise about this, because the temptation to overclaim here is enormous.

**The level-1 cache does not eliminate the per-operation copy.**
`AshVault.KeyProviders.Cached.get_key/3` returns a *binary* to the BEAM. A copy of the
key lands on an Elixir process heap for every single encrypt and decrypt, and that copy
is the garbage collector's business, not ours. This package bounds the lifetime of the
**authoritative** copy. Eliminating the per-operation one needs the opaque-handle path,
which is a separate opt-in.

**Even the opaque path has a boundary.** The wrapped provider returns a binary — that is
its contract — so the key bytes pass through the BEAM once, on the way into the handle.
What goes away is the *repeat*: with a cache in front, key bytes exist as an Elixir term
once per cache miss rather than once per field read.

**The TTL is the erasure SLA, but only for erasures that do not go through the vault.**
`AshVault.KeyProviders.Cached.destroy/2` evicts synchronously on every connected node
before it returns, and fails if any node does not acknowledge — so an erasure performed
through the vault is immediate. An erasure performed **out of band** is not: a
`mix ash_vault.destroy_keys` in an unclustered node, an operator deleting a transit key
in OpenBao by hand, another service revoking it. None of those call `evict_scope/1`, and
the cache serves the erased scope until the entry expires. **That TTL is the number you
can write in a DPA**, and it defaults to 30 seconds.

**`mlock` is best effort, and it is charged per page.** Each cached key gets its own
page, because `mlock`/`munlock` act on whole pages and locking a 32-byte buffer inside a
shared page would unlock whatever else lives there when it is freed. So:

> A 32-byte key costs **4 KiB of locked memory**, and `:max_bytes` counts key bytes, not
> pages. The defaults (`max_entries: 1_024`, `max_bytes: 1_048_576`) can lock up to
> **4 MiB** while the reported byte total reads 32 KiB. **With this backend, size
> `:max_entries` against `RLIMIT_MEMLOCK`**, not against `:max_bytes`.

A default container `ulimit -l` is often 64 KiB — sixteen pages — so locking starts
failing after roughly sixteen cached keys on a stock host. That is not fatal: the failure
is logged once, loudly, after the first write that observes it, and caching continues,
because a key that might reach swap beats a node that refuses to decrypt. Check
`AshVaultRustler.mlock_status/0` in production and alert on a non-zero `:failed`.

**Nothing here defends against a compromised application.** An attacker with code
execution can ask the cache for a key exactly as your app does. See the parent's
[threat model](https://hexdocs.pm/ash_vault/threat-model.html).

---

## The rule that matters most

> **Never cache the absence of a tombstone.**

`{:error, :destroyed}` is cached forever: destruction is monotonic, so a cached tombstone
can only be right, and caching it makes reads fail *closed* even when the provider is
unreachable. The absence of a tombstone is never cached; nor is `{:error, :not_found}`,
nor a transport failure.

Caching `{:ok, key}` is, strictly, an implicit cache of "not destroyed", and pretending
otherwise would be the fifth fail-open tombstone read in this project's history. Two
mechanisms contain it: synchronous cluster-wide eviction on the destroy path, and a short
TTL as the floor for everything else. The policy lives in
`AshVault.KeyProviders.Cached` — in Elixir, where it can be read and reviewed — and the
Rust crate holds no opinion about tombstones at all.

---

## Configuration

### The cipher: set **both** `:cipher` and `:ciphers`

This is the easiest thing to get wrong, and it is a consequence of the wire compatibility
being real:

```elixir
# governs ENCRYPTION
use AshVault.Vault, key_provider: MyApp.Keys, cipher: AshVaultRustler.Cipher

# governs DECRYPTION — required
config :ash_vault, :ciphers, %{"aes_256_gcm_v1" => AshVaultRustler.Cipher}
```

`AshVault.Vault.Runtime.decrypt!/3` resolves the cipher from the **envelope**, through
`AshVault.Cipher.registry/0`, not from the vault's `:cipher` option — deliberately, so
that values keep decrypting after the default cipher changes. Since this cipher shares
the built-in id, a vault configured with only `:cipher` will encrypt in Rust and decrypt
in Elixir. Harmless with a binary key; **impossible** with an opaque handle, which is
where you find out.

### The cache

```elixir
use AshVault.Vault,
  key_provider: MyApp.Keys,
  cache: [
    backend: AshVaultRustler.KeyCache,
    ttl: :timer.seconds(30),          # current key; also the erasure SLA
    historical_ttl: :timer.seconds(30),
    max_entries: 1_024,
    max_bytes: 1_048_576,
    cluster: true,                    # evict on every connected node, fail if any doesn't ack
    evict_timeout: 5_000
  ]
```

Then supervise the generated provider — it owns the cache:

```elixir
children = [MyApp.Vault.CachedKeyProvider]
```

If it is not started, every read goes to the provider. Slower, and correct.

### Opaque keys

```elixir
defmodule MyApp.OpaqueKeys do
  use AshVaultRustler.KeyProviders.Opaque, provider: AshVault.KeyProviders.OpenBao
end

use AshVault.Vault, key_provider: MyApp.OpaqueKeys, cipher: AshVaultRustler.Cipher
config :ash_vault, :ciphers, %{"aes_256_gcm_v1" => AshVaultRustler.Cipher}
```

A cipher that cannot take a handle raises `AshVault.Errors.OpaqueKeyUnsupported` naming
both modules. There is deliberately no fallback that unwraps the handle — that would put
the key back on the BEAM heap, silently, which is the one thing the handle prevents.

---

## Scheduler discipline, and the numbers behind it

Measured on this host (AES-NI, `mix run bench/scheduler_bench.exs`):

| plaintext | normal scheduler | dirty CPU scheduler | OTP `:crypto` |
|---:|---:|---:|---:|
| 16 B | 1.20 µs | 1.73 µs | 1.90 µs |
| 64 B | 0.68 µs | — | — |
| 1 KiB | 1.70 µs | 3.04 µs | 2.72 µs |
| 4 KiB | 3.92 µs | — | — |
| 16 KiB | 13.6 µs | — | — |
| 64 KiB | 46.4 µs | 80.9 µs | 42.0 µs |
| 256 KiB | 319 µs | — | — |
| 1 MiB | 1529 µs | 1055 µs | 715 µs |
| 4 MiB | 5958 µs | 5392 µs | — |

Cache operations: `fetch` 4.07 µs (ETS backend: 3.64 µs), `evict_scope` 0.15 µs.

**Threshold: 64 KiB.** The BEAM's guidance is ~1 ms per NIF call; on this machine that is
crossed around 700 KiB, so 64 KiB carries a ~20x margin. It is not set higher because AES
without hardware acceleration is roughly an order of magnitude slower — 64 KiB still
lands under 1 ms on such a CPU, 256 KiB would not. Dirty dispatch costs a fixed ~0.5–1 µs,
which is why the threshold is not simply zero. Every realistic encrypted column is far
below 64 KiB, so in practice AshVault field operations never pay the dispatch cost.

Cache reads are a `DashMap` lookup — sub-microsecond of actual work — and stay on a
normal scheduler unconditionally.

Override for unusual hardware:

```elixir
config :ash_vault_rustler, dirty_threshold_bytes: 262_144
```

Note the honest result in that table: **for binary keys, this cipher is not faster than
OTP's**, and at 1 MiB it is slower. Performance is not the reason to adopt it; the zeroing
cache and the opaque key path are.

---

## Safety

* **No hand-rolled crypto.** RustCrypto's `aes-gcm`, which has had a third-party audit
  (NCC Group, funded by MobileCoin) and uses AES-NI/CLMUL where available.
* **No NIF can abort the VM.** Every NIF returns a value or an error term. Rustler wraps
  each body in `catch_unwind`, so a panic becomes an error in the *calling process*;
  `test/panic_test.exs` proves that on this OTP build rather than asserting it.
* **`unsafe` appears in exactly one module** (`native/ashvault_nif/src/secret.rs`), for
  the page-aligned allocation and its zeroing drop. Every block carries a `SAFETY:`
  comment, because a bug there takes down the node rather than one process.
* **Constant-time comparison** (`subtle`) for anything whose result is a fact about key
  material.
* **Truncated tags and short nonces are refused** before the key reaches the AEAD — the
  same control, for the same stated reason, as the Elixir cipher. OTP's
  `:crypto.crypto_one_time_aead/7` accepts a truncated GCM tag and compares only its
  leading bytes, which made a ≤256-guess forgery possible through an attacker-controlled
  `tag_len` in the envelope.
* **Unauthenticated plaintext is zeroed** rather than returned when a tag check fails.

### Known coverage gap

`AshVault.KeyProviders.Cached.evict_scope/2` fans out to `Node.list(:connected)`, which is
empty on a single node, so the `:erpc.multicall/5` call itself is never executed by the
test suite — there is no cluster here to stand one up against. What *is* tested is the
decision the fan-out feeds:
`AshVault.KeyProviders.Cached.classify_evictions/2` is public and unit-tested against
every shape `:erpc` produces (`{:ok, :ok}`, `{:ok, {:error, _}}`, `{:error, {:erpc,
:noconnection}}`, `{:error, {:erpc, :timeout}}`, `{:exit, _}`, `{:throw, _}`, an
exception result, and garbage), asserting that **anything other than a positive
`{:ok, :ok}` is a failure** and therefore makes `destroy/1` return an error. The wiring
between `multicall` and that function is four lines and is not covered.

---

## Distribution

Consumers download a precompiled NIF through `rustler_precompiled` and verify it against
`checksum-Elixir.AshVaultRustler.Native.exs`. A Rust toolchain is needed only when
`ASH_VAULT_RUSTLER_BUILD=1` forces a source build. The setting is read at compile time;
run `mix deps.compile ash_vault_rustler --force` after changing it.

Artifacts cover Linux GNU and musl, plus macOS, on x86_64 and aarch64. Each uses NIF API
2.16, supported by OTP 24 and later. The package's Elixir requirement still applies.

### Releasing both packages

1. Update the versions in both Mix projects and the native crate, plus the changelog.
   Run the local suites and the release workflow with `dry_run=true` before merging.
2. Push the matching `v<version>` tag, for example `v0.2.0`. The release workflow tests
   both packages, builds all six NIF artifacts and publishes `ash_vault` to Hex.
3. The workflow generates checksums from the published artifacts, includes them in the
   `ash_vault_rustler` Hex package, and opens a PR to retain the checksum file in Git.
   Merge that PR after checking the artifact names and checksums.
4. Verify both Hex releases and compile the published Rustler package from a clean
   consumer without forcing a source build. Verify that all six artifacts exist under
   `ash_vault_rustler-v<version>`.

Never rebuild or overwrite artifacts of an existing release: a different binary would
invalidate the checksums already shipped to consumers. The release workflow refuses
partial artifact releases rather than mixing builds.

---

## Building and testing

Everything needs a linker. In this repo:

```sh
nix develop --command bash -c 'cd ash_vault_rustler && mix test'
nix develop --command bash -c 'cd ash_vault_rustler/native/ashvault_nif && cargo test'
nix develop --command bash -c 'cd ash_vault_rustler/native/ashvault_nif && cargo clippy --all-targets -- -D warnings'
nix develop --command bash -c 'cd ash_vault_rustler && mix run bench/scheduler_bench.exs'
```

The Elixir suite runs the parent's own `AshVault.KeyProvider` contract cases against
`AshVault.KeyProviders.Cached` with both backends. `test/parent_support/` holds a symlink
to the parent's `key_provider_cases.ex` — the same file, not a copy, because a copied
contract suite drifts and a drifted contract suite is worse than none.