Packages

Embedded Trellis: streaming transforms maintained in your own Postgres, run from an Elixir app through a precompiled Rust NIF.

Current section

Files

Jump to
trellis_pg README.md
Raw

README.md

# Trellis for Elixir

Embedded Trellis for Elixir apps: a [Rustler](https://github.com/rusterlium/rustler)
NIF over the `trellis` crate's `BlockingTrellis`, so a Phoenix app can define
and run streaming transforms without deploying a separate service. The design
is [ADR-0010](https://github.com/salesforce-misc/trellis/blob/main/docs/decisions/0010-embeddable-clients.md).

## Installation

The Hex package is `trellis_pg` (Hex's `trellis` is taken); the modules are
`Trellis.*`, and the OTP application is `:trellis_pg`.

```elixir
def deps do
  [{:trellis_pg, "~> 0.2.0"}]
end
```

Installing needs no Rust: compiling the package downloads the NIF built for
the machine from the release's GitHub page and checks it against the SHA-256
the package ships. Builds exist for

- Linux, glibc 2.17 or newer: `x86_64` and `aarch64`
- Linux, musl (Alpine): `x86_64`
- macOS: Apple silicon and Intel

On any other target, or to build it yourself, build the NIF from source. It
needs a Rust toolchain (the version in the repository's `rust-toolchain.toml`)
and Rustler in the host's own deps:

```elixir
{:rustler, "~> 0.38.0", runtime: false}
```

```sh
TRELLIS_PG_BUILD=1 mix deps.compile trellis_pg --force
```

The choice is made when `trellis_pg` compiles, so `--force` it after changing
the variable. `config :rustler_precompiled, :force_build, trellis_pg: true`
does the same from config. Cargo fetches the `trellis` crate from this
repository at the released commit.

The NIF is built without the `trellis` crate's `otlp` feature, so neither
path pulls in an OpenTelemetry or gRPC stack. The engine's log lines reach
`Logger` through `Trellis.LogBridge`.

## Usage

The surface mirrors the Rust crate's `BlockingTrellis`. Every function has a bang variant.

- `connect/1`, `migrate/1`, `config/1`, `shutdown/1`: the handle's lifecycle.
- `apply/2`: any statement of Trellis's grammar (`TRANSFORM`,
  `PAUSE TRANSFORM`, `RESUME TRANSFORM`, `DROP TRANSFORM`,
  `ALTER TRANSFORM`), returning what it did as a tagged
  value (`{:transform_defined, target_table}`, `:paused`,
  `{:resumed, addresses}`, ...). `define/2` is an `apply/2` that refuses
  anything but a `TRANSFORM`.
- `status/2`, `definitions/1`: the catalog reads.
- `quarantined/1`, `quarantine_status/2`, `sample_quarantined/3`,
  `poisoned_since/2`: the quarantine reads. Resuming is a statement.
  `release_key/4` releases one key a transform holds, once its cause is
  fixed; `status/2` reports how many it holds (`held_keys`). `status/2`
  also reports a page the drain keeps failing on with nothing charged or
  paused (`drain_failure`, a `Trellis.DrainFailure`), and a finished `self_check/3`
  job's report lists every one open (`drain_failures`). `status/2` names the paused upstream columns
  (`paused_upstream_columns`, each `"target.column"`) that make a `:live`
  transform report `:catching_up`. `status/2` lists the source indexes a
  `GROUP BY` or `JOIN` target
  needs and lacks (`source_index_gaps`, each a `Trellis.SourceIndexGap` with
  `table`, `columns`, `role` (`:group_by`, `:join_from` or `:join_joined`: the part
  of the definition that needs it), `opted_out` and a `create index` statement as `fix`);
  `self_check/3` refuses such a target unless it was defined `WITHOUT SOURCE
  INDEX`. `status/2` lists the source indexes a `MIN`/`MAX` aggregate's
  probes read and the source lacks (`extreme_index_gaps`, each a
  `Trellis.ExtremeIndexGap` with `table`, `columns`, `argument`, `opted_out`
  and `fix`); nothing pauses, and a deleted extreme is re-read by scanning
  its group. `status/2` lists the partitions of a partitioned source a transform
  is being corrected for while it reads `:backfilling` for them
  (`correcting`, each a `Trellis.CorrectingNode` with `node`, `since` and
  `keys_remaining`). `status/2` names what a Re-derive build waits on
  instead of its own work (`build_wait_xid`, the transaction id of the
  fence it waits out, `nil` when it waits on nothing). While a process of
  another Trellis version is live, `status/2` reports that this process
  stands by (`version_handover`, a `Trellis.VersionHandover` with
  `this_version`, `processes`, each a `Trellis.OtherProcess` with
  `worker_id` and `version` (`nil` for a release before 0.2.0),
  `unregistered_staging_worker` and `migrations_pending`), and
  state-changing calls refuse with a `version_handover` error until it is
  gone. While this binary is below the catalog floor, `status/2` reports
  that this process stands by (`below_catalog_floor`, a
  `Trellis.BelowCatalogFloor` with `floor` and `this_version`), and
  state-changing calls refuse with a `version_floor` error. For a `JOIN` transform `status/2` reports its hidden tables
  (`join`, a `Trellis.JoinStatus`): `hidden_tables`, each a
  `Trellis.HiddenTable` with `table`, `name`, `estimated_rows` and `bytes`,
  the `P`-size warning (`partials_warning`, a `Trellis.PartialsWarning`),
  a 1-1 `JOIN`'s fan-out warnings (`fan_out_warnings`, one
  `Trellis.FanOutWarning` per joined position past the threshold, naming its
  `position`) and the step its build runs (`building`, a
  `Trellis.JoinBuildStep` with `step` and the `side`, `part`, `leaf`,
  `remaining` or `position` its kind of step carries). `join` is `nil` for any other
  transform. `status/2` reports the oldest Trellis release that can run a
  transform (`floor`, `"major.minor.patch"`), which an application reads
  before rolling back to an earlier Trellis release
  ([upgrading](../../docs/upgrading.md#rolling-back)).
- `request_backfill/2`, `detect_capture_disruptions/1` (pauses now the readers
  of every table whose Trellis capture was changed outside Trellis, and
  returns the tables), `has_live_drain_workers/1`,
  `has_live_staging_worker/1`, `watermark_token/1`, `await_converged/3`.
- `self_check/3` and `self_check_job/2`: check a target against a fresh
  recompute from its source, as a background job. `self_check/3` starts it and
  returns a `Trellis.SelfCheckJob` (`id`, `target`, `mode`, `state`,
  `rows_compared`, `report`, `error`) at once; a drain worker runs it, and
  `self_check_job/2` reads it back by `id` until `state` is `:done`
  (`:queued` and `:running` are in progress; `:failed` and `:cancelled` carry
  an `error`), when `report` is a `Trellis.SelfCheckReport` (`outcome`,
  `divergences`, `rows_compared`, `truncated`, ...). A target whose job is
  still unfinished gets that job back; the next call replaces a finished
  one, and `self_check_job/2` returns `nil` for the replaced job or one whose
  transform was dropped. With no drain worker anywhere in the fleet a job
  stays `:queued`.
- `start_link/1`, as `{Trellis, options}` in a supervision tree: a process
  that owns the handle, so every function above also takes its name.
- `Trellis.Migration`: `define`, `apply` and `status` in an Ecto migration.
- `schema_dump/1`, `filter_schema_dump/2`, `mix trellis.dump`,
  `mix trellis.load` and `Trellis.StructureDump`: the definitions in the
  structure file, restored when it loads (see [Schema dumps](#schema-dumps)).
- `Trellis.Metrics.render_prometheus/0`: the process-wide metrics registry
  as Prometheus text, for a `/metrics` route the host already serves (the
  binding opens no port). It needs no handle. Every series carries the
  `trellis_instance` label (`<database>/<catalog schema>`, as in the log
  lines), so handles in one VM stay apart.
- `Trellis.LogBridge`: started with the `:trellis_pg` application, it forwards
  the engine's log lines to `Logger` (`domain: [:trellis]`, the Rust module
  as `:target` metadata). Every line names its instance in the text, as
  `trellis_instance=<database>/<catalog schema>`, so handles in one VM stay
  apart. Set `config :trellis_pg, log_level: :info` to choose the most
  verbose level forwarded (default: `Logger.level()`, read once at start),
  or `config :trellis_pg, log_bridge: false` to forward nothing. A
  `tracing` subscriber installed by another NIF never sees the engine's
  lines, since each native library links its own `tracing`; see
  `Trellis.LogBridge`. A line can be dropped if the bridge falls behind,
  and a warning says how many; logging never blocks or crashes the engine.

```elixir
# A deploy's migration step: the defaults run nothing in the background.
{:ok, migrator} = Trellis.connect(url: "postgres://localhost/app")
# TLS options go in the URL (docs/embedding.md, "TLS to Postgres"):
#   url: "postgres://db.example.com/app?sslmode=verify-full&sslrootcert=/etc/ssl/ca.pem"
:ok = Trellis.migrate(migrator)
:ok = Trellis.shutdown(migrator)

# The app, once migrated. The staging worker reads the catalog as it starts,
# so a `staging: true` handle can't connect before `migrate/1` has run.
{:ok, trellis} = Trellis.connect(url: "postgres://localhost/app", staging: true, drain_threads: 2)
{:ok, _definition} = Trellis.define(trellis, "TRANSFORM widget_prices FROM widgets SELECT price AS price")
{:ok, %Trellis.Status{status: :live}} = Trellis.status(trellis, "widget_prices")

# A column that paused on bad data: find the rows, fix them, resume it.
{:ok, [%Trellis.QuarantineEntry{target: "widget_prices.price"} | _]} = Trellis.quarantined(trellis)
{:ok, %Trellis.SamplePage{samples: rows, next_cursor: cursor}} =
  Trellis.sample_quarantined(trellis, "widget_prices.price", limit: 50)
{:ok, {:resumed, ["widget_prices.price"]}} =
  Trellis.apply(trellis, "RESUME TRANSFORM widget_prices.price")

:ok = Trellis.shutdown(trellis)
```

`connect/1`'s defaults (`staging: false, drain_threads: 0`) run nothing in
the background. For each instance (catalog schema), some connection in the
fleet must set `staging: true` and some connection must run drain threads, or
no definition ever reaches `:live`. Any number of connections may set
`staging: true`: one takes the producer lock and leads, the rest stand by and
take over when it goes away, so a rolling deploy needs no operator step and
the application boots even when the old worker is still running (see
[Rolling deploys](../../docs/embedding.md#rolling-deploys)).

### Resources

At the defaults, a handle holds up to 20 Postgres connections (its call pool,
opened as calls need them and closed after 60 seconds idle) and 3 threads. A
worker handle (`staging: true, drain_threads: N`) also owns a background pool
of up to `2N + 6` connections (`2N + 5` without staging, `2` with staging
alone), closed after 60 seconds idle, and two connections that stay open: the
staging session and one `LISTEN` shared by its drain threads. It holds 6
threads: drain threads are tasks on its runtime, not OS threads. Its memory is
bounded by row counts, not bytes: about `N` × 100,000 changed rows' images at
its peak. `:max_call_pool_size` lowers the 20 (the background pool is sized by
the handle's roles, not by an option), and `:worker_threads` sets the threads.
Every handle (each `{Trellis, name: ...}` child) costs this again. The full
accounting is in
[`docs/embedding.md`](../../docs/embedding.md#what-a-handle-uses).

### Conventions

- Non-bang functions return `{:ok, value}` (or `:ok`) or
  `{:error, %Trellis.Error{code: code, message: message}}`; bang variants
  return the value or raise that error. `code` is one of a closed set of
  atoms, with `:unknown` for a code newer than the binding.
- No atom is ever built from a string the database returned: every atom in a
  result comes from a closed set the NIF allocates when it loads.
- Times are `DateTime`s. They cross the NIF as epoch microseconds.
- A quarantine target is an address string, `"transform"` or
  `"transform.column"`, exactly as `quarantined/1` reports it.
- `sample_quarantined/3`'s cursor and `watermark_token/1`'s token are
  opaque: pass back what the previous call returned.
- A call on a handle holds no scheduler while it waits. The NIF submits the
  call on a normal scheduler and returns a reference at once; the engine
  sends the result to the calling process, which waits for it in a `receive`.
  The function is synchronous to its caller, a process killed during a call
  exits at once, and a call stuck on the database holds no scheduler or
  thread of its own.
  `connect/1` and `shutdown/1` run on dirty IO schedulers, as do the NIFs
  that touch no handle; `Trellis.Metrics.render_prometheus/0` runs on a
  dirty CPU scheduler.
- Every call returns within 30 seconds, counted from when it is submitted
  (a database that can't answer in time gives `{:error, %Trellis.Error{code:
  :timeout}}`; the server stops what the call started). Calls run in
  parallel, on one handle and across processes: a call stuck on a lock
  doesn't delay another process's `status/2`. A process's own calls stay in
  order, because it waits for each reply; calls from different processes
  have no order.
- `shutdown/1` doesn't wait for calls in flight: it cancels them, and each
  of their callers gets a `:validation` error.

## In a Phoenix app

Let the application's supervisor own the handle. `{Trellis, options}` starts
a process that connects with `connect/1`'s options as it starts, hands the
handle to the callers that make calls through its name, and shuts the handle
down when the supervisor stops it:

```elixir
# config/runtime.exs
database_url = System.fetch_env!("DATABASE_URL")

config :my_app, MyApp.Trellis,
  name: MyApp.Trellis,
  url: database_url,
  # Only the fleet's one worker node runs the background work.
  staging: System.get_env("TRELLIS_WORKER") == "true",
  drain_threads: if(System.get_env("TRELLIS_WORKER") == "true", do: 2, else: 0)

# lib/my_app/application.ex
children = [
  MyApp.Repo,
  {Trellis, Application.fetch_env!(:my_app, MyApp.Trellis)},
  MyAppWeb.Endpoint
]

# Anywhere in the app: the name stands in for the handle.
{:ok, status} = Trellis.status(MyApp.Trellis, "widget_prices")
```

Each call runs in the process that makes it, in parallel with every other,
so no process queues behind another's call. The caller has to be on the same
node: a handle works only on the node that connected it. `shutdown/1` refuses
the name; the supervisor stops it, which cancels the calls still in flight.

Define transforms in Ecto migrations with `Trellis.Migration`. Trellis never
joins a migration's transaction, so a migration that uses it must set
`@disable_ddl_transaction true` (the helpers raise before applying anything
if a transaction is open), and must define `up/0` and `down/0`:

```elixir
defmodule MyApp.Repo.Migrations.DefineWidgetPrices do
  use Ecto.Migration
  use Trellis.Migration

  @disable_ddl_transaction true

  def up do
    define "TRANSFORM widget_prices FROM widgets SELECT price AS price"
  end

  def down do
    apply "PAUSE TRANSFORM widget_prices"
    apply "DROP TRANSFORM widget_prices"
  end
end
```

Ecto doesn't start the application to migrate, so each helper connects a
handle of its own, from the repo's `:trellis` configuration
(`config :my_app, MyApp.Repo, trellis: [url: database_url]`), and runs
`migrate/1` on it first. A define isn't idempotent, and the helpers don't
make it so: Ecto's `schema_migrations` runs each migration once. See
`Trellis.Migration` for the details, and for an optional guard.

Add `import_deps: [:trellis_pg]` to the app's `.formatter.exs` to keep
`define` and `apply` free of parentheses, like Ecto's own commands.

## Fixtures and bulk loads in tests

A loader that disables triggers for the load (`ALTER TABLE ... DISABLE
TRIGGER ALL`, `pg_restore --disable-triggers`) disables Trellis's capture
triggers too, and `ENABLE TRIGGER ALL` leaves them in a mode Trellis treats as
a disruption. The rows it wrote never reached a target, so Trellis pauses
every definition that reads the table (`capture_failure` has `kind:
:disruption`) instead of letting a test read silently wrong data, and waits
for a `RESUME`. `Trellis.TestSupport` does that `RESUME` for a suite, then
waits for the rebuilt targets to catch up:

```elixir
setup do
  # ... load the fixtures with triggers disabled, then re-enabled ...
  {:ok, _resumed} = Trellis.TestSupport.resume_capture_disruptions(trellis)
  :ok
end
```

`resume_capture_disruptions/2` first pauses the readers of any disrupted
table itself (`Trellis.detect_capture_disruptions/1`), so it doesn't wait for
the staging worker's next pass; resumes only the definitions paused with kind
`:disruption`; waits until each is `:live` and every write committed so far
has reached its targets; and returns `{:ok, resumed}` (`[]` when nothing was
disrupted, which costs one catalog read per captured table). A transform an
operator paused, or one paused by a schema change or another failure, stays
paused. A `RESUME` the engine refuses (a table that is gone, a table define
would now refuse) leaves that definition paused; the rest are still resumed,
and the call then returns `{:error, {:resume_refused, refused}}`, a map of
target to reason. `resume_capture_disruptions!/2` returns the targets and
raises `Trellis.TestSupport.ResumeRefused` (or `Trellis.Error`). The
`:timeout_ms` option (default 30 000) bounds the wait. The handle needs its
staging worker and a drain worker (`staging: true, drain_threads: 1`),
because the resume rebuilds through them.

Trellis sees a write when it commits, so with Ecto's SQL sandbox, load the
fixtures that targets must reflect outside the sandbox transaction (in
`setup_all`, or with the sandbox in `:auto` mode) and call the helper once.

## Schema dumps

`mix trellis.dump` writes the structure file `mix ecto.dump` writes, with
the definitions in it the way Ecto puts `schema_migrations` in it, and
`mix trellis.load` loads it. Make them what `mix ecto.dump` and
`mix ecto.load` run, by aliases in `mix.exs`:

```elixir
defp aliases do
  ["ecto.dump": "trellis.dump", "ecto.load": "trellis.load"]
end
```

The file has the application's tables, the target tables (with their
indexes and grants), the `schema_migrations` rows, and one row per
definition: its source, target and `JOIN` table, its text as written, its
grammar version and its floor. Trellis's own schema, the capture triggers on
source tables and the 1-1 targets' ledgers are not in it, so regenerating
the file changes it only where a definition changed. `mix trellis.load` runs
`mix ecto.load` and then `Trellis.migrate/1` for each repo with a `:trellis`
configuration, which creates Trellis's tables and restores each definition
into its dumped target. Each is restored unbuilt, and the worker builds it
from the source rows; a paused or quarantined transform is restored
unpaused, because a pause is runtime state. A file loaded another way
(`psql -f`) needs `Trellis.migrate/1` after it. How the restore works, and
what it refuses, is in
[`docs/embedding.md`](../../docs/embedding.md#schema-dumps).

`mix ecto.dump` can't pass `pg_dump` extra arguments, so for a repo with a
`:trellis` configuration (as for `Trellis.Migration`) `mix trellis.dump`
runs `pg_dump` itself, with `ecto.dump`'s arguments (`-r`, `-d`, `--prefix`,
`-q`) and `Trellis.schema_dump/1`'s, which leave Trellis's schema and the
ledgers out. With `--prefix`, the instance's target schema is dumped too. It
takes the capture triggers out of the output (`Trellis.filter_schema_dump/2`),
and reads the definitions before and after the `pg_dump`, redoing the dump,
up to three times, when a define or drop committed in between, so the
file's DDL and definitions agree. A repo with no `:trellis` configuration
is dumped by `mix ecto.dump`.

`mix trellis.dump` also dumps a repo's `:migration_repo`, as `mix ecto.dump`
does, before the Trellis repo's file, and with `--prefix` it reads
`schema_migrations` from the schemas named only (not the target schema it adds
to the dump). The repo and its `:trellis` url have to name the same database,
by host, port and database name spelled alike (`localhost` is not
`127.0.0.1`; a repo connecting over a socket directory, or naming no
host, matches a local host), or `mix trellis.dump` and `mix trellis.load`
refuse, before anything is written or loaded. A dump that fails once
`pg_dump` has written the file removes it, since its raw `pg_dump` output
would load without the definitions; a file no `pg_dump` wrote to is left as
it was. `mix trellis.load` restores nothing for a repo `mix ecto.load`
left alone (`--skip-if-loaded` on a loaded database, or a "no" to its
prompt).

`pg_dump` 17 writes `CREATE SCHEMA public;` when it dumps `public` by name,
which stops `mix ecto.load` into a database `mix ecto.create` made; the task
writes `CREATE SCHEMA IF NOT EXISTS public;` instead.

A structure file written by an earlier release, which carries the Trellis
schema and the definitions' catalog rows, still loads.
`Trellis.StructureDump.append/2` writes that format onto a structure file
someone else dumped with the Trellis schema in it (reading the definitions
once, so it can't see a define racing that dump), and
`Trellis.dump_definitions_sql/1` returns its SQL.

## Layout

- `lib/`: the public `Trellis` module and its structs, the process that
  owns a supervised handle (`lib/trellis/owner.ex`), and `Trellis.Migration`, which
  is compiled only when the host depends on `ecto_sql`.
- `lib/trellis/test_support.ex`: `Trellis.TestSupport`, the test helper above.
- `native/trellis_nif/`: the NIF crate, a member of the repository's Cargo
  workspace. The release makes the packaged copy stand alone (see
  Releasing). Plain-data conversion and error codes come from
  `clients/embed` (`trellis-embed`), shared with the Ruby binding.

## Development

The Erlang and Elixir versions are pinned in `.tool-versions`. From this
directory:

```sh
cargo build -p testkit --bin trellis-testkit   # the suite's throwaway Postgres
mix deps.get
mix test
```

In this repository (`:dev` and `:test`) the NIF always builds from source,
never downloads. `mix test` compiles it through Cargo into the workspace's
`target/`, and the suite spawns `trellis-testkit` (from `PATH`, else the
workspace's debug build) for a Postgres cluster that is torn down when the
test run exits. It needs the Postgres server binaries on `PATH`, like the
Rust tests.

`test/trellis/parity_test.exs` runs `clients/parity/cases.json`, the fixture
the Ruby suite runs too, so the two bindings can't drift apart unnoticed
(see `clients/parity/README.md`).

## Releasing

`trellis_pg` is released with the rest of Trellis, from one `v<version>` tag
([docs/releasing.md](../../docs/releasing.md)). `@version` in `mix.exs` and
the NIF crate's version (`native/trellis_nif/Cargo.toml`) are the release's.
`.github/workflows/release.yml` builds the NIF for each target above and
attaches them to the tag's GitHub release, where `lib/trellis/native.ex`
downloads them from. The package is then published to Hex by hand with
`.github/scripts/publish-packages.sh`, which writes
`checksum-Elixir.Trellis.Native.exs` from the NIFs that release serves.

`.github/scripts/build-elixir-nif.sh <target> <out-dir>` builds one NIF the
way the workflow does. Linux targets need `cargo-zigbuild` and Zig
(`pip install cargo-zigbuild ziglang`).