Current section
Files
Jump to
Current section
Files
statifier_persistence
README.md
README.md
# StatifierPersistence
[](https://github.com/riddler/statifier_persistence/actions/workflows/ci.yml)
[](https://hex.pm/packages/statifier_persistence)
[](https://hex.pm/packages/statifier_persistence)
[](https://hexdocs.pm/statifier_persistence/)
[](https://github.com/riddler/statifier_persistence/blob/main/LICENSE)
Durable stepper and storage adapters for
[Statifier](https://github.com/riddler/statifier-ex).
Documentation lives on [hexdocs](https://hexdocs.pm/statifier_persistence/),
including the [Surviving a restart](docs/restart-demo.md) guide.
Statifier's pure interpreter contract (machine_state, event -> machine_state,
effects) makes a persistence-first execution model possible: load a persisted
position, step it, execute the effects, persist. Hosts running charts that
span days or survive deploys should not need long-lived Session processes at
all - but every host currently hand-rolls the loop, the storage guard, and the
crash semantics. This package is that loop, packaged.
## Installation
```elixir
def deps do
[
{:statifier_persistence, "~> 0.8"},
# Optional, for the Postgres adapter:
{:ecto_sql, "~> 3.10"}
]
end
```
## A worked run
A card-processing transaction: authorize it, capture it before its
capture window closes, settle it. The whole run is four calls, and no
process holds the chart between them.
```elixir
alias Statifier.{Chart, Event, Machine, MachineState}
alias Statifier.Invoke.Types, as: InvokeTypes
alias StatifierPersistence.{Runs, Storage}
source = """
<scxml xmlns="http://www.w3.org/2005/07/scxml" version="1.0" initial="authorizing">
<state id="authorizing">
<invoke type="myapp:authorize" id="authorize"/>
<transition event="done.invoke.authorize" target="awaiting_capture"/>
</state>
<state id="awaiting_capture">
<transition event="capture.requested" target="settling"/>
</state>
<state id="settling">
<transition event="ack" target="settled"/>
</state>
<final id="settled"/>
</scxml>
"""
```
Compile the chart once and store its bytes under its own content hash.
Nothing is keyed by a name you choose: the identity comes off the
compiled `Machine`, which is what makes the guard unskippable.
```elixir
{:ok, machine} = Statifier.compile(source)
{:ok, chart_blob} = Chart.to_binary(machine)
{:ok, store} = Storage.new(StatifierPersistence.Storage.InMemory, [])
:ok = Storage.save_chart(store, machine, chart_blob)
```
Every effect a step emits reaches your host through one seam - a module
implementing `StatifierPersistence.Executor`, or an arity-2 fun. Effects
arrive one at a time, in list order, as `{tag, payload}` tuples. This one
does the least a real host could do with an outbound authorization:
```elixir
executor = fn
{:invoke, %Statifier.Effect.Invoke{type: "myapp:authorize"} = invoke}, ctx ->
# your own gateway call, keyed for idempotency by run and invocation
MyApp.Payments.authorize(ctx.run_id, invoke.invoke_id)
:ok
_effect, _ctx ->
:ok
end
opts = [executor: executor, invoke_types: InvokeTypes.new(types: ["myapp:authorize"])]
```
`create/4` initializes the chart, hands the resulting effects to the
executor, and persists the quiescent position under a run id you choose
- here the transaction's own key:
```elixir
{:ok, run, state} = Runs.create(store, "txn_01H8", machine, opts)
#=> run.status == :active, active leaf state "authorizing"
```
Each later event is one `step/5`: liveness check, guarded load, step,
effects out through the seam, persist. Between calls there is no live
process and no in-memory position - only the run record.
```elixir
{:ok, run, state} =
Runs.step(
store,
"txn_01H8",
machine,
Event.external("done.invoke.authorize", invokeid: "authorize"),
opts
)
#=> run.status == :active, active leaf state "awaiting_capture"
```
### Across a restart
Nothing above kept state in the beam, so a deploy in the middle of the
run changes nothing about how it continues. Given only the run id, fetch
the record, fetch the chart bytes it names, and recompile:
```elixir
{:ok, record} = Storage.fetch_run(store, "txn_01H8")
{:ok, %{chart_blob: blob}} = Storage.fetch_chart(store, record.content_hash)
{:ok, rebooted} = Chart.from_binary(blob)
```
`rebooted` is compiled afresh from the stored bytes, not carried over
from before the restart, and it is what makes the stored position
readable again: Statifier interns state ids to indices at compile time,
so a position is only meaningful against the exact chart revision that
produced it. The identity guard enforces that on every load. Step a run
with a machine compiled from a *changed* chart and it refuses with
`{:error, {:identity_mismatch, stored, supplied}}` rather than silently
resuming the wrong configuration.
```elixir
{:ok, run, state} =
Runs.step(store, "txn_01H8", rebooted, Event.external("capture.requested"), opts)
#=> run.status == :active, active leaf state "settling"
{:ok, run, state} = Runs.step(store, "txn_01H8", rebooted, Event.external("ack"), opts)
#=> run.status == :completed, no active leaf states
```
`:completed` is reached only by the chart reaching a final state - the
lifecycle consumes the interpreter's `:done` itself and never hands it
to your executor. `Runs.fail/4` is the one host-driven terminal
transition, and a step delivered to a terminal run comes back
`{:discarded, run}` rather than raising.
To read the configuration back as state ids, as the snippets' comments
show it:
```elixir
state
|> MachineState.active_leaf_states()
|> Enum.map(&Machine.id(state.machine, &1))
|> Enum.sort()
```
### What each module is for
| Module | Role |
|---|---|
| `StatifierPersistence.Storage` | The identity-guarded facade: charts, positions, run records. Every load is guarded; there is no unguarded path |
| `StatifierPersistence.Storage.Adapter` | The behaviour a backing store implements. `Storage.InMemory` is the reference one, `Storage.Ecto` the Postgres one (on [another backend](docs/non-postgres-backends.md), minus the lock and the listings) |
| `StatifierPersistence.Runs` | The lifecycle: `create/4`, `step/5`, `fail/4`, in ADR-0004's fixed order |
| `StatifierPersistence.Driver` | Run-to-quiescence over `Runs`: performs the chart's `<invoke>` calls and steps each answer back in |
| `StatifierPersistence.Executor` | The seam every effect crosses on its way to your host |
| `StatifierPersistence.Serialization` | The per-run ordering strategy the fetch-to-persist tail runs inside; defaults to the adapter's own `lock_run/3` |
| `StatifierPersistence.Testing.StorageConformance` | The conformance suite - point it at your own adapter to hold it to the same bar |
Two things the loop deliberately does not do. Effect delivery is
at-least-once: a crash between step and persist re-drives the same event
and re-emits the same effects with identical deterministic keys, and the
loop never dedupes - idempotency on that key is yours. And a resumed run
restores position, not liveness: pending timers and in-flight
invocations are re-established by the host, from its own durable rows.
[Surviving a restart](docs/restart-demo.md) walks a demo embedder
through both.
## Driving a chart that calls out
`Runs` steps a run once. A chart that invokes a service is not finished
when that step returns - it is waiting for an answer it cannot fetch
for itself, and every host that has embedded this package has written
the same loop on top. `StatifierPersistence.Driver` is that loop:
```elixir
driver =
StatifierPersistence.Driver.new(store, machine,
dispatch: fn type, params, _context -> MyApp.perform(type, params) end,
effects: fn effect, _context -> MyApp.Timers.consume(effect) end,
invoke_types: Statifier.Invoke.Types.new(types: ["myapp:authorize"]),
serialization: {MyApp.RunLock, MyApp.RunLock}
)
{:ok, run, state} = StatifierPersistence.Driver.create(driver, run_id)
{:ok, run, state} = StatifierPersistence.Driver.send_event(driver, run_id, Statifier.Event.external("go"))
```
One call is one durable step, every effect through your `effects:`
executor, every `<invoke>` through your `dispatch:` fun inside that same
step, and then one further durable step per answer until the chart rests.
`{:ok, donedata}` answers `done.invoke.<id>`; `{:error, failure}` answers
`error.communication.invoke.<id>` with `Statifier.Session.failed_invocation/3`'s
own `reason`/`attempts`/`detail` payload, and means permanently failed
rather than "retry".
Both events are built field for field from the two doors
`Statifier.Session` gives a handler-backed invocation's host, `origin`
and `origintype` included, so the same chart sees the same event whether
it runs in a session or out of storage. That is asserted rather than
claimed: `test/statifier_persistence/driver_session_conformance_test.exs`
answers one document both ways and compares the `_event` each chart saw.
An answer whose invocation the chart has since cancelled is dropped, per
spec 6.4.3, and a chart whose answer re-arms its own call is bounded by
`max_turns:` rather than driven forever.
### Fanning one invocation out over N children
A durable subchart is one child per `<invoke>`, created inside the
parent's own step. An `<invoke>` that maps over a list is N of them, and N
creates cannot hold the parent's exclusion, so the children are started
afterwards - one call per child, from whatever job picks it up:
```elixir
StatifierPersistence.Driver.start_child_at(driver, parent_run_id, effect, index, count,
policy: :all
)
```
`effect` is the resolved `Statifier.Effect.Invoke` (or the whole
`{:start_child, resolved, {:invoke, invoke}}` instruction), `index` is the
child's 0-based position, and `count` is N. The call is idempotent on the
child's derived run id, so a re-delivered start adopts the child it
already created instead of making a second one. Scheduling those calls is
a job runner's business, not this package's.
Each child then runs as an ordinary run. When one reaches a terminal
status its answer is stored on its own run record, and a settlement
section under the **parent's** exclusion asks - through an indexed status
projection, never a listing of whole records - whether all N have. Only
the settlement that finds them all terminal assembles the dense,
index-ordered list and answers the parent's ordinary door, once:
```elixir
[
%{"index" => 0, "status" => "completed", "donedata" => %{"id" => "acct_1"}},
%{"index" => 1, "status" => "failed", "failure" => %{"reason" => "declined", ...}},
%{"index" => 2, "status" => "cancelled"}
]
```
`policy: :first_error` cancels the rest as soon as one child fails: the
started siblings through the cascading cancel, and the ones whose start
job has not run yet through the `child_canceller:` seam, which is handed
the parent run id, the invocation id, and the indices with no run. Both
kinds read `"cancelled"` at their index in the same list.
A child fails on its own word, with no host in the loop, by settling in a
**failure-classed final** - a top-level `<final>` whose `<donedata>`
carries the reserved key `statifier_persistence:run_status` set to
`"failed"`:
```xml
<final id="declined">
<donedata>
<param name="statifier_persistence:run_status" expr="'failed'"/>
<param name="reason" expr="decline_reason"/>
</donedata>
</final>
```
That step is an ordinary successful one; the run record takes `:failed`
with the `failure` string `"failed_final"`, and the whole `<donedata>` -
tag included, alongside whatever else the final carries - reaches the
parent's list verbatim. Macrostep-budget exhaustion is the other route to
`:failed`, and the `failure` string is what tells the two apart. An
unhandled `error.*` event is not a route: a chart that cannot continue
stays `:active` until its author routes the error to a final.
See `StatifierPersistence.Runs` for the full rule.
An adapter that cannot store a child's answer, or cannot answer the
status projection, is refused at open - a child whose invocation could
never be settled is not started. On the Ecto adapter both arrive with the
V03 migration.
## Status
Early, under active development, and the API is not frozen before 1.0.
The storage-adapter behaviour with its identity guard, the in-memory
reference adapter, the run lifecycle and executor seam, per-run
serialization, and the Ecto layer (configurable keys/tables, versioned
migrations, and the Postgres adapter below) all exist and are
conformance-tested.
## The Ecto adapter
Configure a persistence module on your own repo once, and migrate:
defmodule MyApp.Persistence do
use StatifierPersistence.Ecto, repo: MyApp.Repo
end
defmodule MyApp.Repo.Migrations.AddStatifierPersistence do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence)
end
One migration covers every version of the package DDL on a fresh
database. If you already ran that migration when this package shipped
only V01, pick the later versions up with a second ordinary migration
rather than re-running the first:
defmodule MyApp.Repo.Migrations.AddStatifierPersistenceRunMetadata do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, from: 2)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, version: 2)
end
`from:` says where a call starts and `version:` where it ends, in both
directions, so a migration's two calls always cover the same span: `up`
from `from:` (default V01) up to `version:` (default the newest), `down`
from `from:` (default the newest) back to `version:` (default V01). A
migration that caps one end caps the other to match - see "Upgrading to
V03 before deploying 0.7.0" below for the case that bites.
then build the guarded store the rest of the package works through:
{:ok, store} =
StatifierPersistence.Storage.new(
StatifierPersistence.Storage.Ecto,
persistence: MyApp.Persistence
)
The adapter passes the same conformance suite the in-memory reference
does (`StatifierPersistence.Testing.StorageConformance` - point it at
your own adapter to hold it to the identical bar), stores engine
identities verbatim, and implements the optional per-run `lock_run/3`
as a transaction-scoped advisory-plus-row lock (ADR-0004 as amended).
In your test suite, pass `sandbox: true` so each test runs in its own
`Ecto.Adapters.SQL.Sandbox` checkout via the adapter's `isolate/1`.
### Running on a backend that is not Postgres
The adapter is written against Postgres and this package's gate runs
against a real Postgres server, but only three of its callbacks are
actually Postgres SQL: `lock_run/3` (advisory lock plus `FOR UPDATE`) and
the two metadata listings (`jsonb` containment). Everything else - charts,
positions, run records, the identity guard, the executor seam, resume,
and the versioned migrations, V03's Postgres-only index included - runs
on any Ecto backend.
A host on SQLite or another backend therefore **declines the lock
callback** rather than getting a portable imitation of it: pass your own
`serialization: {module, config}` strategy, backed by an exclusion the
host already owns (a job queue keyed per run id, a single consumer) or by
a pass-through when the deployment is single-writer by construction. What
you must not do is leave the default in place, which reaches the
Postgres-only `lock_run/3` and raises mid-run.
The four conformance cases those three callbacks generate carry
`@tag :postgres`, so such a host runs the shipped suite green and honest:
mix test --exclude postgres #=> 31 tests, 0 failures, 4 excluded
[Running on a backend that is not Postgres](docs/non-postgres-backends.md)
is the full guide: what is Postgres-only and why, how to write the
strategy, what declining costs (durable subcharts and the run listings
refuse rather than break), and how to verify your own setup.
### Upgrading to V03 before deploying 0.7.0
0.7.0 needs V03 of the package DDL, and the order matters: **run the
migration first, then deploy the new code.** `outcome_blob` is an
unconditional field on the generated runs schema, so 0.7.0 against a V02
database fails on every query that touches the runs table, not only on
the fan-out write that introduced the column. The reverse order is safe:
V03 on a database still served by 0.6.x adds a column nobody writes and
an index nobody's query needs yet.
A host already on V02 picks V03 up with an ordinary migration of its own:
defmodule MyApp.Repo.Migrations.AddStatifierPersistenceOutcomeBlob do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, from: 3)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, version: 3)
end
A host whose *first* migration is capped - `up(for: MyApp.Persistence,
version: 2)`, which is what keeps a fresh clone and an already-migrated
database on the same sequence of steps - caps its rollback the same way,
with `from: 2`:
defmodule MyApp.Repo.Migrations.AddStatifierPersistence do
use Ecto.Migration
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, version: 2)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, from: 2)
end
Leaving that `down` uncapped is the failure this ceiling exists to
prevent: Ecto rolls migrations back newest first, so `mix ecto.rollback
--all` runs the V03 migration's `down` and then this one's, which without
`from:` starts at V03 again and fails on a column that is already gone
(`no such column: outcome_blob`). Rolling back a single step is
unaffected either way.
V03 does two things, and only one of them is cheap.
**The `outcome_blob` column** is a nullable `:binary` added to the runs
table. Postgres adds a nullable column with no default as a catalog-only
change, so this part is fast whatever the table's size. It takes the
configured `:blob_type` with the other three blob columns, so a
`:blob_type` whose underlying database type is not binary needs the same
hand-written `ALTER` this README's encryption section already describes
for those three - now for four columns, not three.
**The `metadata` GIN index** is the part to plan for. V03's `up/1`
issues a plain `CREATE INDEX`, **not** `CREATE INDEX CONCURRENTLY`: it
takes a `SHARE` lock on the runs table for the whole build, which blocks
every `INSERT`, `UPDATE` and `DELETE` against that table until the index
is finished. Reads are unaffected. On a small or idle runs table this is
imperceptible. On a large one it is an outage of every write the runs
table takes - which, for a host stepping runs durably, means every step
of every run.
How long that is depends on the row count, the width of the `metadata`
maps, and the server, so measure rather than guess. (`sp-461` is a
separate measurement issue and not a number for this build: it measures
the settlement read cost against a GIN-indexed `metadata` column at
increasing fan-out widths.)
The concurrent build is what a host with a large runs table wants, and
0.8.0 ships it as V04 - see below.
The index is not optional in effect: without it, every fan-out child
completion asks whether its N siblings are terminal with a `jsonb`
containment query, and each one is a sequential scan of the whole runs
table.
**On an Ecto adapter that is not Postgres, the index is skipped.** `GIN`
and `jsonb_path_ops` are Postgres spellings, so `up/1` creates the index
only when the migration's repo runs on `Ecto.Adapters.Postgres`, and
`down/1` drops it under the same condition. Everything else in V03 - the
`outcome_blob` column included - is created on every adapter, which is
what lets a SQLite host run this package's DDL at all. (In 0.7.0 it could
not: the index raised, the whole migration rolled back, and the column
went with it. Fixed in 0.7.1.)
What is skipped with the index is what the index served. Both metadata
queries this package issues -
`StatifierPersistence.Storage.Ecto.list_runs_by_metadata/2` and the
status projection `list_run_states_by_metadata/2` - are `jsonb`
containment SQL, which a non-Postgres backend does not parse. So on such
an adapter the Ecto adapter declares no metadata support: a `metadata:`
map at create is refused with `{:error, :metadata_unsupported}`, the two
listings refuse with `{:error, :child_listing_unsupported}` and
`{:error, :run_states_unsupported}`, and a durable subchart or a fan-out
over that store is **refused at open** rather than started and left with
children nothing can settle. Storing, loading, stepping and resuming runs
are unaffected. Per-run locking is a separate Postgres-only surface -
`lock_run/3` is `pg_advisory_xact_lock` plus `SELECT ... FOR UPDATE` -
and is tracked in `sp-5lm`.
### Building the metadata index concurrently: V04
0.8.0 adds V04, whose whole job is to rebuild V03's `metadata` GIN index
with `CREATE INDEX CONCURRENTLY` - same name, same expression, same
`jsonb_path_ops` opclass, built without the `SHARE` lock. It ships in the
same versioned helper as everything else, so a fresh database picks it up
with the one-call recipe and needs nothing else.
The one thing V04 cannot do for you is turn off the transaction it runs
in. `CREATE INDEX CONCURRENTLY` cannot run inside a transaction block,
and Ecto reads `@disable_ddl_transaction` and `@disable_migration_lock`
from the module `Ecto.Migrator` runs - **your** migration, not a module
it delegates to. So V04 wants a migration of its own:
defmodule MyApp.Repo.Migrations.RebuildStatifierPersistenceMetadataIndex do
use Ecto.Migration
@disable_ddl_transaction true
@disable_migration_lock true
def up, do: StatifierPersistence.Ecto.Migrations.up(for: MyApp.Persistence, from: 4)
def down, do: StatifierPersistence.Ecto.Migrations.down(for: MyApp.Persistence, version: 4)
end
Called from an ordinary transactional migration instead, V04 leaves V03's
index in place and does nothing else. That is deliberate rather than a
failure mode: the index it would have built is the one already there,
under the same name, and raising would break the one-call recipe every
fresh database and test harness uses, where a plain build on an empty
runs table costs nothing. It logs a warning when the runs table already
holds rows, which is the case where the plain build did block writes and
the two attributes are what you were missing.
`down/1` for V04 does nothing at all: what it leaves behind is V03's
index, and V03's `down/1` is what drops it. A non-transactional migration
has no rollback, so if the rebuild is interrupted, re-run the migration -
the drop is `drop_if_exists`, so it clears a missing or an invalid
leftover either way.
**A host with a large runs table that has not yet reached V03** is the
one case V04 does not solve by itself, because V03 still builds the index
plainly on the way past. Such a host adds the `outcome_blob` column by
hand, skipping V03's helper call entirely:
defmodule MyApp.Repo.Migrations.AddStatifierPersistenceOutcomeBlob do
use Ecto.Migration
def up do
alter table("runs") do
add(:outcome_blob, :binary, null: true)
end
end
end
substituting your configured table name and prefix, and then runs the V04
migration above for the index. That is a smaller hand-written migration
than 0.7.x asked for - the index half is V04's now - and the warning
about never calling `up(from: 3)` afterwards still applies: V03's
`create/1` is a plain `create`, not `create_if_not_exists`, so a second
run fails on the index that is already there.
**On an Ecto adapter that is not Postgres, V04 is a no-op**, both
directions, under the same adapter check V03 uses: there is no index to
rebuild, because V03 created none. Such a host needs neither attribute.
### Listing runs by host scope
A run record carries engine identities and opaque blobs. Nothing on it
answers the question a multi-tenant host asks first - "list the runs for
scope X" - so ADR-0006 adds one optional, opaque `metadata` map to a run,
stored beside it and handed back unchanged.
Take a card-processing host running a `myapp:authorize` / `myapp:capture`
chart, one run per payment attempt, and a support screen that lists every
run for one processor account. Tag the run at create with the account ids
the host already keys its own tables by:
{:ok, run, _machine_state} =
StatifierPersistence.Runs.create(store, payment_id, machine,
executor: MyApp.Executor,
metadata: %{
"tenant_id" => "acct_01H8X",
"processor_account_id" => "pacct_4471"
}
)
and read them back with an equality match on every pair:
{:ok, runs} =
StatifierPersistence.Storage.Ecto.list_runs_by_metadata(store.opts, %{
"processor_account_id" => "pacct_4471"
})
Equality on all given pairs is the whole query surface: no ranges, no
partial matches, no ordering guarantee. Anything richer is a query you
write against your own column - the table name is yours to configure, so
that is a supported thing to do. The V02 migration adds the column as
nullable `jsonb` with no index of its own, because which pairs you query
by is your call. V03 (above) adds the one containment index this
package's own settlement query needs, a GIN `jsonb_path_ops` index on the
whole column; an expression index on particular keys, or the wider
`jsonb_ops` operator set, is still yours to add when the volume asks.
Two rules come with it.
**Identities only, never personal data.** Keys and values are host
identities - a tenant id, a subject-entity id, a correlation id - and
never a name, an email address, a postal address, a card number, or any
other personal or cardholder data. This is a rule of the contract, not
advice: `:blob_type` encryption (below) covers the three blob columns and
does not reach this one, so anything you file here is at rest in the clear
no matter how the blobs are configured. The map is opaque to this package
by design, so nothing here can inspect a value and reject it - the rule is
kept by you.
**An adapter may refuse it.** An adapter that cannot store the map refuses
a non-empty one at the create with `{:error, :metadata_unsupported}`, so
you learn on the first call rather than finding a silently dropped scope
later. An empty or absent map is never refused. The shipped in-memory and
Ecto adapters both support it; a third-party adapter that does not is
still conformant, and the conformance suite tests both answers. The Ecto
adapter refuses at the same point for a value `jsonb` cannot hold - a
tuple, an atom, a pid, or a binary that is not valid UTF-8 - rather than
storing something that is not what you handed it. The map is write-once:
it is set at create and a later step or abandonment carries it forward
untouched.
### Encrypting the blob columns
`use StatifierPersistence.Ecto` hard-codes `:binary` for its three blob
columns (`identity_blob`, `chart_blob`, `position_blob`) by default -
plain `bytea`, byte-identical round trip, nothing extra. Pass
`:blob_type` to put a custom Ecto type on those three columns instead,
and encryption at rest needs no wrapping adapter:
defmodule MyApp.Persistence do
use StatifierPersistence.Ecto,
repo: MyApp.Repo,
blob_type: MyApp.EncryptedBlob
end
`:blob_type` accepts a bare module implementing `Ecto.Type`, or a
`{module, opts}` tuple for an `Ecto.ParameterizedType`. It reaches only
those three columns: keys and lookup columns (`content_hash`,
`session_id`, `run_id`, `status`, `failure`) always stay plain text,
because the identity guard and the unique indexes depend on reading
them back verbatim.
The shape a production `MyApp.EncryptedBlob` needs is a vault-backed or
envelope-encrypting `Ecto.Type` - `dump/1` encrypts on the way in,
`load/1` decrypts on the way out. This package takes no position on
which key-management scheme backs it; that choice belongs to the host.
To prove the shape without any encryption dependency, here is a minimal
`Ecto.Type` that reversibly transforms every byte (not encryption - a
stand-in to show the wiring):
defmodule MyApp.ReversibleBlob do
use Ecto.Type
@mask 0xA5
def type, do: :binary
def cast(binary) when is_binary(binary), do: {:ok, binary}
def cast(_other), do: :error
def dump(binary) when is_binary(binary), do: {:ok, transform(binary)}
def dump(_other), do: :error
def load(binary) when is_binary(binary), do: {:ok, transform(binary)}
def load(_other), do: :error
defp transform(binary) do
for <<byte <- binary>>, into: <<>>, do: <<Bitwise.bxor(byte, @mask)>>
end
end
The shipped V01 migration always emits `:binary` (`bytea`) for the
three blob columns and does not read `:blob_type`. A `:blob_type` whose
underlying database type is still binary - an envelope-encrypting type
that dumps to and loads from raw bytes, like the sketch above - needs
no DDL change. A `:blob_type` that dumps to a different underlying type
(text, jsonb, a Postgres domain) needs you to alter those three columns
yourself; the migrations helper does not do it for you.
## Running the tests
The suite includes database-backed tests against a real Postgres server -
ADR-0005 rejects a skip tag for when one is absent, so `mix quality` and
`mix test` both need one reachable. Start it once with:
docker compose up -d db
which brings up `postgres:17` on `localhost:5432` with user/password
`postgres`. Override host, port, user, password, or database name with the
`PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, and `PGDATABASE` env vars (see
`config/test.exs` for the defaults) if a server is already running
elsewhere.
## Surviving a restart
`docs/restart-demo.md` walks through the demo embedder that drives this
package's whole surface across a simulated restart with no Session
process: persist mid-run with a pending durable timer and an in-flight
async invocation, drop everything volatile, cold-boot from the run id
alone, and finish with zero duplicate side effects and a replay that
reproduces the path. The executable version lives in
`test/statifier_persistence/demo/restart_demo_test.exs` (and its
Postgres variant beside it).
## The contract this package builds on
The persisted-position story is already specified upstream, and this package
is one consumer of it rather than the definition of it:
- `docs/persistence.md` in statifier-ex covers what MachineState contains, the
interned-index hazard, chart identity, and the resume recipe.
- ADR-0052 there records the rules: a persisted position is only meaningful
against the exact chart revision that produced it, so every load is guarded
by the Machine identity / content-hash. Loading a position against the wrong
revision does not error - it silently resumes the wrong configuration.
- ADR-0060 records the resume API: the `:resume` option on
`Session.start_link/2`, the pure-core rehydration path, and what a resume
deliberately does not restore (in-flight delayed-send timers and live
invoked children).
Read all three before adding code here.
## Scope
In scope:
- A storage-adapter behaviour: save/load of MachineState snapshots (or
Recordings), guarded by the Machine identity so a position can never be
loaded against the wrong chart revision.
- Run lifecycle as a library: create/step/complete/fail, with a serialization
guarantee per run so concurrent event deliveries to one run are ordered.
- The load -> handle_event -> execute effects -> persist loop, with effect
execution delegated to the host.
- An Ecto adapter shipping schemas and migrations for chart definitions,
versions, and runs; the host supplies the Repo and any tenancy columns.
Out of scope: domain actions, authoring UI, and job scheduling -
[statifier_oban](https://github.com/riddler/statifier_oban) owns timers and
async work.