replicant

# The reference pipeline — replicant as a durable Postgres→Postgres replicator

One `docker compose up` brings up the whole reference stack:

```
Postgres 18 (source, wal_level=logical + example_pub) ──pgoutput──> [this app]
                                                                     ├─ idempotent orders replica (upsert by PK)
                                                                     └─ durable commit-LSN checkpoint
                                                                  ──> Postgres 18 (destination)
```

This is a **go-forward change replicator**: it streams changes that commit from
the moment the pipeline starts. Pre-existing source rows are **not** backfilled
— `snapshot: true` is the one-flag alternative for that (it additionally
requires `handle_snapshot/2` + `handle_snapshot_complete/1` on the sink; see
the repo README's "Start modes").

All credentials are throwaway local-example values, not secrets. Ports bind to
127.0.0.1 only (15432 source / 15433 destination).

## Run it

```bash
cd examples/replication_pipeline
docker compose up -d --build          # source + destination + pipeline
```

First boot is a full OTP-release build — a few minutes. The pipeline creates
its replication slot a few seconds after boot; an insert committed BEFORE the
slot exists is never streamed (go-forward starts at the slot's creation
point), so confirm the slot is active first:

```bash
docker compose exec source-pg psql -U postgres -d example_src \
  -c "SELECT slot_name, active FROM pg_replication_slots"
# one row, active = t — ready

# write a row on the source
docker compose exec source-pg psql -U postgres -d example_src \
  -c "INSERT INTO orders (id, note) VALUES (1, 'hello')"

# see it on the destination, with its checkpoint
docker compose exec dest-pg psql -U postgres -d example_dst \
  -c "SELECT * FROM orders; SELECT commit_lsn FROM pipeline_checkpoint"
```

Teardown: `docker compose down -v`.

### Prove the durability

```bash
docker compose restart pipeline                     # kill/restart mid-life
docker compose exec source-pg psql -U postgres -d example_src \
  -c "INSERT INTO orders (id, note) VALUES (3, 'after restart')"
docker compose exec dest-pg psql -U postgres -d example_dst \
  -c "SELECT id, note FROM orders ORDER BY id" \
  -c "SELECT count(*) FROM cdc_receipts"            # no re-delivery of prior rows
```

The durable checkpoint row made that resume gap-free and effect-once: the
post-restart row arrives alongside the earlier ones, and the receipts for
prior transactions are unchanged — a re-delivery would have doubled them.

## What each piece teaches

| Piece | The lesson |
| --- | --- |
| `ReplicationPipeline.Sink` | at-least-once delivery made effect-once by the commit-LSN watermark: data + checkpoint in ONE destination transaction, and the `commit_lsn <= checkpoint` skip IS the dedup (Critical Rule 3) |
| `ReplicationPipeline.Sink` (receipts) | the value-free `cdc_receipts` ledger (commit_lsn/schema/table/op, never a value) exists exactly when its transaction took effect — re-delivery is skipped whole |
| `ReplicationPipeline.Sink` (session identity) | the first connect BINDS the source's `{system_identifier, database}` into the checkpoint row; every later connect COMPARES — a rebuilt source container HALTS the pipeline instead of silently resuming a different database (ADR-0007) |
| `ReplicationPipeline.Sink` (upsert) | unchanged-TOAST columns are OMITTED from the upsert SET — the sentinel never appears in `record` (Critical Rule 4) |
| `docker-compose.yml` (source flags) | `wal_level=logical` is the load-bearing flag; the publication is the operator's SQL and a missing one fails closed |

The demo is intentionally minimal (one table, `id` PK). Real deployments
extend the sink per table — the seam is the point, not the schema.

## Swapping the decoder (pre-15 sources through pglogical / wal2json)

The stack above targets `pgoutput` on Postgres 18. A pre-15 source that already carries
the `pglogical` or `wal2json` output plugin streams through the SAME sink and checkpoint
contract — the same container image, one env change (`application.ex` builds the
per-decoder table-set key from `REPLICANT_DECODER`):

```bash
# pglogical (a pglogical 2.x node; the table rides the "default" replication set)
REPLICANT_DECODER=pglogical
REPLICANT_REPLICATION_SETS=default            # comma list

# wal2json (format-version 2, wal2json >= 2.6)
REPLICANT_DECODER=wal2json
REPLICANT_TABLES=public.orders               # comma list of schema.table pairs
REPLICANT_ALLOW_KEYLESS=true                 # ONLY for a genuinely insert-only table
                                              # (no replica-identity key): without this a
                                              # keyless table HALTS the pipeline at start —
                                              # the fail-closed default, because wal2json
                                              # drops its updates/deletes plugin-side with
                                              # no wire signal
```

A dropped column halts `:destructive` on every decoder; under wal2json the residual
TOASTable-on-update-only case is caught by the periodic catalog guard
(`schema_check_interval`, default 30s). Delivery is byte-identical across the three
decoders (proven by `test/integration/decoder_parity_test.exs` — CI's 12 plugin row
runs all three on one server, plus both plugins again on a real 9.6 secondary).

## CI

The `reference-example` CI job builds this stack every push and drives the
full sequence: the example's own static gates (format, compile-warnings,
credo --strict, dialyzer), then insert → replica + receipts + checkpoint
assertions, TOAST-sentinel survival, and restart-resume with zero duplicate
receipts — so the example can never silently rot out of sync with
replicant's public sink API.