capstan

# capstan ROADMAP

Authored definitions only. Status and evidence columns are DERIVED from the tracker by
tooling — never hand-edited here.

| ID | What | Acceptance | Depends | Why |
|---|---|---|---|---|
| C1 | **Streaming spine** — connect to MySQL as a replica, tail the row-based binlog from a GTID position, assemble committed transactions, deliver to a sink, durably advance a processed-GTID checkpoint, and halt fail-closed on every silent-loss condition. slug:c1-streaming-spine | A sink receives every committed row change of the configured tables exactly once across a kill/restart, proven on live MySQL by an append-only ledger; every fail-closed halt has a red-capable tripwire (gap, precondition, unmapped table_id, XA, unknown type, compression, server_id conflict); Rule 1 proven on all four leak vectors; README + usage-rules + CHANGELOG authored | — | [ADR-0001](adr/0001-position-and-dedup-model.md), [ADR-0002](adr/0002-fail-closed-server-preconditions.md), [ADR-0003](adr/0003-transaction-shape-and-checkpoint-semantics.md) |
| C2 | **Initial snapshot** — consistent backfill of existing rows woven into the running stream, via cursor-gated suppression + a brief per-chunk `LOCK TABLES … READ` that captures an exact GTID position (the read-only-replica-safe substitute for DBLog's written watermark; MySQL has no `EXPORT_SNAPSHOT`). slug:c2-snapshot | A table with pre-existing rows is backfilled and then streamed with no gap and no duplicate at the handoff, proven live under concurrent load; resumable mid-backfill; strict-once in normal operation; each snapshot-path fail-closed halt tripwired; README + usage-rules + CHANGELOG authored | C1 | [ADR-0005](adr/0005-initial-snapshot-cursor-gate-brief-lock.md) (extends [ADR-0002](adr/0002-fail-closed-server-preconditions.md); honors [ADR-0001](adr/0001-position-and-dedup-model.md), [ADR-0003](adr/0003-transaction-shape-and-checkpoint-semantics.md), [ADR-0004](adr/0004-c1-scope-lib-owned-checkpoint-only.md)) |
| C3 | **Batching** — lib-owned batched checkpointing and sink-owned `handle_batch/1` atomic batch delivery. slug:c3-batching | Batch modes preserve their stated guarantees under kill/restart; the composition of each batch mode with every durability-signal path is tripwired (replicant's cross-mode-composition class) | C1 | TBD at design |
| C4 | **Type + payload breadth** — exotic column types (spatial, `SET`, pre-5.6 temporals) and zstd `TRANSACTION_PAYLOAD_EVENT` decompression. slug:c4-type-breadth | Each type previously halting `:unsupported_column_type` decodes correctly against captured real bytes; a compressed transaction decodes; the fail-closed halts remain for anything still unsupported | C1 | TBD at design |
| C5 | **XA transactions** — track `XA_PREPARE` → commit/rollback rather than halting. slug:c5-xa | A prepared-then-rolled-back XA transaction delivers zero rows; a prepared-then-committed one delivers exactly once; bounded state under an unbounded prepare window | C1 | TBD at design |
| C6 | **Downstream consumer sink adapter** — a thin per-source adapter package (~250 lines) that consumes a capstan transaction stream as a source for a downstream pipeline, with that consumer's core unchanged. slug:c6-downstream-adapter | A capstan change stream is delivered through a downstream pipeline via a thin adapter, the consumer's core unchanged; the `source_coordinate` is a no-ordinal GTID-set map `%{"kind"=>"capstan_transaction_v1","gtid"=>…,"gtid_set"=>…}` (ordering is set-membership, never an ordinal) | C1 | Downstream consumer path. Detailed acceptance + adapter specifics are defined when the consumer package is public. |

## Backlog — deferred from C1 (ADR-0004)

Sink-owned checkpoint mode and explicit start positions were scoped in the C1 design (§ Scope item 7) and **deferred by user decision** (2026-07-21); C1 refuses them fail-closed today. Authored facts only; status is DERIVED.

| ID | What | Acceptance | Depends | Why |
|---|---|---|---|---|
| C1a | **Sink-owned checkpoint mode** — the sink persists the processed position atomically with its own data write and hands it back via `Capstan.Sink.checkpoint/0` (effect-once on the atomic path). Refused today as `:sink_owned_mode_unsupported`. slug:c1-sink-owned-checkpoint | A sink-owned pipeline runs: `AssemblerServer` reads the resume position from `Sink.checkpoint/0`, delivers, and the sink checkpoints atomically with its own write; a kill/restart replays zero and skips zero (effect-once), proven live on an append-only ledger | C1 | [ADR-0004](adr/0004-c1-scope-lib-owned-checkpoint-only.md). The `Capstan.Sink` behaviour and `Capstan.Pipeline` per-mode validation already exist; only the `AssemblerServer` run-path wiring is owed |
| C1b | **Explicit start positions** — honor `start_position:` as an explicit `%Capstan.Position{}` override and as `:current`, end-to-end. Refused today as `:start_position_override_unsupported` / `:start_position_current_unsupported`. slug:c1-explicit-start-position | A pipeline started with an explicit `%Position{}` resumes the `Connection` dump AND seeds the `AssemblerServer` watermark from the SAME position (no silent checkpoint hole); `:current` starts from the server's live position via a pre-connect query | C1 | [ADR-0004](adr/0004-c1-scope-lib-owned-checkpoint-only.md). The spine threads a position to the `Connection` dump, but the `AssemblerServer` seeds its watermark from the checkpoint store alone today — honoring an override needs both to agree |

## Backlog — deferred from C2 (ADR-0005)

Two C2 capability boundaries ship as **loud fail-closed refusals** (ADR-0005 § Consequences), plus one contract-completeness gap surfaced at the C2 closeout cross-vendor review — each a separable follow-up. Authored facts only; status is DERIVED.

| ID | What | Acceptance | Depends | Why |
|---|---|---|---|---|
| C2a | **Collation-ordered-string PK** — snapshot a table whose primary key is a collation-ordered `CHAR`/`VARCHAR`/`TEXT` column. LANDED for `CHAR`/`VARCHAR` (ADR-0012 — server-computed weight bytes; any charset, any collation, composites); `TEXT` and `ENUM`/`SET` remain refused on measured grounds (per-page filesort; position-order divergence). slug:c2-collation-string-pk | A collation-string-PK table backfills gap-free/dup-free: the cursor-gate's `k ≤ cursor` comparison reproduces the source's collation order (not Elixir byte-order) so no row is mis-classified | C2 | [ADR-0005](adr/0005-initial-snapshot-cursor-gate-brief-lock.md). Order-faithful integer/binary/composite PKs ship; a collation order Elixir cannot reproduce is a silent gap/dup, so it is refused rather than shipped wrong. Needs collation-aware cursor comparison |
| C2b | **`tables: :all` snapshot resolution** — resolve an `:all` snapshot set (which arises when the capture allowlist is itself `:all`) to a concrete, scoped table list. Refused today as `:config_invalid`. slug:c2-snapshot-tables-all | An `:all` snapshot backfills a well-defined, scoped set of base tables (never accidentally the whole server), enumerated from `information_schema` | C2 | [ADR-0005](adr/0005-initial-snapshot-cursor-gate-brief-lock.md). "Snapshot every table on the server" is ambiguous and dangerous, so C2 requires an explicit snapshot table list when capture is `:all`; a concrete capture allowlist (from which snapshot defaults) already works |
| C2c | **Zero-row snapshot table completion signal** — a configured snapshot table with no pre-existing rows takes the `{:done}` path and emits NO `handle_snapshot/2` beat, so the sink never receives `final_chunk?: true` for it. slug:c2-empty-table-final-signal | A configured snapshot table with zero rows delivers exactly one completion signal (an empty `final_chunk?: true` chunk, or an equivalent per-table done marker) so a sink gating per-table readiness on `final_chunk?` does not wait forever | C2 | Pre-existing behavior (unchanged by the F6 finality fix, which only corrects the exact-multiple *non-empty* last page). Surfaced by the closeout cross-vendor delta review; a design decision on the empty-table sink contract, deliberately not folded into the C2 closeout blocking-fix pass |

## C3-forward couplings (implementation notes, not backlog rows).

Two C1 sites are correct today but coupled to C3's `changes`-as-lazy-enumerable / a future multi-output assembler; each carries an inline guard comment:

- `lib/capstan/assembler_server.ex` — the filtered/empty dispatch matches `changes` as a concrete `[]`. When C3 makes `changes` a lazy single-pass enumerable, an empty `Stream` would not match `[]`; the Assembler must then emit an explicit filtered/empty signal rather than have this site infer it from the representation (`Enum.empty?/1` would consume a single-pass reader).
- `lib/capstan/assembler_server.ex` — DDL delivery reads the assembler's **post-step** position, exact only because `Assembler.step/2` emits at most one output per event in C1. A future multi-output step must carry a per-output position, or the watermark could advance past undelivered work (an effect-once violation).