capstan

# capstan ROADMAP

Authored scope and acceptance definitions. Current executable work belongs to the
configured GitHub owner queue (`baselabs/capstan`; `.kimosabe/config.toml` selects
its labels). Reconcile that queue and retained acceptance evidence before picking
up or adding work; a roadmap row is not an installed task or completion receipt.
The earlier `.forge/` derived-board convention is historical, not current pickup.

**Source-currency amendment — September 21, 2026:** OBSERVED with `rg -n`, `nl -ba`, and `sed` reads: `mix.exs` declares 1.2.3; the 1.2.0 changelog records batching, type/compression breadth, XA tracking, sink-owned checkpoints, explicit starts, collation-string snapshots, table-set resolution, and empty-table completion. Source inspection confirms sink-owned initialization in `AssemblerServer`, selected-position handling in `Pipeline`, and XA prepare/recovery paths in `Assembler`/`Connection`. Earlier rows are authored requirements, not evidence that their original refusals remain active. [ADR 0004](adr/0004-c1-scope-lib-owned-checkpoint-only.md), [ADR 0006](adr/0006-xa-prepare-commit-tracking.md), and [ADR 0011](adr/0011-transaction-compression-precondition.md) carry their amendments. C6 remains generic producer/consumer conformance work; consumer domain adapters have separate owners. This reconciliation runs no runtime acceptance and authorizes no publication or consumer activation. Capstan remains an independently usable transport library.

| ID | What | Acceptance | Depends | Why |
|---|---|---|---|---|
| C1 | **Streaming spine** — connect to MySQL as a replica, tail the row-based binlog from a GTID position, assemble committed transactions, deliver to a sink, durably advance a processed-GTID checkpoint, and halt fail-closed on every silent-loss condition. slug:c1-streaming-spine | A sink receives every committed row change of the configured tables exactly once across a kill/restart, proven on live MySQL by an append-only ledger; every fail-closed halt has a red-capable tripwire (gap, precondition, unmapped table_id, XA, unknown type, compression, server_id conflict); Rule 1 proven on all four leak vectors; README + usage-rules + CHANGELOG authored | — | [ADR-0001](adr/0001-position-and-dedup-model.md), [ADR-0002](adr/0002-fail-closed-server-preconditions.md), [ADR-0003](adr/0003-transaction-shape-and-checkpoint-semantics.md) |
| C2 | **Initial snapshot** — consistent backfill of existing rows woven into the running stream, via cursor-gated suppression + a brief per-chunk `LOCK TABLES … READ` that captures an exact GTID position (the read-only-replica-safe substitute for DBLog's written watermark; MySQL has no `EXPORT_SNAPSHOT`). slug:c2-snapshot | A table with pre-existing rows is backfilled and then streamed with no gap and no duplicate at the handoff, proven live under concurrent load; resumable mid-backfill; strict-once in normal operation; each snapshot-path fail-closed halt tripwired; README + usage-rules + CHANGELOG authored | C1 | [ADR-0005](adr/0005-initial-snapshot-cursor-gate-brief-lock.md) (extends [ADR-0002](adr/0002-fail-closed-server-preconditions.md); honors [ADR-0001](adr/0001-position-and-dedup-model.md), [ADR-0003](adr/0003-transaction-shape-and-checkpoint-semantics.md), [ADR-0004](adr/0004-c1-scope-lib-owned-checkpoint-only.md)) |
| C3 | **Batching** — lib-owned batched checkpointing and sink-owned `handle_batch/1` atomic batch delivery. slug:c3-batching | Batch modes preserve their stated guarantees under kill/restart; the composition of each batch mode with every durability-signal path is tripwired (replicant's cross-mode-composition class) | C1 | TBD at design |
| C4 | **Type + payload breadth** — exotic column types (spatial, `SET`, pre-5.6 temporals) and zstd `TRANSACTION_PAYLOAD_EVENT` decompression. slug:c4-type-breadth | Each type previously halting `:unsupported_column_type` decodes correctly against captured real bytes; a compressed transaction decodes; the fail-closed halts remain for anything still unsupported | C1 | TBD at design |
| C5 | **XA transactions** — track `XA_PREPARE` → commit/rollback rather than halting. slug:c5-xa | A prepared-then-rolled-back XA transaction delivers zero rows; a prepared-then-committed one delivers exactly once; bounded state under an unbounded prepare window | C1 | TBD at design |
| C6 | **Sink-consumer contract and conformance** — Capstan owns the generic stream/checkpoint contract and nonconfidential executable consumer checks; downstream products own their domain adapters. slug:c6-downstream-adapter | Independently usable checks cover initial snapshot, schema changes, GTID-set position semantics (set membership, never an ordinal), crash/replay, lost response, and acknowledgment only after the sink’s durable data/position commit. Qualification names exact producer and consumer artifacts and retains the applicable evidence. Generic producer checks do not certify a particular consumer’s atomicity or domain composition; each consumer owns that connected receipt. Public consumer-package publication is not a prerequisite. Private acceptance evidence stays private and is not represented as publicly reproducible. No completion is asserted by this definition | C1 contract; no publication dependency | Reconcile the existing C6 obligation in the configured GitHub owner queue before adding work; executable pickup is unresolved in this document. First local action: define the generic contract/check coverage and retained evidence requirements. A reusable public adapter remains a separate explicit ownership decision |
| C7 | **Amazon Aurora MySQL as a named, tested source** (Aurora MySQL version 3, the MySQL 8.0-compatible line). Binlog through the cluster parameter group, GTID mode on, retention hours through `mysql.rds_set_configuration`, the cluster writer endpoint only, and what a failover does to `@@server_uuid`, `@@server_id` and the GTID set, each mapped onto the connect-time gate, `:source_identity_mismatch`, `:data_gap`, `:server_id_conflict` and the start-position rules. Adds `log_bin` to the precondition gate (`:binlog_disabled`), an Aurora section to `scripts/capstan-preflight.sql`, and a cycle-counter reset when `@@server_uuid` changes between cycles (a failover, not an eviction). slug:c7-aurora-mysql-source | On a real Aurora MySQL cluster (never a mock or a stand-in MySQL): the gate passes with the documented parameter group and refuses each documented misconfiguration by its distinct reason; a pipeline streams from the writer endpoint and resumes after a forced failover (`failover-db-cluster`) with loss 0 on an append-only ledger and the new writer's UUID in the checkpoint set; a reader endpoint is refused before the dump; a checkpoint behind the retention window halts `:data_gap`; `start_position: :current` seeds from the writer; the receipt (versions, variable values, endpoint role, outcomes; no credential value) is committed under a named path, or the tagged `:aurora_mysql` module runs in CI against a cluster the workflow creates and destroys | C1, C1b, C2 | Proposed 2026-09-29: [ADR-0013](adr/0013-aurora-mysql-source.md) (documentation-verified the same day — every fact checked against the AWS guide, the corrected table and the named live-probe remainder in its Evidence section); intent `aurora-mysql-source`. **LANDED 2026-09-30** — code half 2026-09-29 (six-variable gate with `:binlog_disabled`, the `@@server_uuid`-aware cycle reset, preflight/docs/Aurora-module wiring, red-first proven); the simulator arm (`scripts/aurora-sim/` + `:aurora_sim`) verified the failover contract locally on real MySQL; and the real-cluster run landed 2026-09-30 (`test/receipts/aurora-mysql-2026-09-30.md`): gate healthy + all six misconfigurations by distinct reason + reader refused before the dump + two forced failovers with loss 0 + snapshot-across-failover gap-free + `:current`, with two product fixes the run forced (TLS SNI default; budgeted `log_bin`) and its corrections (cluster-wide `@@server_uuid`) folded into ADR-0013 (Accepted). The retention `:data_gap` marquee needs a cluster past its retention window — the only outstanding observation |

## Backlog — deferred from C1 (ADR-0004)

Sink-owned checkpoint mode and explicit start positions were scoped in the C1 design (§ Scope item 7) and **deferred by user decision** (2026-07-21); C1 originally refused them fail-closed; the source-currency amendment above supersedes that implementation status. Authored requirements; current execution state comes from the configured owner queue and receipts.

| ID | What | Acceptance | Depends | Why |
|---|---|---|---|---|
| C1a | **Sink-owned checkpoint mode** — the sink persists the processed position atomically with its own data write and hands it back via `Capstan.Sink.checkpoint/0` (effect-once on the atomic path). The original `:sink_owned_mode_unsupported` refusal was retired in the recorded 1.2.0 delivery. slug:c1-sink-owned-checkpoint | A sink-owned pipeline runs: `AssemblerServer` reads the resume position from `Sink.checkpoint/0`, delivers, and the sink checkpoints atomically with its own write; a kill/restart replays zero and skips zero (effect-once), proven live on an append-only ledger | C1 | [ADR-0004](adr/0004-c1-scope-lib-owned-checkpoint-only.md). The sink contract, per-mode validation, and `AssemblerServer` sink-owned run path are present in current source; consumer atomicity acceptance remains required |
| C1b | **Explicit start positions** — honor `start_position:` as an explicit `%Capstan.Position{}` override and as `:current`, end-to-end. The original override/current refusals were retired in the recorded 1.2.0 delivery. slug:c1-explicit-start-position | A pipeline started with an explicit `%Position{}` resumes the `Connection` dump AND seeds the `AssemblerServer` watermark from the SAME position (no silent checkpoint hole); `:current` starts from the server's live position via a pre-connect query | C1 | [ADR-0004](adr/0004-c1-scope-lib-owned-checkpoint-only.md). The selected position must seed both `Connection` and `AssemblerServer`; current source threads the override to both |

## Backlog — deferred from C2 (ADR-0005)

The original C2 contract recorded two **loud fail-closed refusals** (ADR-0005 § Consequences) and one empty-table completion gap. The recorded 1.2.0 changes add table-set resolution and empty-table completion; the rows retain their acceptance obligations, not the original implementation status. Authored requirements; current execution state comes from the configured owner queue and receipts.

| ID | What | Acceptance | Depends | Why |
|---|---|---|---|---|
| C2a | **Collation-ordered-string PK** — snapshot a table whose primary key is a collation-ordered `CHAR`/`VARCHAR`/`TEXT` column. LANDED for `CHAR`/`VARCHAR` (ADR-0012 — server-computed weight bytes; any charset, any collation, composites); `TEXT` and `ENUM`/`SET` remain refused on measured grounds (per-page filesort; position-order divergence). slug:c2-collation-string-pk | A collation-string-PK table backfills gap-free/dup-free: the cursor-gate's `k ≤ cursor` comparison reproduces the source's collation order (not Elixir byte-order) so no row is mis-classified | C2 | [ADR-0005](adr/0005-initial-snapshot-cursor-gate-brief-lock.md). Order-faithful integer/binary/composite PKs ship; a collation order Elixir cannot reproduce is a silent gap/dup, so it is refused rather than shipped wrong. Needs collation-aware cursor comparison |
| C2b | **`tables: :all` snapshot resolution** — resolve an `:all` snapshot set (which arises when the capture allowlist is itself `:all`) to a concrete, scoped table list. The original `:config_invalid` refusal for this mode is superseded by the recorded 1.2.0 table-set resolution. slug:c2-snapshot-tables-all | An `:all` snapshot backfills a well-defined, scoped set of base tables (never accidentally the whole server), enumerated from `information_schema` | C2 | [ADR-0005](adr/0005-initial-snapshot-cursor-gate-brief-lock.md). Resolve a concrete table set from the admitted capture scope; never interpret `:all` as permission to expand that scope |
| C2c | **Zero-row snapshot table completion signal** — the original empty-table path emitted no completion beat; the recorded 1.2.0 change supplies an explicit final signal. slug:c2-empty-table-final-signal | A configured snapshot table with zero rows delivers exactly one completion signal (an empty `final_chunk?: true` chunk, or an equivalent per-table done marker) so a sink gating per-table readiness on `final_chunk?` does not wait forever | C2 | Historical closeout finding; the later 1.2.0 change records the selected empty-table final-chunk contract |

## C3-forward couplings (implementation notes, not backlog rows).

Two C1 sites are correct today but coupled to C3's `changes`-as-lazy-enumerable / a future multi-output assembler; each carries an inline guard comment:

- `lib/capstan/assembler_server.ex` — the filtered/empty dispatch matches `changes` as a concrete `[]`. When C3 makes `changes` a lazy single-pass enumerable, an empty `Stream` would not match `[]`; the Assembler must then emit an explicit filtered/empty signal rather than have this site infer it from the representation (`Enum.empty?/1` would consume a single-pass reader).
- `lib/capstan/assembler_server.ex` — DDL delivery reads the assembler's **post-step** position, exact only because `Assembler.step/2` emits at most one output per event in C1. A future multi-output step must carry a per-output position, or the watermark could advance past undelivered work (an effect-once violation).