flowstone

# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.5.2] - 2025-12-22

### Added

#### Route Decisions Cleanup
- `FlowStone.RouteDecisions.cleanup_older_than/1` - Remove decisions older than a specified duration
- `FlowStone.RouteDecisions.cleanup_by_run_id/1` - Remove all decisions for a specific run
- Telemetry event `[:flowstone, :route_decisions, :cleanup]` emitted on cleanup

#### Resilient Resource Initialization
- `FlowStone.Resources` now continues startup even if some resources fail to initialize
- `FlowStone.Resources.list_failed/1` - Query resources that failed during setup
- Telemetry event `[:flowstone, :resources, :setup_failed]` emitted on resource failure
- Resources that fail return `{:error, {:setup_failed, reason}}` from `get/2`

#### Distributed Deployment Documentation
- Added comprehensive documentation for multi-node Oban deployments
- `FlowStone.RunConfig` moduledoc documents ETS limitations and workarounds
- README section on distributed deployment best practices
- Telemetry event `[:flowstone, :run_config, :fallback]` when falling back to app config

#### API Improvements
- `FlowStone.backfill/3` now caps `parallel` at 32 to prevent resource exhaustion
- `FlowStone.backfill/3` default timeout of 30 minutes per partition (was `:infinity`)
- `FlowStone.backfill/3` accepts `:timeout` option to customize per-partition timeout

#### Code Quality
- `FlowStone.Error.wrap_reason/1` - Shared function for error reason normalization
- Added typespecs to `FlowStone.Registry` public functions
- Improved JSON encoding error messages in `FlowStone.IO.Postgres`
- Fixed dialyzer warnings

### Changed

- `FlowStone.Resources.override/2` is now synchronous (returns `:ok` instead of casting)
- Resources state now tracks `:status` (`:ok` or `:failed`) for each resource

### Fixed

- Guard clause warnings in `load_deps/3`

## [0.5.1] - 2025-12-20

### Added

#### HTTP Client Resource
- `FlowStone.HTTP.Client` - HTTP client resource for REST API integrations
  - Implements `FlowStone.Resource` behaviour for lifecycle management
  - GET, POST, PUT, PATCH, DELETE operations with JSON body support
  - Configurable base URL, timeout, and headers
  - Automatic retry with exponential backoff and jitter
  - Rate limit handling with `Retry-After` header support
  - Idempotency key support for safe retries
  - Health check endpoint for resource monitoring
- `FlowStone.HTTP.Retry` - Retry logic with exponential backoff
  - Retries on 5xx server errors
  - Retries on 429 rate limit (respects `Retry-After` header)
  - Retries on transport errors (timeouts, connection refused)
  - Configurable max attempts, base delay, max delay, and jitter

#### Telemetry
- HTTP telemetry events: `[:flowstone, :http, :request, :start | :stop | :error]`
- Metadata includes method, URL, path, base_url, status, duration

### Documentation
- Added HTTP Client section to README
- Added HTTP module group to ExDoc configuration

## [0.5.0] - 2025-12-18

### Added

#### New Simplified API
- `FlowStone.run/2-3` - Run an asset with automatic dependency resolution
- `FlowStone.get/2-3` - Retrieve cached asset results
- `FlowStone.exists?/2-3` - Check if an asset result exists
- `FlowStone.invalidate/2-3` - Remove cached results
- `FlowStone.status/2-3` - Get asset execution status
- `FlowStone.backfill/3` - Process multiple partitions efficiently
- `FlowStone.graph/1-2` - Get DAG visualization (ASCII, Mermaid)
- `FlowStone.assets/1` - List all assets in a pipeline
- `FlowStone.asset_info/2` - Get detailed asset information

#### Configuration System
- `FlowStone.Config` - Centralized configuration with sensible defaults
- Auto-detection: Postgres storage when repo is configured
- Zero-config mode: Works with in-memory storage by default

#### DSL Improvements
- Short-form assets: `asset :name, do: {:ok, value}`
- Implicit result wrapping with `use FlowStone.Pipeline, wrap_results: true`
- Pipeline module injection: `MyPipeline.run(:asset)`, `MyPipeline.get(:asset)`
- Auto-generated `assets/0`, `graph/0`, `exists?/1` on pipeline modules

#### Test Helpers
- `FlowStone.Test` - Test module with isolation per test
- `run_asset/3` - Run assets with mocked dependencies
- `assert_asset_exists/3` and `refute_asset_exists/3` helpers

#### Error Improvements
- `FlowStone.Error.asset_not_found_with_suggestion/3` - Suggests similar asset names
- Structured errors with `type`, `message`, and `context` fields

### Changed
- Pipelines are now auto-registered on first use when using `FlowStone.run/3`
- Materializer supports `wrap_results` option for implicit `{:ok, _}` wrapping

### Notes
- Both the new simplified API (`run/3`, `get/3`, `backfill/3`) and the existing API
  (`materialize/2`, `register/2`) are fully supported
- The new API is recommended for new code; existing code continues to work unchanged

## [0.4.0] - 2025-12-18

### Added

- Conditional routing DSL: `route` (block or fn), `choice`, `default`, `on_error`, `routed_from`, `optional_deps`
- Routing decision persistence: `flowstone_route_decisions`, `FlowStone.RouteDecision`, `FlowStone.RouteDecisions`
- Routing telemetry events: `[:flowstone, :route, :start | :stop | :error]`
- Example `conditional_routing_example.exs` and routing docs/README sections
- Migration `0009_create_route_decisions.exs`
- Parallel branches DSL: `parallel`, `branch`, `parallel_options`, `join`
- Parallel persistence tables: `flowstone_parallel_executions`, `flowstone_parallel_branches`
- Parallel join worker with durable coordination and IO-based joins
- Parallel telemetry events: `[:flowstone, :parallel, :start | :stop | :error | :branch_start | :branch_complete | :branch_fail]`
- Example `parallel_branches_example.exs` and updated docs/README sections
- Migration `0010_create_parallel_tables.exs`
- ItemReader DSL: `scatter_from`, `item_selector`, source config macros, and reader options (`mode`, `batch_size`, `max_batches`)
- `FlowStone.Scatter.ItemReader` behavior with built-in readers (S3, DynamoDB, Postgres, Custom)
- `FlowStone.Workers.ItemReaderWorker` for distributed reader execution
- Telemetry events for ItemReader and distributed scatter batches
- Example `item_reader_example.exs` and ItemReader docs/README sections
- Migration `0011_add_item_reader_fields.exs`
- ItemBatcher DSL: `batch_options`, `max_items_per_batch`, `max_bytes_per_batch`, `size_fn`, `group_by`, `max_items_per_group`, `batch_fn`, `batch_input`, `on_item_error`
- `FlowStone.Scatter.BatchOptions` struct for batch configuration
- `FlowStone.Scatter.Batcher` module for batch grouping strategies (fixed size, size-based, group_by, custom)
- `FlowStone.Workers.ScatterBatchWorker` for batched scatter execution
- Context batch fields: `batch_index`, `batch_count`, `batch_items`, `batch_input`
- Telemetry events for batching: `[:flowstone, :scatter, :batch_create | :batch_start | :batch_complete | :batch_fail]`
- Example `item_batcher_example.exs` and ItemBatcher docs/README sections
- Migration `0012_add_item_batcher_fields.exs`

### Changed

- Materialization status now includes `:skipped` for unselected routed assets
- Executor/materializer/worker honor persisted routing decisions and skip IO writes when not selected
- Optional dependencies resolve to `nil` and are ignored in dependency gating and lineage
- DAG validation enforces routed_from router assets, optional_deps subset, and implicit router edges
- Executor handles parallel pending results without storing IO; join worker records final materializations
- DAG adds virtual edges from parallel assets to branch finals; lineage includes branch finals on join
- Scatter barriers no longer store scatter keys; keys are stored in `flowstone_scatter_results` only
- Scatter barriers allow `total_count` to start at 0 for streaming readers

## [0.3.0] - 2025-12-18

### Added

#### Scatter (Dynamic Fan-Out)
- `FlowStone.Scatter` - Core module for runtime-discovered parallel execution
- `FlowStone.Scatter.Barrier` - Ecto schema tracking scatter completion
- `FlowStone.Scatter.Key` - Scatter key serialization and hashing
- `FlowStone.Scatter.Options` - Configurable scatter execution options
- `FlowStone.Scatter.Result` - Ecto schema for scatter instance results
- `FlowStone.Workers.ScatterWorker` - Oban worker for scatter instances
- DSL macros: `scatter`, `scatter_options`, `gather`, `max_concurrent`, `rate_limit`, `failure_threshold`, `failure_mode`
- Migration `0006_create_scatter_tables.exs` - Barrier and result tables

#### Signal Gate (Durable External Suspension)
- `FlowStone.SignalGate` - Core module for durable external suspension
- `FlowStone.SignalGate.Gate` - Ecto schema for signal gates
- `FlowStone.SignalGate.Token` - HMAC-signed token generation and validation
- `FlowStone.Workers.SignalGateTimeoutWorker` - Timeout handling worker
- `FlowStone.Workers.SignalGateSweeper` - Periodic sweep for missed timeouts
- DSL macros: `on_signal`, `on_timeout`
- Migration `0007_create_signal_gate_tables.exs` - Gate table with token indexing
- Secure callback URL generation with expiring signed tokens

#### Global Rate Limiting
- `FlowStone.RateLimiter` - Distributed rate limiting using Hammer and Postgres advisory locks
- Token bucket rate limiting with `check/2`, `with_limit/4`
- Semaphore-based concurrency control with `acquire_slot/2`, `release_slot/2`, `with_slot/4`
- Status inspection and bucket reset

#### Observability
- New telemetry events for Scatter: `[:flowstone, :scatter, :start | :complete | :failed | :cancel | :instance_complete | :instance_fail | :gather_ready]`
- New telemetry events for Signal Gate: `[:flowstone, :signal_gate, :create | :signal | :timeout | :timeout_retry | :cancel]`
- New telemetry events for Rate Limiting: `[:flowstone, :rate_limit, :check | :wait | :slot_acquired | :slot_released]`

#### Examples
- `scatter_example.exs` - Web scraping with parallel execution
- `signal_gate_example.exs` - External task integration with callbacks
- `rate_limiter_example.exs` - Rate limiting patterns

### Changed
- `FlowStone.Asset` struct now includes scatter and signal gate fields
- `FlowStone.Pipeline` DSL expanded with scatter and signal gate macros
- Updated documentation groups in mix.exs for new modules

## [0.2.0] - 2025-12-14

### Added
- `FlowStone.RunConfig` - ETS-based store for runtime configuration that survives Oban job persistence
- Migration for unique index on materializations (asset_name, partition, run_id)
- Safe atom handling throughout - no unbounded atom creation from external inputs
- SQL injection protection in Postgres IO manager with identifier validation
- Safe binary term decoding using `:erlang.binary_to_term/2` with `[:safe]` option

### Changed
- **BREAKING**: Oban job args are now JSON-safe - only strings, numbers, booleans, and maps
  - Asset names stored as strings (resolved via registry lookup)
  - Partitions serialized using tagged format
  - Runtime config (servers, IO settings) stored in RunConfig keyed by run_id
- **BREAKING**: Partition serialization uses tagged format for unambiguous round-trip
  - Dates: `d:2024-01-01`
  - DateTimes: `dt:2024-01-01T00:00:00Z`
  - Tuples: `t:base64_encoded_json`
  - Strings with special chars: `s:base64_encoded`
  - Legacy `|` separator format still supported for reading (backward compatibility)
- `FlowStone.Error` is now a proper `defexception` - can be raised directly
- Resource injection uses `get_with_default` - nil no longer overrides defaults
- Materializations use upsert logic with unique constraint for idempotent writes
- `MaterializationContext.latest/3` uses deterministic ordering (by started_at, then inserted_at)
- IO.Memory normalizes keys for consistent lookups regardless of partition format

### Fixed
- Oban args JSON round-trip - jobs survive database persistence correctly
- Resource server defaults not being overridden by nil values
- Partition serialization collision when strings contain `|` character
- Non-deterministic "latest" query in fallback MaterializationStore
- Potential SQL injection in Postgres IO manager table/column names
- Unsafe atom creation from persisted/external string inputs

### Security
- Identifier validation in Postgres IO manager prevents SQL injection
- Safe binary decoding prevents untrusted data attacks
- No unbounded atom creation from external inputs (use String.to_existing_atom/1)

## [0.1.0]

### Initial Release

[0.5.2]: https://github.com/nshkrdotcom/flowstone/compare/v0.5.1...v0.5.2
[0.5.1]: https://github.com/nshkrdotcom/flowstone/compare/v0.5.0...v0.5.1
[0.5.0]: https://github.com/nshkrdotcom/flowstone/compare/v0.4.0...v0.5.0
[0.4.0]: https://github.com/nshkrdotcom/flowstone/compare/v0.3.0...v0.4.0
[0.3.0]: https://github.com/nshkrdotcom/flowstone/compare/v0.2.0...v0.3.0
[0.2.0]: https://github.com/nshkrdotcom/flowstone/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/nshkrdotcom/flowstone/releases/tag/v0.1.0