oban_sentinel

# Architecture

ObanSentinel is a local extension to a single Oban instance. It has no database
tables, background network connection, account requirement, or shared state.

## Event flow

```text
Oban job telemetry
  -> ObanSentinel.Plugin telemetry handler
  -> non-blocking GenServer cast
  -> CircuitBreaker-owned private ETS table
  -> threshold / cooldown state transition
  -> optional Oban queue action + Sentinel telemetry + optional notifier task
```

The telemetry callback only pattern-matches metadata and casts to the breaker.
It never calls Oban queue APIs or a webhook synchronously.

## State ownership

Each plugin starts one linked `CircuitBreaker`. Its private ETS table stores
bounded, timestamped worker outcomes; execution starts; rolling duration samples;
and open-circuit timestamps. All ETS reads and writes are performed by the
breaker process, including `snapshot/2` reads.

This avoids a globally named ETS table, supports multiple named Oban instances,
and prevents external processes from mutating safety state.

## Circuit semantics

Circuits are keyed by worker and tracked over `window_ms` (or `lookback`). A
circuit opens after at least `minimum_samples` when the ratio of failures meets
or exceeds `failure_threshold`. The default action pauses the job's queue.
Per-worker policies may override thresholds, sample minimums, cooldowns, and
queue action mode.

Recovery is best effort after `cooldown_ms` (or `cooldown`). A shared queue is
only resumed after the final open circuit associated with it has cooled down.
Use distinct queues for workers with distinct failure domains whenever possible.
Operators may use `reset/3` to recover a circuit early; it follows the same
shared-queue safety rule.

## Guard semantics

`ObanSentinel.Guard` runs `perform_guarded/1` in an isolated task. The calling
Oban executor remains alive when Sentinel terminates that task for duration or
BEAM-process-memory limits. It returns a structured error, allowing normal
Oban retry and backoff behavior to apply.

Guard limit breaches also emit Sentinel telemetry, allowing alerting without
polling worker logs.