Current section
Files
Jump to
Current section
Files
oban_sentinel
README.md
README.md
<p align="center">
<img src="assets/oban-sentinel-logo.png" width="180" alt="ObanSentinel logo">
</p>
# ObanSentinel
[](https://github.com/magnexis/oban_sentinel/actions/workflows/ci.yml)
[](https://hex.pm/packages/oban_sentinel)
[](LICENSE)
[](https://elixir-lang.org/)
[](https://magnexis.github.io/oban_sentinel/)
`ObanSentinel` is a telemetry-driven circuit breaker and execution guard for
[Oban](https://hex.pm/packages/oban). It pauses queues when a worker's rolling
failure rate indicates a dependency or code-path failure, then safely recovers
the queue after a cooldown. Worker guards prevent unbounded job duration and
process memory growth.
## Installation
Add the dependency:
```elixir
def deps do
[{:oban_sentinel, "~> 0.1.0"}]
end
```
See the complete [configuration reference](docs/CONFIGURATION.md) for every
plugin, policy, and webhook option.
`Req` is optional and only required for the built-in webhook notifier:
```elixir
{:req, "~> 0.5"}
```
## Oban configuration
Add Sentinel to the same configuration that starts your named Oban instance:
```elixir
# config/config.exs
config :my_app, Oban,
repo: MyApp.Repo,
queues: [critical: 10, default: 20],
plugins: [
{ObanSentinel.Plugin,
# `lookback` / `cooldown` are friendly aliases for the explicit *_ms keys.
lookback: :timer.minutes(5),
failure_threshold: 0.25,
minimum_samples: 20,
cooldown: :timer.minutes(5),
recovery_check_interval_ms: :timer.seconds(30),
# Set :emit_only first to observe breaches without pausing queues.
action: :pause_queue,
auto_recover: true,
max_samples_per_worker: 10_000,
notifier: ObanSentinel.Notifier.Webhook,
notifier_opts: [url: System.fetch_env!("SENTINEL_WEBHOOK_URL")]}
]
```
When using a named instance, Oban supplies `:oban` to plugins. The breaker
tracks each worker independently but pauses the worker's configured queue;
therefore workers with different risk profiles should use separate queues.
## Guarding a worker
The guard deliberately requires `perform_guarded/1`: macro expansion cannot
safely replace an already-defined `perform/1` in every Elixir compilation
order. `perform/1` remains the normal Oban callback generated by the guard.
```elixir
defmodule MyApp.ExternalApiWorker do
use Oban.Worker, queue: :critical, max_attempts: 8
use ObanSentinel.Guard,
max_duration: :timer.seconds(20),
max_memory_mb: 80
@impl Oban.Worker
def perform_guarded(%Oban.Job{args: %{"id" => id}}) do
MyApp.ExternalApi.sync(id)
end
end
```
Duration and memory breaches return `{:error, {:oban_sentinel, reason}}` to
Oban, which follows the worker's ordinary retry policy. The memory guard polls
the isolated task at 25 ms intervals. It bounds the task's BEAM heap; it is not
a substitute for OS/container memory limits or for validating untrusted code.
## Telemetry
Sentinel consumes Oban `:start`, `:stop`, and `:exception` job telemetry. It
emits these events:
* `[:oban_sentinel, :circuit_breaker, :tripped]`
* `[:oban_sentinel, :circuit_breaker, :recovered]`
Both emit metadata containing `:worker`, `:queue`, and the relevant
failure/cooldown fields. A `:recovered` event emits `%{}`; a `:tripped` event emits
`%{failure_rate: float(), queue_action: :ok | :skipped | :deferred | {:error, term()}}`.
Notification delivery runs in a separate task and never blocks Oban telemetry
handlers.
## Inspection and dry runs
Use `snapshot/2` to expose Sentinel state through your own authenticated admin
surface or metrics exporter. The ETS table remains private to the breaker
process, so callers never need direct table access.
```elixir
ObanSentinel.CircuitBreaker.snapshot(MyApp.ObanSentinelBreaker, MyApp.Workers.ApiSync)
# => %{status: :open, samples: 25, failures: 12, failure_rate: 0.48,
# starts: 25, duration: %{count: 25, average: 3_200_000, max: 9_000_000}}
```
For an authenticated operations surface, enumerate observed workers with
`ObanSentinel.CircuitBreaker.workers/1`. On-call operators can close a circuit
early with `reset/3`; it resumes the queue only when no other Sentinel circuit
for that queue remains open.
```elixir
:ok = ObanSentinel.CircuitBreaker.reset(MyApp.ObanSentinelBreaker, MyApp.Workers.ApiSync)
```
Telemetry durations use native `:telemetry` units. Avoid displaying them as
milliseconds unless your exporter explicitly converts them.
Guard limit breaches emit `[:oban_sentinel, :guard, :duration_exceeded]` or
`[:oban_sentinel, :guard, :memory_exceeded]`. Their metadata contains the
active limit in milliseconds (`:max_duration`) or bytes (`:max_memory`).
Set `action: :emit_only` during rollout to emit Sentinel events without calling
`Oban.pause_queue/2` or `Oban.resume_queue/2`. This is the recommended first
production deployment mode.
## Per-worker policies
Set stricter or more cautious policies for individual workers without running
another plugin. Oban reports worker names as strings, so policy keys should use
the worker's fully-qualified module name.
```elixir
policies: %{
"MyApp.Workers.PaymentCapture" => %{
failure_threshold: 0.10,
minimum_samples: 5,
cooldown_ms: :timer.minutes(10)
},
"MyApp.Workers.ExperimentalImport" => %{
action: :emit_only,
auto_recover: false
}
}
```
Policy fields inherit the plugin defaults when omitted. Keep workers with
different queue-control policies in separate queues: Oban’s pause/resume
operation controls the entire queue.
## Operational notes
* A resumed queue may retrip immediately if failures continue; that is the
expected protective behavior.
* Manual queue pauses are indistinguishable from Sentinel's pause to Oban. Do
not enable automatic recovery for queues you pause manually for maintenance.
* Set a sensible `minimum_samples` so a few transient failures cannot trip a
production queue.
* Sentinel can only safely resume queues it paused itself. If your operational
process manually pauses queues, set `auto_recover: false` for that instance.
* Memory guards observe the isolated task's BEAM memory. They cannot constrain
native allocations, ports, or memory consumed by external programs.
## Development
```sh
mix deps.get
mix format --check-formatted
mix test
```
See [ARCHITECTURE.md](ARCHITECTURE.md) for event flow and state ownership, and
[SECURITY.md](SECURITY.md) for operational boundaries and deployment guidance.