Current section
Files
Jump to
Current section
Files
lib/resiliency/backoff_retry.ex
defmodule Resiliency.BackoffRetry do
@moduledoc """
Functional retry with backoff for Elixir.
`Resiliency.BackoffRetry` provides a simple `retry/2` function that executes a function
and retries on failure using composable, stream-based backoff strategies.
Zero macros, zero processes, injectable sleep for fast tests.
## When to use
* Calling an external HTTP API that occasionally returns transient 5xx errors
or connection timeouts — retry with exponential backoff to give the service
time to recover.
* Writing to a database that may temporarily reject connections under load —
retry with a time budget so the caller does not block indefinitely.
* Consuming messages from a queue where processing occasionally fails due to
upstream flakiness — retry a bounded number of times before dead-lettering.
* Performing DNS lookups or certificate refreshes at startup where a brief
network blip should not crash the application.
## How it works
`retry/2` executes the given zero-arity function and inspects its return value.
Any `{:ok, _}` or bare value is treated as success and returned immediately.
Any `{:error, _}`, raised exception, caught exit, or caught throw is treated as
failure. On failure the optional `:retry_if` predicate is consulted — if it
returns `false`, the error is returned at once.
When a retry is warranted, the next delay is pulled from a pre-built list of
delay values. That list is produced by taking `max_attempts - 1` elements from
an infinite `Stream` generated by `Resiliency.BackoffRetry.Backoff` (exponential,
linear, or constant), each capped at `:max_delay`. Before sleeping, the
optional `:on_retry` callback fires, then the configured `:sleep_fn` is called
with the delay in milliseconds.
A time `:budget` may be specified. Before each sleep, the engine checks whether
the remaining budget can absorb the upcoming delay. If not, retries stop and the
last error is returned. This provides a hard ceiling on total wall-clock time
independent of the number of attempts. When `:reraise` is `true` and the
original failure was a rescued exception, the exception is re-raised with its
original stacktrace once retries are exhausted — useful for letting crash
reporters capture the real origin.
## Algorithm Complexity
| Function | Time | Space |
|---|---|---|
| `retry/2` | O(n) where n = `max_attempts` — each attempt is O(1) overhead beyond the user function | O(n) — the pre-built delay list holds at most n - 1 elements |
| `abort/1` | O(1) | O(1) |
## Quick start
# Retry with defaults (3 attempts, exponential backoff)
{:ok, body} = Resiliency.BackoffRetry.retry(fn -> fetch(url) end)
# With options
{:ok, body} = Resiliency.BackoffRetry.retry(fn -> fetch(url) end,
backoff: :exponential,
max_attempts: 5,
retry_if: fn
{:error, :timeout} -> true
{:error, :econnrefused} -> true
_ -> false
end,
on_retry: fn attempt, delay, error ->
Logger.warning("Attempt \#{attempt} failed: \#{inspect(error)}")
end
)
## Options
* `:backoff` — `:exponential` (default), `:linear`, `:constant`, or any `Enumerable` of ms
* `:base_delay` — initial delay in ms (default: `100`)
* `:max_delay` — cap per-retry delay in ms (default: `5_000`)
* `:max_attempts` — total attempts including first (default: `3`)
* `:budget` — total time budget in ms (default: `:infinity`)
* `:retry_if` — `fn {:error, reason} -> boolean end` (default: retries all errors)
* `:on_retry` — `fn attempt, delay, error -> any` callback before sleep
* `:sleep_fn` — sleep function, defaults to `Process.sleep/1`
* `:reraise` — `true` to re-raise rescued exceptions with original stacktrace when retries are exhausted (default: `false`)
## Telemetry
All events are emitted in the caller's process. See `Resiliency.Telemetry` for the
complete event catalogue.
### `[:resiliency, :retry, :start]`
Emitted before the first attempt.
**Measurements**
| Key | Type | Description |
|-----|------|-------------|
| `system_time` | `integer` | `System.system_time()` at emission time |
**Metadata**
| Key | Type | Description |
|-----|------|-------------|
| `max_attempts` | `integer` | Configured maximum number of attempts |
### `[:resiliency, :retry, :stop]`
Emitted after the operation completes — either success or exhausted retries (without re-raise).
**Measurements**
| Key | Type | Description |
|-----|------|-------------|
| `duration` | `integer` | Elapsed native time units (`System.monotonic_time/0` delta) |
**Metadata**
| Key | Type | Description |
|-----|------|-------------|
| `max_attempts` | `integer` | Configured maximum number of attempts |
| `attempts` | `integer` | Actual number of attempts made |
| `result` | `:ok \| :error` | `:ok` on success, `:error` on failure |
### `[:resiliency, :retry, :exception]`
Emitted instead of `:stop` when `reraise: true` and a rescued exception exhausts all retries.
**Measurements**
| Key | Type | Description |
|-----|------|-------------|
| `duration` | `integer` | Elapsed native time units |
**Metadata**
| Key | Type | Description |
|-----|------|-------------|
| `max_attempts` | `integer` | Configured maximum number of attempts |
| `attempts` | `integer` | Actual number of attempts made |
| `kind` | `:error` | Always `:error` (rescued exception) |
| `reason` | `Exception.t()` | The exception struct |
| `stacktrace` | `list` | Original exception stacktrace |
### `[:resiliency, :retry, :retry]`
Emitted before each retry sleep, after a failed attempt that will be retried.
**Measurements**
| Key | Type | Description |
|-----|------|-------------|
| `delay` | `integer` | Sleep duration in milliseconds before next attempt |
**Metadata**
| Key | Type | Description |
|-----|------|-------------|
| `attempt` | `integer` | The attempt number that just failed (1-based) |
| `error` | `term` | The error that triggered the retry (`{:error, reason}` form) |
"""
defmodule Abort do
@moduledoc """
Wraps a reason to signal that retry should stop immediately.
Return `{:error, Resiliency.BackoffRetry.abort(reason)}` from the retried function
to abort without further attempts, regardless of `retry_if`.
"""
defstruct [:reason]
@type t :: %__MODULE__{reason: any()}
end
@type option ::
{:backoff, :exponential | :linear | :constant | Enumerable.t()}
| {:base_delay, non_neg_integer()}
| {:max_delay, non_neg_integer()}
| {:max_attempts, pos_integer()}
| {:budget, :infinity | non_neg_integer()}
| {:retry_if, (any() -> boolean())}
| {:on_retry, (pos_integer(), non_neg_integer(), any() -> any()) | nil}
| {:sleep_fn, (non_neg_integer() -> any())}
| {:reraise, boolean()}
@doc """
Creates an `%Abort{}` struct to signal immediate retry termination.
## Parameters
* `reason` -- any term describing why the retry should be aborted.
## Returns
A `Resiliency.BackoffRetry.Abort.t()` struct wrapping the given reason.
## Example
Resiliency.BackoffRetry.retry(fn ->
case api_call() do
{:error, :not_found} -> {:error, Resiliency.BackoffRetry.abort(:not_found)}
other -> other
end
end)
"""
@spec abort(any()) :: Abort.t()
def abort(reason), do: %Abort{reason: reason}
@doc """
Executes `fun` and retries on failure with configurable backoff.
See the module documentation for available options.
With `reraise: true`, re-raises rescued exceptions with the original
stacktrace when retries are exhausted instead of returning `{:error, exception}`.
## Parameters
* `fun` -- a zero-arity function to execute. Must return `{:ok, value}`, `{:error, reason}`, or a bare value (see result normalization in the module docs).
* `opts` -- keyword list of options. Defaults to `[]`.
* `:backoff` -- backoff strategy: `:exponential`, `:linear`, `:constant`, or any `Enumerable` of ms. Defaults to `:exponential`.
* `:base_delay` -- initial delay in milliseconds. Defaults to `100`.
* `:max_delay` -- cap per-retry delay in milliseconds. Defaults to `5_000`.
* `:max_attempts` -- total attempts including the first. Defaults to `3`.
* `:budget` -- total time budget in milliseconds. Defaults to `:infinity`.
* `:retry_if` -- `fn {:error, reason} -> boolean end` predicate controlling whether to retry. Defaults to retrying all errors.
* `:on_retry` -- `fn attempt, delay, error -> any` callback invoked before each sleep. Defaults to `nil`.
* `:sleep_fn` -- function used to sleep between retries. Defaults to `Process.sleep/1`.
* `:reraise` -- when `true`, re-raises rescued exceptions with the original stacktrace when retries are exhausted. Defaults to `false`.
## Returns
`{:ok, value}` on success, or `{:error, reason}` when all retries are exhausted (or the retry is aborted). When `reraise: true`, rescued exceptions are re-raised instead of being returned as errors.
"""
@spec retry((-> any()), [option()]) :: {:ok, any()} | {:error, any()}
def retry(fun, opts \\ []) do
max_attempts = Keyword.get(opts, :max_attempts, 3)
retry_if = Keyword.get(opts, :retry_if, fn {:error, _} -> true end)
on_retry = Keyword.get(opts, :on_retry)
sleep_fn = Keyword.get(opts, :sleep_fn, &Process.sleep/1)
budget = Keyword.get(opts, :budget, :infinity)
reraise = Keyword.get(opts, :reraise, false)
delays = build_delays(opts, max_attempts)
deadline =
case budget do
:infinity -> :infinity
ms -> System.monotonic_time(:millisecond) + ms
end
start_time = System.monotonic_time()
:telemetry.execute(
[:resiliency, :retry, :start],
%{system_time: System.system_time()},
%{max_attempts: max_attempts}
)
ctx = %{
fun: fun,
retry_if: retry_if,
on_retry: on_retry,
sleep_fn: sleep_fn,
deadline: deadline,
reraise: reraise,
start_time: start_time,
max_attempts: max_attempts
}
do_retry(ctx, delays, 1)
end
defp do_retry(ctx, delays, attempt) do
case execute(ctx.fun) do
{:ok, value} ->
duration = System.monotonic_time() - ctx.start_time
:telemetry.execute(
[:resiliency, :retry, :stop],
%{duration: duration},
%{max_attempts: ctx.max_attempts, attempts: attempt, result: :ok}
)
{:ok, value}
{:error, %Abort{reason: reason}} ->
duration = System.monotonic_time() - ctx.start_time
:telemetry.execute(
[:resiliency, :retry, :stop],
%{duration: duration},
%{max_attempts: ctx.max_attempts, attempts: attempt, result: :error}
)
{:error, reason}
{:error, {:__rescued__, exception, stacktrace}} ->
error = {:error, exception}
maybe_retry(ctx, delays, attempt, exception, error, stacktrace)
{:error, reason} = error ->
maybe_retry(ctx, delays, attempt, reason, error, nil)
end
end
defp maybe_retry(ctx, [], attempt, reason, _error, stacktrace) do
emit_retry_stop(ctx, attempt, reason, stacktrace)
maybe_reraise(ctx, reason, stacktrace)
end
defp maybe_retry(ctx, [delay | rest], attempt, reason, error, stacktrace) do
cond do
not ctx.retry_if.(error) ->
emit_retry_stop(ctx, attempt, reason, stacktrace)
maybe_reraise(ctx, reason, stacktrace)
budget_exceeded?(ctx.deadline, delay) ->
emit_retry_stop(ctx, attempt, reason, stacktrace)
maybe_reraise(ctx, reason, stacktrace)
true ->
:telemetry.execute(
[:resiliency, :retry, :retry],
%{delay: delay},
%{attempt: attempt, error: error}
)
if ctx.on_retry, do: ctx.on_retry.(attempt, delay, error)
ctx.sleep_fn.(delay)
do_retry(ctx, rest, attempt + 1)
end
end
defp emit_retry_stop(ctx, attempt, reason, stacktrace) do
duration = System.monotonic_time() - ctx.start_time
max_attempts = ctx.max_attempts
if ctx.reraise and is_exception(reason) and is_list(stacktrace) do
:telemetry.execute(
[:resiliency, :retry, :exception],
%{duration: duration},
%{
max_attempts: max_attempts,
attempts: attempt,
kind: :error,
reason: reason,
stacktrace: stacktrace
}
)
else
:telemetry.execute(
[:resiliency, :retry, :stop],
%{duration: duration},
%{max_attempts: max_attempts, attempts: attempt, result: :error}
)
end
end
defp maybe_reraise(%{reraise: true}, exception, stacktrace)
when is_exception(exception) and is_list(stacktrace) do
reraise exception, stacktrace
end
defp maybe_reraise(%{reraise: true}, {:exit, exit_reason}, _stacktrace) do
exit(exit_reason)
end
defp maybe_reraise(%{reraise: true}, {:throw, throw_value}, _stacktrace) do
throw(throw_value)
end
defp maybe_reraise(_ctx, reason, _stacktrace), do: {:error, reason}
defp execute(fun) do
normalize(fun.())
rescue
e -> {:error, {:__rescued__, e, __STACKTRACE__}}
catch
:exit, reason -> {:error, {:exit, reason}}
:throw, value -> {:error, {:throw, value}}
end
defp normalize({:ok, value}), do: {:ok, value}
defp normalize({:error, %Abort{}} = abort), do: abort
defp normalize({:error, reason}), do: {:error, reason}
defp normalize({:error, %Abort{} = abort, _extra}), do: {:error, abort}
defp normalize({:error, reason, extra}), do: {:error, {reason, extra}}
defp normalize(:ok), do: {:ok, :ok}
defp normalize(:error), do: {:error, :error}
defp normalize(value), do: {:ok, value}
defp build_delays(opts, max_attempts) do
delays_stream = build_delay_stream(opts)
delays_stream
|> Resiliency.BackoffRetry.Backoff.cap(Keyword.get(opts, :max_delay, 5_000))
|> Enum.take(max(max_attempts - 1, 0))
end
defp build_delay_stream(opts) do
case Keyword.get(opts, :backoff, :exponential) do
:exponential ->
Resiliency.BackoffRetry.Backoff.exponential(base: Keyword.get(opts, :base_delay, 100))
:linear ->
Resiliency.BackoffRetry.Backoff.linear(base: Keyword.get(opts, :base_delay, 100))
:constant ->
Resiliency.BackoffRetry.Backoff.constant(delay: Keyword.get(opts, :base_delay, 100))
delays when is_list(delays) ->
delays
%Stream{} = stream ->
stream
enumerable ->
enumerable
end
end
defp budget_exceeded?(:infinity, _delay), do: false
defp budget_exceeded?(deadline, delay) do
System.monotonic_time(:millisecond) + delay > deadline
end
end