Packages

CP key ownership for Elixir clusters: at most one node owns a given key, in every failure mode.

Current section

Files

Jump to
fief lib fief supervisor.ex
Raw

lib/fief/supervisor.ex

defmodule Fief.Supervisor do
@moduledoc """
Root supervisor of one Fief instance — a thin **outer** supervisor over two
children (`docs/leave-on-shutdown.md` §2.6):
1. `Fief.Kernel` — the instance's live machinery, a `rest_for_one`
supervisor whose children and ordering carry the fencing and
cache-survival semantics (see its moduledoc). Mounted
`significant: true`, `restart: :temporary`;
2. `Fief.Node.ShutdownDrain` — the shutdown-leave sentinel, appended
**last** and present only under the shape gate below.
## Fail-fast: a Fief node going down means this process exits
The outer never restarts the kernel. The kernel is a `rest_for_one`
supervisor with its own restart budget, so by the time the kernel *process*
exits it has already exhausted local recovery; an outer retry would just
replay a strategy that just failed. So the kernel is `significant` and the
outer runs `auto_shutdown: :any_significant` — a kernel exit brings the whole
instance down and `Fief.Supervisor` itself exits (reason `:shutdown`). A Fief
node going down surfaces as its top process exiting; the **user's**
supervision tree decides recovery (restart via a `:permanent`/`:transient`
child spec, or not). Fief does not hide a sick instance behind a silent
internal reboot — the failure path (lease expiry, takeover) is the verified
fallback, and a persistently sick node should become visibly dead quickly.
## The sentinel sits outside the kernel's restart unit
The split is load-bearing. Were the sentinel a flat tail child of the
`rest_for_one` kernel, an internal sibling crash (a `Fief.Vnode.Manager`
blip) would restart it and deliver `terminate(:shutdown, _)` while a healthy
`Fief.Node` is still alive — tripping a spurious full graceful drain on a
routine local fault. As an **outer** child it is never terminated for a
kernel-internal fault, so the only `:shutdown` it sees is a genuine teardown
of the whole instance. On that teardown the outer terminates children in
reverse start order — the sentinel (last) first, its drain running while the
kernel is still alive behind it (§2.3); and on a kernel exit that triggers
`auto_shutdown`, the sentinel is likewise terminated first, but the kernel is
already gone, so its `leave/1` no-ops on a dead node and it returns at once
(§2.6, §2.7 — the escalation path is self-gating).
The outer strategy is `:one_for_one` so a sentinel crash restarts only the
(inert) sentinel; the kernel's `significant` flag, not the strategy, governs
what a kernel exit does.
## Shutdown leave
`leave_on_shutdown: pos_integer() | :infinity | false`, default `20_000` ms.
When enabled (the default), tearing the instance down — `System.stop/0`,
SIGTERM, application shutdown, or the parent terminating this supervisor —
runs `Fief.Node.leave/1` and blocks the fall of the tree until the node
drains to `:stopped` or the deadline expires. The value **is** the
sentinel's child-spec `shutdown:` deadline; there is no second timer. On
timeout the supervisor kills the sentinel and the tree falls — a timed-out
shutdown is an ordinary node failure (`guides/protocols/failure.md`), no new
failure mode. Per-node tuning, not fingerprinted. Default 20 s sits under
the k8s 30 s `terminationGracePeriodSeconds` so the drain gets a real chance
before SIGKILL.
The sentinel is present only when **all** hold (else there is nothing to
drain, or nothing that could ever drain it, so a wait is a guaranteed
timeout): `leave_on_shutdown` is not `false`; a `:vnode_impl` is configured;
the planner subtree is enabled (`leadership` is not `false`); and the
instance is **not** sim-joined. A simulation drives its own lifecycle by
stepping scheduler events — there is no OTP-driven graceful shutdown to hook,
and a drain during ExUnit teardown could only spin against a quiescent
scheduler until the deadline (leave-on-shutdown §3.2).
"""
use Supervisor
@default_leave_on_shutdown 20_000
def start_link(opts) do
name = Keyword.fetch!(opts, :name)
Supervisor.start_link(__MODULE__, opts, name: Fief.Instance.supervisor_name(name))
end
@impl true
def init(opts) do
name = Keyword.fetch!(opts, :name)
if not is_atom(name), do: raise(ArgumentError, "Fief instance :name must be an atom")
leave_timeout = validate_leave_on_shutdown!(opts)
children =
Enum.reject(
[
kernel_child(opts),
shutdown_drain_child(name, opts, leave_timeout)
],
&is_nil/1
)
Supervisor.init(children, strategy: :one_for_one, auto_shutdown: :any_significant)
end
# The kernel is the instance's whole serving machinery, and it is
# `significant: true` + `restart: :temporary`: the outer never restarts it.
# The kernel is itself a `rest_for_one` supervisor with its own restart
# budget, so by the time the *kernel process* exits it has already tried and
# exhausted local recovery — an outer retry would only be an immediate replay
# of a strategy that just failed. Instead, `auto_shutdown: :any_significant`
# makes the kernel's exit bring the whole instance down (sentinel first, in
# reverse order — see below), so `Fief.Supervisor` exits and the *user's*
# supervisor decides recovery. That is the contract: a Fief node going down
# surfaces as its top process exiting, not as a silent internal reboot
# (`docs/leave-on-shutdown.md` §2.6).
defp kernel_child(opts) do
%{
id: Fief.Kernel,
start: {Fief.Kernel, :start_link, [opts]},
type: :supervisor,
restart: :temporary,
significant: true,
shutdown: :infinity
}
end
# The option value doubles as the sentinel's supervisor-enforced shutdown
# deadline (leave-on-shutdown §3.1), so it must be a legal child-spec
# `shutdown:` value: a positive-integer ms, `:infinity`, or `false` to
# disable the sentinel outright.
defp validate_leave_on_shutdown!(opts) do
case Keyword.get(opts, :leave_on_shutdown, @default_leave_on_shutdown) do
false ->
false
:infinity ->
:infinity
ms when is_integer(ms) and ms > 0 ->
ms
other ->
raise ArgumentError,
":leave_on_shutdown must be a positive integer (ms), :infinity, or false, " <>
"got: #{inspect(other)}"
end
end
# The shutdown-leave sentinel, gated by shape (§3.2). `nil` — no child —
# when the option is off, when no `:vnode_impl` is configured (nothing is
# owned, so nothing to drain), when the planner subtree is disabled
# (`leadership: false` — nothing would ever drain the node, so the wait
# could only time out), or when the instance is sim-joined (a simulation
# steps its own shutdown; an OTP-teardown drain would only spin against a
# quiescent scheduler until the deadline). The gate reads the raw opts, not
# the kernel's resolved leadership module: `leadership` anything other than
# `false` means the planner is intended, and the kernel raises on a bad
# value before the sentinel could matter.
defp shutdown_drain_child(_name, _opts, false), do: nil
defp shutdown_drain_child(name, opts, timeout) do
vnode_impl? = Keyword.get(opts, :vnode_impl) != nil
planner? = Keyword.get(opts, :leadership, :from_authority) != false
real? = Keyword.get(opts, :sim) == nil
if vnode_impl? and planner? and real? do
sentinel_opts = Keyword.take(opts, [:sim, :leave_drain_interval]) ++ [instance: name]
%{
id: Fief.Node.ShutdownDrain,
start: {Fief.Node.ShutdownDrain, :start_link, [sentinel_opts]},
shutdown: timeout,
type: :worker,
restart: :permanent
}
end
end
end