Packages
Dead-man's-switch / heartbeat monitor — flags a job as down when it stops checking in — with its dashboard panel bundled in the same package as a separate module (Integrations.Heartbeat.Display) — one install, both halves; a release without raven_web simply runs the monitor headless.
Current section
Files
Jump to
Current section
Files
lib/integrations/heartbeat.ex
defmodule Integrations.Heartbeat do
@moduledoc """
Dead-man's-switch / heartbeat monitor.
Unlike every other integration, this one never reaches out to a target —
there's nothing to poll for a cron job, backup script, or batch pipeline.
Instead, the *job* checks in by hitting Raven's `/checkin/:checkin_id` HTTP
endpoint (any method, no auth beyond the id itself — same model as
Healthchecks.io/Cronitor ping URLs) each time it runs successfully. This
monitor flags itself unhealthy when a check-in is *missing* for too long,
not when one fails.
Collection only — see `Integrations.Heartbeat.Display` (same package) for
the dashboard panel. `Display.BundledDefault` auto-hooks it whenever this
monitor starts, same as a single-module package would; a release without
`raven_web` simply never compiles the display half and runs this monitor
headless.
## Params
* `:checkin_id` — the id this monitor listens for. Required.
Generate an unguessable value yourself (e.g.
`openssl rand -hex 16`) — treat it like a
password, since knowing it is enough to fake a
"still alive" signal. Configure the external job
to call `POST /checkin/<checkin_id>` on success.
* `:down_after_ms` — no check-in for this long → `:down`. Required.
* `:degraded_after_ms` — no check-in for this long → `:degraded`.
Optional; the degraded state is skipped if unset.
Size these with margin over the job's real schedule (2-3x its interval) to
absorb ordinary scheduling jitter without a false alarm.
## Health signal
* `:up` — a check-in arrived within `:degraded_after_ms` (or
`:down_after_ms` if degraded isn't configured).
* `:degraded` — no check-in for longer than `:degraded_after_ms` but
less than `:down_after_ms`.
* `:down` — no check-in for longer than `:down_after_ms`.
A freshly (re)started monitor uses its own boot time as the baseline, not
an immediate `:down` — a Raven restart shouldn't itself look like an
outage before the job has had a chance to check in even once. Whether a
monitor was checking in before the restart is not persisted; the grace
window simply starts over.
"""
use CodeNameRaven.Monitor
use CodeNameRaven.Monitor.Checkin
@impl true
def params_template do
%{
checkin_id: "",
down_after_ms: "600000",
degraded_after_ms: "300000"
}
end
@impl true
def params_schema do
[
checkin_id: [type: :string, required: true, doc: "Id the external job POSTs to /checkin/:checkin_id"],
down_after_ms: [type: :non_neg_integer, required: true, doc: "No check-in for this long -> :down"],
degraded_after_ms: [type: :non_neg_integer, required: false, doc: "No check-in for this long -> :degraded"]
]
end
@impl true
def target_uri(params) do
case get_param(params, :checkin_id) do
nil -> :none
id -> {:ok, "heartbeat:#{id}"}
end
end
@impl true
def identity_params(params) do
%{checkin_id: get_param(params, :checkin_id)}
end
# ---------------------------------------------------------------------------
# Init — Raven subscribes this monitor to check-ins automatically, since
# it's CodeNameRaven.Monitor.Checkin-shaped (see on_checkin/3 below)
# ---------------------------------------------------------------------------
@impl true
def init(_params) do
{:ok, %{last_checkin_at: nil, started_at: DateTime.utc_now()}}
end
@impl true
def on_checkin(%{checkin_id: incoming_id}, params, state) do
if incoming_id == get_param(params, :checkin_id) do
{:ok, %{state | last_checkin_at: DateTime.utc_now()}, :check_now}
else
{:ok, state}
end
end
# ---------------------------------------------------------------------------
# Collect — no network call, just a staleness check against our own state
# ---------------------------------------------------------------------------
@impl true
def collect(params, state) do
checkin_id = get_param(params, :checkin_id)
down_after_ms = parse_int(get_param(params, :down_after_ms))
degraded_after_ms = parse_int(get_param(params, :degraded_after_ms))
cond do
is_nil(checkin_id) ->
{:error, "missing required param :checkin_id", state}
is_nil(down_after_ms) ->
{:error, "missing required param :down_after_ms", state}
true ->
now = DateTime.utc_now()
baseline = state.last_checkin_at || state.started_at
elapsed_ms = DateTime.diff(now, baseline, :millisecond)
result = %{
elapsed_ms: elapsed_ms,
last_checkin_at: state.last_checkin_at,
down_after_ms: down_after_ms,
degraded_after_ms: degraded_after_ms
}
{:ok, result, state}
end
end
# ---------------------------------------------------------------------------
# Healthy?
# ---------------------------------------------------------------------------
@impl true
def healthy?(%{elapsed_ms: elapsed_ms, down_after_ms: down_after_ms, degraded_after_ms: degraded_after_ms}) do
cond do
elapsed_ms > down_after_ms -> :down
degraded_after_ms && elapsed_ms > degraded_after_ms -> :degraded
true -> :up
end
end
# ---------------------------------------------------------------------------
# Metrics
# ---------------------------------------------------------------------------
@impl true
def metrics(result) do
%{elapsed_ms: result.elapsed_ms}
end
# ---------------------------------------------------------------------------
# Private
# ---------------------------------------------------------------------------
defp get_param(params, key) when is_atom(key) do
v =
case Map.fetch(params, key) do
{:ok, val} -> val
:error -> params[to_string(key)]
end
if is_binary(v) and String.trim(v) == "", do: nil, else: v
end
defp parse_int(nil), do: nil
defp parse_int(v) when is_integer(v), do: v
defp parse_int(v) when is_binary(v) do
case Integer.parse(v) do
{n, _} -> n
:error -> nil
end
end
defp parse_int(_), do: nil
end