bellwether

# Bellwether — BEAM Adapter

Crash reporting for embedded Linux, from the runtime's side.

This captures supervised crashes, redacts them **on the device**, and hands
them to the Bellwether Agent — a small native daemon that owns the spool and
the uplink. The Adapter never talks to the network: it runs inside a process
that may be in the middle of dying, so its job is to shape, fingerprint and
hand over, then get out of the way.

    def deps do
      [{:bellwether, "~> 0.1"}]
    end

## Wiring

One child. It starts the socket client and the Reporter and attaches the
`:logger` handler, so there is one place to configure rather than four to keep
in step.

    children = [
      {Bellwether.Supervisor,
       socket: "/tmp/bellwether.sock",
       own_apps: [:my_app],
       backtrace_depth: 32}
    ]

No call sites are needed. The BEAM puts `:crash_reason` into Logger metadata
for every crash a supervisor reports, so the handler catches the domain
without your code mentioning it.

Options may equally live in application config, where a release can change
them without a code change:

    config :bellwether,
      socket: "/tmp/bellwether.sock",
      own_apps: [:my_app],
      backtrace_depth: 32,
      keep_messages_for: [RuntimeError],
      block: ["otp:MyApp.KnownNoisy*"]

## The two settings that decide how good a report is

**`own_apps`** separates your code from the framework in a Fingerprint. Get it
wrong and every crash groups on whichever OTP function happened to sit above
yours.

**`backtrace_depth`** raises the VM's limit, which is **eight frames**. Eight
is often short of the call that explains a fault, and it costs grouping too: a
crash inside a dependency can carry none of your frames within eight, so
unrelated faults collapse onto the framework frames they share. It is a global
VM flag, which is why this sets it only when asked.

## Redaction

Every payload that leaves is a **Shape**: code-derived parts intact,
data-derived parts replaced by their type.

    %Lock{id: 4, site: "acme-1", state: :unlocked}
    #=> "%Lock{id: int, site: binary(6), state: :unlocked}"

Atoms, struct names and `module.function/arity` are written in source and
survive. Binaries, integers, floats, pids and refs are data and become their
type. On a fleet where process state holds site credentials, this is what
makes a crash reporter deployable at all.

The exception to know about: a message-carrying exception reports almost
nothing by default, because a message is where people interpolate an address
or a serial.

    %RuntimeError{message: binary(37)}      # by default
    %KeyError{key: :missing, term: %{site: binary(6)}}   # structured, and readable

If you know your own messages are safe, name them with `keep_messages_for`.
Only your application can judge that, which is why the default is the one that
cannot leak.

## Reporting something by hand

Supervised crashes arrive on their own. For anything else:

    # A fault you caught rather than let crash.
    Bellwether.capture_exception(e, __STACKTRACE__)

    # Something the runtime has no opinion about.
    Bellwether.capture_event(:coproc, :alarm, %{radio: :zwave, fault: :no_response},
      fingerprint: "radio:zwave:no_response")

Both Shape what you pass. Give anything recurring a `:fingerprint` — without
one the server has nothing to group on, which is right for a one-off and wrong
for everything else.

## The Agent

The Adapter is half of it. The Agent is a static binary that explains each
boot, harvests kernel panics and `erl_crash.dump`, holds the spool across
reboots, and uploads. On Nerves:

    mix bellwether.agent --endpoint https://... --key bwk_...

which installs it into `rootfs_overlay/` with a matching config, so the socket
path is stated once rather than in two files that must agree.

**`aarch64` is bundled with this package**, which covers most Nerves boards.
A Hex package cannot carry every target and two megabytes of the wrong
architecture helps nobody, so other targets are pointed at with `--from` or
built locally — `mix help bellwether.agent` lists where it looks.

Nothing is downloaded. A binary that arrived from the network with no
provenance is not something to put in firmware.

## What is in here

| Module | |
| --- | --- |
| `Bellwether` | `capture_exception/3`, `capture_event/4` |
| `Bellwether.Supervisor` | everything above, as one child |
| `Bellwether.Shape` | redaction — code survives, data becomes its type |
| `Bellwether.Fingerprint` | grouping, without line numbers or byte sizes |
| `Bellwether.Crash` | a crash reason and stacktrace to an Event payload |
| `Bellwether.Dedupe` | one exemplar per window, plus a count |
| `Bellwether.Filter` | faults an operator asked never to be told about |

Apache-2.0.

## Publishing

`priv/bin/` is committed, so refresh it before a release or the package ships
whichever Agent was built last:

    mix bellwether.bundle
    mix hex.publish

`mix test` fails if the binaries are missing or are not static aarch64. It
cannot tell a one-commit-stale binary from a current one, which is why the
first command is not optional.