libcluster_mesh

# libcluster_mesh

[![CI](https://github.com/kentaro/libcluster_mesh/actions/workflows/ci.yml/badge.svg)](https://github.com/kentaro/libcluster_mesh/actions/workflows/ci.yml)

[libcluster](https://github.com/bitwalker/libcluster) strategies for mesh VPNs
that discover peers by **polling the local mesh daemon** — no cloud API, no
API keys, no static host lists.

Four strategies are included:

| Strategy | Mesh | Peer source | Member selection |
|---|---|---|---|
| `Cluster.Strategy.TailscaleLocal` | [Tailscale](https://tailscale.com/) (incl. [Headscale](https://github.com/juanfont/headscale)-based networks) | `tailscale status --json` | hostname prefixes |
| `Cluster.Strategy.Netbird` | [NetBird](https://netbird.io/) (cloud or self-hosted) | `netbird status --json` | hostname prefixes |
| `Cluster.Strategy.LolipopZTL` | [LOLIPOP Zero Trust Link](https://lolipop.jp/) | `ztlctl status` | hostname prefixes |
| `Cluster.Strategy.WireGuard` | plain [WireGuard](https://www.wireguard.com/) (hand-built meshes) | `wg show <if> dump` | IPv4 CIDR |

Because peers are read from the node's own daemon, these strategies work with
self-hosted control servers where cloud-API-based strategies (such as
[libcluster_tailscale](https://github.com/moomerman/libcluster_tailscale))
cannot: the daemon already holds the full netmap, including each peer's mesh
IP, hostname, and online state. Node discovery generates no traffic outside
the host, and distribution traffic flows only over the encrypted mesh.

## Installation

```elixir
def deps do
  [
    {:libcluster_mesh, "~> 0.1"}
  ]
end
```

Requires Elixir 1.18+, for the built-in `JSON` module. That is the only
requirement it adds: Elixir 1.18 already requires OTP 25+, which is well past
the OTP 23.1 that made `-erl_epmd_port` work, and the EPMD-less setup below
needs both that and `-start_epmd false`.

## Prerequisites

This library only does peer discovery. Joining the mesh — installing the
daemon, authenticating, connecting — is out of scope and must be completed
on **every node** before the strategies are of any use:

1. **The node is a connected mesh member.** Tailscale: `tailscale up`
   (browser login or `--authkey tskey-auth-…`). NetBird:
   `netbird up --setup-key …`. ZTL:
   `ztlctl login --controller-url … --controller-auth-key …` followed by
   `ztlctl up`. Plain WireGuard: the interface is configured with keys and
   peers. Authentication state is persisted by the daemon, so this is a
   one-time provisioning step per node — bake it into cloud-init, your
   container entrypoint, or whatever provisions the machine. The library
   itself never touches mesh credentials.
2. **The daemon and its CLI are present.** Tailscale: `tailscaled` running
   and the `tailscale` CLI on `$PATH` (or set `cli:`). NetBird: the
   `netbird` daemon running and its CLI on `$PATH`. ZTL: `ztld` running in
   TUN mode and `ztlctl` on `$PATH` (or set `cli:` / `socket:`). WireGuard:
   the interface up and `wg` on `$PATH`.
3. **The BEAM's OS user can query the daemon.** Discovery goes over the
   daemon's local socket, so its file permissions are the access control:
   for Tailscale on Linux, root or the user named in
   `tailscale set --operator=<user>`; for ZTL, the socket owner (root mode:
   0660 root, userspace mode: the owning user); for NetBird and `wg show`,
   typically root.

4. **The node's own mesh hostname matches what the others look for.** On the
   named meshes, `hostnames:` decides which peers *this* node will dial. It
   does not decide who may dial this one, and nothing here asks the mesh to
   advertise a particular name — the daemon does that, from the machine's
   hostname, when it joins:

   ```sh
   tailscale up --hostname=myapp-1
   netbird up --hostname myapp-1 --setup-key …
   ztlctl set --hostname myapp-1
   ```

   A daemon may take the machine's hostname instead, and keep the name it
   registered with — in which case the command above reports success and
   changes nothing. Check the daemon's own `status` rather than its exit code.
   Where that is what it does, the machine's hostname has to be right *before*
   the daemon first registers.

   Erlang distribution is symmetric, so a node with a name nobody recognises
   still joins the cluster if it dials someone who is in it. That is why
   getting this wrong often looks like it works. It stops working when no node
   has a matching name — a fleet of embedded devices all still called
   `nerves-a1b2`, say — and then nothing dials anything, forever, and no
   error is logged because an empty peer list is what a quiet mesh looks like.

Once a node satisfies these, starting the application is all that remains —
discovery and clustering are automatic from there.

## Security model

**The Erlang cookie is the authentication.** Nothing else here is.

`hostnames` and `cidr` are discovery filters: they decide which of the
daemon's peers are worth dialling, so a strategy does not spend every poll
connecting to laptops and phones. They are not a trust boundary. Any device on
the mesh that calls itself `myapp-evil` passes the prefix test, and any address
inside the CIDR passes that one. What stops it is the cookie it does not have.

So: give every node the same cookie, keep it out of source, and treat anything
that can reach the distribution port as able to try. Two things follow from
that being true —

- **Distribution traffic is not encrypted by this library.** It is encrypted
  because the mesh encrypts everything, which is the point of running over
  one. Bind the distribution listener to the mesh interface
  (`inet_dist_use_interface`) so a node with another interface up is not
  reachable on it.
- **Revoking a device from the mesh does not immediately eject it from the
  cluster.** An established connection lives until `net_ticktime` notices it
  is gone — 60 seconds by default.

## Nerves

These strategies run on [Nerves](https://nerves-project.org). The
prerequisites above are still the whole of the work, but a Nerves device
supplies none of them for you, and four things about the platform shape how
you meet them: the root filesystem is read-only, there is no init system, the
VM boots without distribution, and the kernel module the mesh needs is present
but not loaded.

### Picking a system

Every official system for real hardware ships `CONFIG_TUN=m` and
`CONFIG_WIREGUARD=m` — rpi0 through rpi5, bbb, grisp2, trellis — so `tun.ko`
and `wireguard.ko` are in the image and no custom system is needed.
`nerves_system_x86_64` has neither; on that target everything below requires
building a system of your own first.

What the images do not carry is `iptables`, `iproute2` and `wg`. The first two
are not missed: Tailscale-derived daemons configure their interface and routes
over netlink. `wg` is another matter — it is the entirety of
`Cluster.Strategy.WireGuard`'s input, and no official system sets
`BR2_PACKAGE_WIREGUARD_TOOLS`. Point `cli:` at the `wg` that
[vintage_net_wireguard](https://github.com/nerves-networking/vintage_net_wireguard)
builds into its `priv`, or enable the package in a custom system. The other
three strategies need nothing that is not already in the image.

### Putting the daemon in the firmware

These meshes distribute static Go binaries, which makes this the easy part:
drop the daemon and its CLI into the overlay and they are in the image with
their execute bits intact.

```
rootfs_overlay/usr/bin/ztld      # or tailscaled, netbird
rootfs_overlay/usr/bin/ztlctl    # the CLI the strategy polls
```

```elixir
# mix.exs
{:libcluster_mesh, "~> 0.1"},
{:muontrap, "~> 1.6"}
```

Give the strategy the absolute path with `cli:` rather than relying on `$PATH`.

Four things then differ from starting the same daemon on an ordinary host:

**Load `tun` first.** The module is in the image but nothing has loaded it, and
`/dev/net/tun` does not exist until something does. Tailscale-derived daemons
run `/sbin/modprobe` themselves and busybox puts modprobe exactly there, so
this is belt-and-braces — but it turns a confusing daemon failure into an
obvious one.

```elixir
System.cmd("/sbin/modprobe", ["tun"])
```

**Put the daemon's state under `/data`.** Defaults point at `/var/lib`, which
is read-only. `/data` is the writable partition and survives firmware updates.
Every daemon has a flag for this and they disagree on its name
(`tailscaled --statedir`, `ztld --state-dir`).

**Turn the daemon's DNS handling off.** By default these daemons rewrite
`/etc/resolv.conf`, which on Nerves is a symlink into `/tmp` owned by
`vintage_net`. The attempt fails, and it takes the daemon's whole backend with
it — reported as nothing more specific than a backend that would not
initialize. `ztlctl set --dns-mode none`, `tailscaled --no-dns`.

**Do not use userspace networking.** It looks like the right choice for an
embedded system and it is not: in that mode outbound traffic only leaves
through a SOCKS5 proxy, so `:gen_tcp.connect` to a peer's mesh address times
out and Erlang distribution has no way through. In kernel TUN mode the same
connections succeed, and the module is already in the image.

**Supervise it with
[MuonTrap](https://hexdocs.pm/muontrap/MuonTrap.Daemon.html).** A plain `Port`
leaves the OS process alive when the BEAM process that owns it dies. There is
no init system here; the supervision tree is it.

How the daemon authenticates is its own business and unchanged by Nerves: an
auth key for unattended joins, or the interactive flow where `up` prints a URL
for someone to authorize the device at. Worth knowing only that a key reached
through `config :nerves, :erlinit, env: …` is baked into the image, and so is
shared by every device flashed with that firmware.

### Distribution

Nerves boots the VM undistributed, and the node has to be named for the
address the mesh assigns — which nothing knows at boot. So distribution starts
from your code, after the daemon reports an address, and `Cluster.Supervisor`
starts after that: a strategy that comes up before the node is named has
nothing to connect from.

EPMD is not running either. Take the EPMD-less route described under
[Usage](#usage) and put the flags in `rel/vm.args.eex`, among the others
rather than at the end of the file — as its comments explain, everything after
its `-extra` line is passed on as a plain argument instead of to the emulator.

```
-start_epmd false
-erl_epmd_port 45892
```

`:init.get_argument(:start_epmd)` answers `{:ok, [[~c"false"]]}` once the VM
has them. If you would rather keep EPMD, it is inside the release's own ERTS
and starting it is enough:

```elixir
System.cmd("/srv/erlang/erts-#{:erlang.system_info(:version)}/bin/epmd", ["-daemon"])
```

### All of it together

One child of your supervision tree owns the sequence. It is a chain of
`handle_continue/2` because each step needs the one before it: no address
until the daemon has connected, no node name until there is an address,
nothing for a strategy to connect from until the node has a name.

```elixir
defmodule MyApp.Mesh do
  use GenServer
  require Logger

  @sock "/run/lolipopztl/ztld.sock"
  @state_dir "/data/lolipopztl"

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  @impl true
  def init(_opts), do: {:ok, %{}, {:continue, :daemon}}

  @impl true
  def handle_continue(:daemon, state) do
    File.mkdir_p!(@state_dir)
    File.mkdir_p!(Path.dirname(@sock))
    # Belt and braces, so it must not be the thing that stops the node: the
    # daemon loads this itself, and a system without modprobe is one where it
    # is built in.
    if File.exists?("/sbin/modprobe"), do: System.cmd("/sbin/modprobe", ["tun"])

    {:ok, _} =
      MuonTrap.Daemon.start_link(
        "/usr/bin/ztld",
        ["--state-dir=#{@state_dir}", "--socket=#{@sock}"],
        log_output: :info,
        name: :ztld
      )

    {:noreply, state, {:continue, :join}}
  end

  def handle_continue(:join, state) do
    wait_for(fn -> File.exists?(@sock) end)

    # /etc is read-only, and the daemon takes its whole backend down rather
    # than carrying on without being able to write resolv.conf.
    ztlctl(["set", "--dns-mode", "none"])
    ztlctl(["login", "--controller-url", url(), "--controller-auth-key", key()])
    ztlctl(["up"])

    {:noreply, state, {:continue, {:cluster, wait_for(&address/0)}}}
  end

  def handle_continue({:cluster, ip}, state) do
    # The EPMD-less flags in rel/vm.args.eex are what make this succeed;
    # without them it fails at :nodistribution.
    {:ok, _} = :net_kernel.start([:"myapp@#{ip}", :longnames])
    Node.set_cookie(String.to_atom(System.fetch_env!("CLUSTER_COOKIE")))

    topologies = [
      mesh: [
        strategy: Cluster.Strategy.LolipopZTL,
        config: [
          node_basename: "myapp",
          hostnames: ["myapp-"],
          cli: "/usr/bin/ztlctl",
          socket: @sock
        ]
      ]
    ]

    {:ok, _} = Cluster.Supervisor.start_link([topologies, [name: MyApp.Cluster]])
    Logger.info("clustering as #{Node.self()}")
    {:noreply, state}
  end

  defp ztlctl(args) do
    System.cmd("/usr/bin/ztlctl", ["--socket=#{@sock}" | args], stderr_to_stdout: true)
  end

  defp address do
    {out, _} = ztlctl(["status", "--no-peers"])

    if out =~ "Connected" do
      Regex.run(~r/^ip:\s*(\S+)/m, out, capture: :all_but_first) |> List.first()
    end
  end

  # The daemon is talking to a controller over the network, so every step of
  # this takes as long as it takes. Raising rather than returning nil: the
  # supervisor restarts this and the reason is in the log, where carrying a
  # nil forward would fail somewhere else entirely.
  defp wait_for(check, tries \\ 60)
  defp wait_for(_check, 0), do: raise("the mesh daemon never became ready")

  defp wait_for(check, tries) do
    case check.() do
      falsy when falsy in [nil, false] ->
        Process.sleep(2_000)
        wait_for(check, tries - 1)

      value ->
        value
    end
  end

  defp url, do: System.get_env("ZTL_CONTROLLER_URL")
  defp key, do: System.get_env("ZTL_AUTH_KEY")
end
```

The complete sequence is verified with `nerves_system_rpi4` and LOLIPOP ZTL.
Tailscale and NetBird use the same Nerves facilities, but their daemon flags
and startup behavior still need validation on the target system before they
are used in a firmware release.

## Usage

Every node must run with a longname built from its mesh IPv4 address and a
shared basename. The simplest distribution setup is EPMD-less with a fixed
port, one BEAM node per host:

```sh
elixir --name myapp@100.64.0.7 --cookie "$CLUSTER_COOKIE" \
  --erl "-start_epmd false -erl_epmd_port 45892" \
  -S mix run --no-halt
```

`-erl_epmd_port` doubles as the local listen port and the port assumed for
every remote node, so all nodes must use the same one. Distribution traffic is
plaintext, but it only flows inside the WireGuard-encrypted mesh. The listener
binds all interfaces by default; to keep it mesh-only add
`-kernel inet_dist_use_interface '{100,64,0,7}'`.

### Tailscale

```elixir
config :libcluster,
  topologies: [
    tailnet: [
      strategy: Cluster.Strategy.TailscaleLocal,
      config: [
        node_basename: "myapp",
        hostnames: ["myapp-"]
      ]
    ]
  ]
```

### NetBird

```elixir
config :libcluster,
  topologies: [
    netbird: [
      strategy: Cluster.Strategy.Netbird,
      config: [
        node_basename: "myapp",
        hostnames: ["myapp-"]
      ]
    ]
  ]
```

### LOLIPOP Zero Trust Link

Run `ztld` in TUN mode (in userspace mode there is no TUN device, so the BEAM
cannot bind its distribution listener to the mesh address).

```elixir
config :libcluster,
  topologies: [
    ztl: [
      strategy: Cluster.Strategy.LolipopZTL,
      config: [
        node_basename: "myapp",
        hostnames: ["myapp-"]
      ]
    ]
  ]
```

### WireGuard

Plain WireGuard has no hostnames, so members are selected by an IPv4 `cidr`
over the peers' allowed-ips, and a peer counts as online when its latest
handshake is recent (`handshake_max_age`, default 180s). Keep
`persistent_keepalive` enabled on the mesh so idle peers stay visibly alive.

```elixir
config :libcluster,
  topologies: [
    wireguard: [
      strategy: Cluster.Strategy.WireGuard,
      config: [
        node_basename: "myapp",
        interface: "wg0",
        cidr: "10.77.0.0/24"
      ]
    ]
  ]
```

### Options

All strategies accept:

- `node_basename` (required) — node name part before `@`, shared by all
  cluster nodes. It is that part and nothing else, so an `@` or whitespace
  in it is refused: the node name it would build is one no node answers to,
  and every connect would simply fail
- `cli` — path to the mesh CLI binary (defaults: `"tailscale"` /
  `"netbird"` / `"ztlctl"` / `"wg"`)
- `polling_interval` — poll interval in ms (default `5_000`)
- `cmd_timeout` — CLI timeout in ms (default `10_000`)
- `runner` — a `LibclusterMesh.Runner` implementation, injectable for tests

`polling_interval` and `cmd_timeout` are millisecond counts handed to a BEAM
timer, which takes at most `4_294_967_295` of them (a little under 50 days);
a larger one is refused at startup rather than raising mid-poll.

Member selection per strategy:

- `hostnames` (required by TailscaleLocal / Netbird / LolipopZTL) — list of
  mesh hostname prefixes identifying cluster members; other mesh peers
  (laptops, phones, …) are ignored. Give your cluster hosts a common prefix
  (`myapp-1`, `myapp-2`, …)
- `cidr` + `interface` (required by WireGuard) — members are the peers on one
  named interface whose allowed-ips hold a `/32` inside `cidr`, which is the
  address they are dialled at; `"all"` and `"interfaces"` are reserved
  `wg show` operands and are refused. A peer whose only entry covers a range
  is advertising a route rather than a host, and is not a member.
  `handshake_max_age` (default `180` seconds) decides how fresh a handshake
  must be to count as online

`Cluster.Strategy.LolipopZTL` additionally accepts `socket` — the `ztld`
socket path passed to `ztlctl --socket`. Without it, `ztlctl` selects the
daemon socket.

A key a strategy does not support raises `ArgumentError` at startup rather
than being ignored — a `cidr:` handed to Tailscale, a `socket:` handed to
NetBird, or a mistyped `hostnames`, otherwise means no peer ever matches or
no setting ever takes effect, so the cluster never forms and nothing says
why.

## Relation to libcluster_tailscale

[libcluster_tailscale](https://github.com/moomerman/libcluster_tailscale)
(`Cluster.Strategy.Tailscale`) solves the same problem with a different
information source: it queries the **Tailscale cloud API**
(`api.tailscale.com`, authenticated with a `tskey-api-…` key), while this
library asks the **node's own daemon**. Neither replaces the other; pick by
constraint:

| | `Cluster.Strategy.Tailscale` (cloud API) | `Cluster.Strategy.TailscaleLocal` (this library) |
|---|---|---|
| API key required | Yes (issue + rotate `tskey-api-…`) | No |
| Works with Headscale / self-hosted control servers | No (cloud API only) | Yes |
| Peer online state | Not consulted (lists registered devices) | Real-time, from the daemon's netmap |
| Node without a local `tailscaled` (e.g. reached via subnet router) | Works | Not supported — the strategy polls the local daemon |
| Discovery network traffic | Outbound HTTPS to the cloud API | None (local socket only) |

Having an API key does not conflict with this library — `TailscaleLocal`
simply never needs it. The module is named `TailscaleLocal` (after
tailscaled's own "LocalAPI") precisely so both libraries can coexist in one
project without a module clash, e.g. daemon-run nodes discovered locally and
daemon-less nodes discovered via the API:

```elixir
config :libcluster,
  topologies: [
    tailnet: [strategy: Cluster.Strategy.TailscaleLocal, config: [...]],
    via_api: [strategy: Cluster.Strategy.Tailscale, config: [authkey: "tskey-api-...", ...]]
  ]
```

## How it works

Each poll runs the mesh CLI through `LibclusterMesh.Runner` (over a port, with
a timeout covering the whole call and every failure returned rather than
raised, so a missing binary or a hung daemon can never take the strategy down),
parses the peer list into `LibclusterMesh.Peer` structs, keeps the online
peers that belong to the cluster — by hostname prefix on the meshes that
have hostnames, by IPv4 CIDR on plain WireGuard, which does not — maps them
to `basename@mesh_ip` node names, and hands them to
`Cluster.Strategy.connect_nodes/4`. The next poll is
scheduled only after the current one finishes, so polls never overlap. Nodes
joining the mesh are picked up on the next poll. The strategy does not
disconnect a node merely because a later daemon response omits it or marks it
offline: a mesh control-plane interruption can produce that view while the
direct data path and Erlang connection remain healthy. The mesh removes a
lost route, and `:net_kernel` closes the unusable connection after
`net_ticktime`.

Two consequences of that design are worth knowing before you rely on the poll
interval:

- **An unreachable member delays the next poll.**
  `Cluster.Strategy.connect_nodes/4` works through the nodes to connect one at
  a time, and `:net_kernel.connect_node/1` blocks for several seconds against
  a host that does not answer. Since the next poll is only scheduled once the
  current one returns, N unreachable members push it out by N × that wait.
  Nothing is lost — discovery simply runs slower than the configured interval
  until those members answer or leave the mesh — but a mesh with many dead
  entries is a mesh whose discovery is slow.
- **The WireGuard strategy only sees one interface's peers.** It reads
  `wg show <interface> dump`, which lists the peers configured on that
  interface and nothing else. In a hub-and-spoke setup a spoke has only the
  hub as a peer, so it discovers the hub and never the other spokes; a full
  cluster needs every node configured with every other as a WireGuard peer.
  The three named meshes ask a daemon that knows the whole network, so this
  applies to `Cluster.Strategy.WireGuard` alone.

The test suite covers captured daemon response shapes, command failures,
configuration boundaries, member selection, and the complete path from each
strategy's runner through `Cluster.Strategy.connect_nodes/4`. Network-level
validation still belongs to the deployment environment because loopback
cannot reproduce latency, packet loss, or asymmetric reachability.

## Development

```sh
mix test          # unit tests and doctests
mix check         # format check, warnings-as-errors compile, tests, credo, dialyzer
```

## License

[MIT](LICENSE)