Current section
Files
Jump to
Current section
Files
ash_metrics
README.md
README.md
# AshMetrics
[](https://hex.pm/packages/ash_metrics) [](https://hexdocs.pm/ash_metrics)
**Experimental.** The DSL, the configuration keys and the public API may
change between minor releases until 1.0. Every such change is listed in the
[changelog](CHANGELOG.md).
## What it is
AshMetrics is an Ash resource extension that adds a `metrics do` block for
declarative *business* metrics: counts and rates of business events (emails
sent, invoices captured, syncs completed), the current depth of state-machine
backlogs (how many records are `pending` right now), and latency
distributions. Declarations sit next to the action they describe, are validated
at compile time, carry a uniform set of tags without every call site repeating
them, and allowlist tag keys, closing any tag to an enumerated set of values.
Those declarations compile into a list of `Telemetry.Metrics` structs, which the
host application's existing reporter ships to whatever backend it already uses —
OTLP, StatsD, Prometheus, AppSignal. Counter emission is a synchronous
`:telemetry.execute/3`; periodic polling is only needed for gauges.
## Primitives
- **`counter`** — "How many events? How fast?" Emitted where the fact is
written: by hand with `increment/3`, or by an action change.
- **`gauge`** — "How many right now? How old is the oldest?" Filled by a
package-managed periodic poll over the resource.
- **`distribution`** — "What's the spread?" Observed by hand with `observe/4`,
or by `observe_elapsed/2` on an action.
## Installation
One command does the whole installation:
```sh
mix igniter.install ash_metrics
```
It adds the dependency, writes `prefix` and `otp_app` to `config/config.exs`
and `poll: false` to `config/test.exs`, imports the package's formatter
configuration, appends `++ AshMetrics.metrics()` to the `metrics/0` of the
module that imports `Telemetry.Metrics` — `MyAppWeb.Telemetry` in a generated
Phoenix application — and adds `AshMetrics.Supervisor` to your application's
children after the repositories and Oban. It never overwrites a value you have
already chosen, prints the optional configuration keys with their defaults,
prints how to select the Oban poller or the OpenTelemetry backend when
`ash_oban` or `otel_telemetry_metrics` is among your dependencies, and prints a
reporter snippet to add by hand when it finds no telemetry module. See
`mix ash_metrics.install`.
To install by hand, add the dependency:
```elixir
{:ash_metrics, "~> 0.6"}
```
Then write the configuration block below and wire the metrics into your
reporter and supervision tree as
[Wiring into your reporter](#wiring-into-your-reporter) describes.
Configuration:
```elixir
config :ash_metrics,
prefix: "myapp", # REQUIRED
otp_app: :my_app, # REQUIRED
name_builder: AshMetrics.NameBuilder.Default,
tag_extractor: AshMetrics.TagExtractor.Default,
backend: AshMetrics.Backend.Noop, # Backend.Otel for OpenTelemetry
poller: AshMetrics.Poller.GenServer,
poll: true, # false starts no poller
tenant_source: MyApp.Tenants # per-tenant gauges only
# Only when the Oban poller is chosen; see "Polling with Oban".
config :ash_metrics, AshMetrics.Poller.AshOban,
queue: :default,
max_attempts: 1
```
`prefix` must be set in compile-time configuration — `config/config.exs`, not
`config/runtime.exs` alone — and is never derived from `otp_app`. A
compile-time verifier rejects a resource that declares metrics unless it is
configured; the metric names themselves are built when `AshMetrics.metrics/0`
runs.
`otp_app` is the application whose Ash domains `AshMetrics.metrics/0` searches
to find the resources that declare metrics.
## Usage
Declare metrics on the resource:
```elixir
defmodule MyApp.Mailings.TemplatedDelivery do
use Ash.Resource,
domain: MyApp.Mailings,
extensions: [AshMetrics]
metrics do
# Optional; defaults to the resource short name.
name :templated_delivery
# → myapp.mailings.templated_delivery.delivery.count
# tags: provider, template, status, tenant
counter :delivery,
tags: [:provider, :template, status: [:queued, :sent, :bounced, :delivered, :error]],
description: "Templated deliveries by status"
# → myapp.mailings.templated_delivery.backlog.gauge
# tags: status, provider
gauge :backlog,
filter: expr(status in [:pending, :processing]),
group_by: [:status, :provider],
period: :timer.minutes(1),
description: "Deliveries waiting to be sent"
# → myapp.mailings.templated_delivery.send_latency.duration
# tags: provider, tenant
distribution :send_latency,
unit: {:native, :millisecond},
buckets: [10, 50, 100, 250, 500, 1_000, 5_000],
tags: [provider: [:ses, :smtp]]
end
end
```
A tag entry written `key: [value, ...]` is a closed tag: every emission must
carry it, with one of the listed values, and the whole enumeration lives in one
metric name rather than one name per value. An entry written `key` is open, and
a call site may pass any value or none at all. A distribution's value is
measured by the call site, not by the package.
An entry written `key: [path: [...]]` says where in the written record the
action changes below read the tag's value: its first segment names an
attribute, a calculation or an aggregate of the resource, and each further
segment an attribute of the embedded resource the previous one holds; add
`values:` to close it as well. The key is the tag's name, not an attribute
name, and `nil` at any segment of the path leaves the tag off the emission.
```elixir
counter :placement,
tags: [
state: [path: [:location, :state]],
shipping_state: [path: [:location, :shipping_address, :state], values: [:tx, :ca]]
]
```
The declarations themselves are checked while the resource compiles: metric
names must be unique, tag keys must be unique and must not collide with the
keys the tag extractor adds, a closed tag needs at least one value and no
duplicates, a tag the action changes read must reach a single value, with or
without a `path:`, buckets must be strictly ascending positive numbers, a gauge
must group by attributes of the resource, none of them twice, its strategy must
be a loadable module defining `compute/3`, and the gauge must pass its
strategy's `verify/2`, when the strategy implements one.
Spark reports a failed check as a compiler warning pointing at the offending
declaration, so compile with `mix compile --warnings-as-errors` in CI if a bad
declaration should fail the build.
See the [DSL reference](documentation/dsls/DSL-AshMetrics.md) for every option.
Emit them where the outcome becomes known: an action returning `{:ok, _}` does
not mean the email was delivered, so nothing is emitted from the action
lifecycle unless the action says so.
```elixir
AshMetrics.increment(MyApp.Mailings.TemplatedDelivery, :delivery,
tags: %{status: :sent, provider: "ses", template: "welcome_v2"},
metadata: changeset.context
)
AshMetrics.observe(MyApp.Mailings.TemplatedDelivery, :send_latency, 142,
tags: %{provider: :ses},
metadata: changeset.context
)
```
`metadata:` is the bridge to the `AshMetrics.TagExtractor`. Pass anything shaped
like Ash event metadata — a changeset's context will do — and the extractor
pulls the tags that belong on every emission. The default pulls the tenant, and
refuses to stringify a struct into a tag value.
A missing closed tag, a value that tag does not declare, or a tag key that was
not declared at all raises rather than emitting.
### Emitting from actions
When the fact is written by an Ash action, that action can emit it:
```elixir
update :update_status do
accept [:status, :delivered_at]
require_atomic? false
change AshMetrics.increment_on_change(:delivery, :status)
change AshMetrics.observe_elapsed(:delivery_time,
from: :inserted_at,
to: :delivered_at
),
where: [attribute_equals(:status, :delivered)]
end
```
`increment_on_change/2` counts one `delivery` whenever the action leaves
`status` holding a value the record did not have before, tagged with that
value and with every other declared tag of the counter that names an
attribute, a calculation or an aggregate of the resource, or declares a
`path:`. `observe_elapsed/2` records the time between two timestamps of the
written record into `delivery_time`, in that distribution's unit; `where:`
narrows it to the one transition that means delivered. Both take their
extractor metadata from the changeset, emit nothing when the action fails,
bar one case inside a surrounding transaction, and leave its result alone,
bar a calculation or aggregate that fails to load there.
See `AshMetrics.Changes.IncrementOnChange` and
`AshMetrics.Changes.ObserveElapsed`.
`increment_on_write/2` counts every write of the attribute instead of every
change of it, so a create counts and an update writing the same value again
counts again:
```elixir
update :record_attempt do
accept [:status]
change AshMetrics.increment_on_write(:delivery, :status)
end
```
Four caveats. An open tag that names an attribute, a calculation or an
aggregate, or declares a `path:`, is read from the written record and carries
whatever it holds, one timeseries per distinct value; name a bounded field or
close the tag with `values:`. A tag that names a calculation or an aggregate
costs one load for every record the change emits for, without authorization;
see `AshMetrics.Dsl.Tags`. A value outside a closed tag's declared set is not
counted and never fails the action: a status the counter does not enumerate is
skipped silently, and any other rejected emission is logged at error level,
except the failed load inside an open transaction described above.
`increment_on_change/2` refuses to run atomically: the action needs
`require_atomic? false`, and `Ash.bulk_update/4` needs `:stream` among its
strategies, or it emits nothing and returns
`Ash.Error.Invalid.NoMatchingBulkStrategy`. `increment_on_write/2` and
`observe_elapsed/2` run atomically unless the `change` carries a `where:`
whose condition reads an attribute, as the one above does.
`Ash.bulk_create/4` needs nothing extra.
### Gauges
A gauge is the one primitive you never emit. AshMetrics polls it every `period`
and emits one value per group, so the declaration above publishes
`myapp.mailings.templated_delivery.backlog.gauge` with a `status` and a
`provider` tag and one timeseries per combination that exists.
A poll costs queries: the default `:count` strategy runs `1 + groups` of them
per period, again per tenant for a resource that is polled per tenant, and runs
them with `authorize?: false`. See `AshMetrics.Gauge.Strategy.Count`. On
`AshPostgres.DataLayer`, `strategy: AshMetrics.Gauge.Strategy.Postgres.Count`
returns the same counts from one `GROUP BY` query per period; it needs
`ash_postgres`, an optional dependency, and selecting it without that
dependency is a compile error. When neither will do, declare
`strategy: MyApp.Stats.Backlog` — any module implementing
`AshMetrics.Gauge.Strategy`, or `{module, options}` for one that takes
options — and compute the number however you like.
`AshMetrics.Gauge.Strategy.OldestAge` reports a different number: the age, in
whole seconds, of the oldest matching row in each group, measured from
`inserted_at` or from the timestamp named by its `attribute:` option, at the
cost of `:count`. `AshMetrics.Gauge.Strategy.Postgres.OldestAge` takes the same
option and returns the same ages from one query, and needs `ash_postgres` as
`AshMetrics.Gauge.Strategy.Postgres.Count` does.
```elixir
gauge :backlog_age,
filter: expr(status == :pending),
group_by: [:provider],
strategy: {AshMetrics.Gauge.Strategy.OldestAge, attribute: :queued_at}
```
Sub-minute periods are usually wasted resolution: most collectors flush on a
ten second interval anyway, and every poll costs the queries above.
A group that disappears is emitted once as a zero, so a drained backlog does
not keep reporting its last value; see `AshMetrics.Gauge.Runner`. A backend
that takes the poll's result, such as `AshMetrics.Backend.Otel`, zeroes it
itself.
### Polling with Oban
The default `AshMetrics.Poller.GenServer` runs a timer on every node.
`AshMetrics.Poller.AshOban` polls from Oban's cron instead: one job per period
for the whole cluster, a failed poll as a failed job with its error in Oban
Web, and one worker and queue entry per gauge.
```elixir
# In config/config.exs: the choice is read while resources compile.
config :ash_metrics, poller: AshMetrics.Poller.AshOban
config :ash_metrics, AshMetrics.Poller.AshOban,
queue: :default,
max_attempts: 1
```
Like `prefix`, the poller has to be compile-time configuration: the schedules
are generated while the resource compiles, and a poller that differs between
compile time and runtime leaves the gauges with no poller at all.
The poller can also be chosen per resource, for example one expensive backlog
on the queue and the remaining gauges on the timer:
```elixir
metrics do
poller AshMetrics.Poller.AshOban
gauge :backlog, filter: expr(status == :pending), period: :timer.minutes(5)
end
```
Such a resource must use the `AshOban` extension, must declare gauge periods
cron can express exactly, and needs its queue and its `AshOban.config/2`
crontab in the host application's Oban configuration; the first two are compile
errors naming the gauge. Zeroing a drained group is weaker than with the timer,
since an Oban job has no state between runs. See `AshMetrics.Poller.AshOban`
for all of it, and for the private action and schedule it generates per gauge.
With `AshMetrics.Backend.Otel`, the poll's result is exported from the node
that ran the job, one series per gauge for the cluster, and drained groups are
zeroed by the backend, not by that memory; see
[OpenTelemetry](#opentelemetry).
### Multitenancy
A gauge is tagged with `tenant` under either of Ash's multitenancy strategies.
A resource Ash will not read without a tenant — `:context`, or `:attribute`
without `global? true` — is polled once per tenant of the configured
`tenant_source`, a module implementing `AshMetrics.TenantSource`, and a
compile-time verifier rejects such a resource that declares a gauge while
`tenant_source` is unset. `:attribute` with `global? true` needs no
configuration.
Note the multiplier: a gauge polled per tenant costs its queries once per
tenant per period. See `AshMetrics.Gauge.Runner` for how each strategy is
polled.
## Wiring into your reporter
`AshMetrics.metrics/0` returns the `Telemetry.Metrics` definitions of every
resource that declares metrics. Splice it into whatever reporter you already
run:
```elixir
defmodule MyApp.Telemetry do
use Supervisor
def start_link(arg), do: Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
@impl true
def init(_arg) do
children = [
{Telemetry.Metrics.ConsoleReporter, metrics: my_own_metrics() ++ AshMetrics.metrics()},
AshMetrics.Supervisor
]
Supervisor.init(children, strategy: :one_for_one)
end
end
```
The same list works for a Prometheus reporter:
```elixir
{TelemetryMetricsPrometheus, metrics: AshMetrics.metrics()}
```
`:telemetry` detaches a handler that raises, so a bug in the reporter takes
every metric that handler served silent, with nothing raised or logged by
AshMetrics. Monitor the
[`[:telemetry, :handler, :failure]`](https://hexdocs.pm/telemetry/telemetry.html#attach/4)
event, or the reporter's own logs, to see it happen.
`AshMetrics.Supervisor` starts whatever the configured backend needs, followed
by every poller in use. Put it after your repository, since a gauge is answered
by a query. `AshMetrics.child_specs/1` returns the same children as a list, for
a tree that would rather splice them in than add a supervisor. With the default
`AshMetrics.Backend.Noop` the backend adds nothing. Pass an explicit resource
list to `AshMetrics.metrics_for/1` if domain discovery is not what you want.
With the default poller every node polls, so in a cluster each gauge is
computed and emitted once per node per period; see
`AshMetrics.Poller.GenServer`, [polling with Oban](#polling-with-oban), and
`AshMetrics.Poller` for implementing anything else.
### OpenTelemetry
An application exporting through OpenTelemetry with the
`otel_telemetry_metrics` bridge selects the backend that adapts the definitions
for it:
```elixir
config :ash_metrics, backend: AshMetrics.Backend.Otel
```
The application keeps its own `OtelTelemetryMetrics` instance and splices
`AshMetrics.metrics()` into the list it hands it:
```elixir
{OtelTelemetryMetrics, metrics: my_own_metrics() ++ AshMetrics.metrics()}
```
Counters and distributions go through the bridge; a declared `buckets` list is
carried on as the histogram's bucket boundaries. Gauges do not: the backend
exports each one as an OpenTelemetry observable gauge serving the values the
configured poller reports to it, so the pollers must run — with `poll: false`
no gauge is exported. With `AshMetrics.Poller.AshOban` that is one
query and one series per gauge for the whole cluster; with the default timer
poller, one of each per node. A series carries the exporting node's resource
attributes, so aggregate over `host` when querying, and when the poll moves to
another node both export it for up to one period. See
`AshMetrics.Backend.Otel`.
## Testing
```elixir
defmodule MyApp.MailingsTest do
use ExUnit.Case, async: true
use AshMetrics.Test
test "a delivery emits a sent counter" do
MyApp.Mailings.deliver!(...)
assert_metric_emitted "myapp.mailings.templated_delivery.delivery",
tags: %{status: :sent, provider: "ses"}
refute_metric_emitted "myapp.mailings.templated_delivery.delivery",
tags: %{status: :bounced}
end
end
```
`use AshMetrics.Test` attaches a handler for the duration of each test and
imports the assertions. Name the metric as the declaration produces it, without
the aggregation suffix a reporter adds. A test receives only the emissions of
processes it owns, its own and those of the `Task`s it starts, so such modules
can be `async: true`; see `AshMetrics.Test` for the ownership rule, `allow/1`
and `shared: true`.
The installer writes `config :ash_metrics, poll: false` to `config/test.exs`,
which keeps gauges from being polled while tests run. To poll them in a test,
set it to `true` there or pass `poll: true` to `AshMetrics.Supervisor`; the
poller emits from its own process, so assert on it after `allow/1` with the
poller's pid or with `shared: true`, or call `AshMetrics.Gauge.Runner.emit/3`
from the test process instead.
## Development
`mix test` runs the whole suite except the Postgres integration tests, and
needs no database and no container. The tests that do need one are tagged
`:postgres`, excluded by default, and run against the container this
repository's `docker-compose.yml` defines:
```sh
docker compose up -d
mix test.integration
docker compose stop
```
`mix test.integration` creates the database, migrates it and runs everything
with `--include postgres`. The container is named `ash_metrics-postgres-1` and
publishes Postgres on `${ASH_METRICS_PG_PORT:-54329}`; set that variable if
54329 is taken. Only ever drive it through `docker compose` from the repository
root, so that no container outside this project is touched.
The rest of the checks:
```sh
mix format --check-formatted
mix credo --strict
mix dialyzer
mix docs
```
`mix docs` regenerates the DSL cheat sheet in `documentation/dsls`, which is
checked in; commit it with whatever DSL change produced it.
## Non-goals
- Not a new emit/aggregate/export pipeline. AshMetrics produces
`Telemetry.Metrics` structs and lets the existing reporter ecosystem ship them.
- Not an APM or tracing tool. `ash_appsignal` and `opentelemetry_ash` cover that.
- Not a replacement for `Oban.Telemetry`, which already emits job `queue_time`
and `duration`.
## License
MIT. See the `LICENSE` file in the repository root.