Packages
spark
2.7.1
2.7.2
2.7.1
2.7.0
2.6.1
2.6.0
2.5.0
2.4.1
2.4.0
2.3.14
2.3.13
2.3.12
2.3.11
2.3.10
2.3.9
2.3.8
2.3.7
2.3.6
2.3.5
2.3.4
2.3.3
2.3.2
2.3.1
2.3.0
2.2.69
2.2.68
2.2.67
2.2.66
2.2.65
2.2.64
2.2.63
2.2.62
2.2.61
2.2.60
2.2.59
2.2.58
2.2.57
2.2.56
2.2.55
2.2.54
2.2.53
2.2.52
2.2.51
2.2.50
2.2.49
2.2.48
2.2.47
2.2.46
2.2.45
2.2.44
2.2.43
2.2.42
2.2.41
2.2.40
2.2.39
2.2.38
2.2.37
2.2.36
2.2.35
2.2.34
2.2.33
2.2.32
2.2.31
2.2.30
2.2.29
2.2.28
2.2.27
2.2.26
2.2.25
2.2.24
2.2.23
2.2.22
2.2.21
2.2.20
2.2.19
2.2.18
2.2.17
2.2.16
2.2.15
2.2.14
2.2.13
2.2.12
2.2.11
2.2.10
2.2.9
2.2.8
2.2.7
2.2.6
2.2.5
2.2.4
2.2.3
2.2.2
2.2.1
2.2.0
2.1.24
2.1.23
2.1.22
2.1.21
2.1.20
2.1.19
2.1.18
2.1.17
2.1.16
2.1.15
2.1.14
2.1.13
2.1.12
2.1.11
2.1.10
2.1.9
2.1.8
2.1.7
2.1.6
2.1.5
2.1.4
2.1.3
2.1.2
2.1.1
2.1.0
2.0.1
2.0.0
1.1.55
1.1.54
1.1.53
1.1.52
1.1.51
1.1.50
1.1.49
1.1.48
1.1.47
1.1.46
1.1.45
1.1.44
1.1.43
1.1.42
1.1.41
1.1.40
1.1.39
1.1.38
1.1.37
1.1.36
1.1.35
1.1.34
1.1.32
1.1.31
1.1.30
1.1.29
1.1.28
1.1.27
1.1.26
1.1.25
1.1.24
1.1.22
1.1.21
1.1.20
1.1.19
1.1.18
1.1.17
1.1.16
1.1.15
1.1.14
retired
1.1.13
1.1.12
1.1.11
1.1.10
1.1.9
1.1.8
1.1.7
1.1.6
1.1.5
1.1.4
1.1.3
1.1.2
1.1.1
1.1.0
1.0.9
1.0.8
1.0.7
1.0.6
1.0.5
1.0.4
1.0.3
1.0.2
1.0.1
1.0.0
0.4.12
0.4.11
0.4.10
0.4.9
0.4.8
0.4.7
0.4.6
0.4.5
0.4.4
0.4.3
0.4.2
0.4.1
0.3.12
0.3.11
0.3.10
0.3.9
0.3.8
0.3.7
0.3.6
0.3.5
0.3.4
0.3.3
0.3.2
0.3.1
0.3.0
0.2.18
0.2.17
0.2.16
0.2.15
0.2.14
0.2.13
0.2.12
0.2.11
0.2.10
0.2.9
0.2.8
0.2.7
0.2.6
0.2.5
0.2.4
0.2.3
0.2.2
0.2.1
0.2.0
0.1.29
0.1.28
0.1.27
0.1.26
0.1.25
0.1.24
0.1.23
0.1.22
0.1.21
0.1.20
0.1.19
0.1.18
0.1.17
0.1.16
0.1.15
0.1.14
0.1.13
0.1.12
0.1.11
0.1.10
0.1.9
0.1.8
0.1.7
0.1.6
0.1.5
0.1.4
0.1.3
0.1.2
0.1.1
0.1.0
Generic tooling for building DSLs
Current section
Files
Jump to
Current section
Files
documentation/how_to/test-spark-verifiers.md
<!--
SPDX-FileCopyrightText: 2025 spark contributors <https://github.com/ash-project/spark/graphs.contributors>
SPDX-License-Identifier: MIT
-->
# Testing Spark Verifiers
Verifier errors raised inside Spark's `@after_verify` hook are caught by the
framework and emitted as stderr warnings instead of propagated as exceptions.
This keeps compilation flowing when a DSL is invalid, but it means the
natural ExUnit pattern — `assert_raise/2` — does not work.
`Spark.Test` provides ExUnit helpers that turn the captured errors and
warnings back into structured data your tests can pattern-match on.
## Quick reference
| Intent | Errors | Warnings |
|---|---|---|
| Collect everything as a list | `dsl_errors/1` | `dsl_warnings/1` |
| Assert at least one matches a pattern | `assert_dsl_error/2` (or `/1` for any) | `assert_dsl_warning/2` (or `/1` for any) |
| Assert nothing was produced | `refute_dsl_errors/1` | `refute_dsl_warnings/1` |
`import Spark.Test` makes all six available.
## Defining modules inside the helpers
This rule catches everyone. Inside the macro's anonymous-function body, a
bare alias like `BadPerson` resolves against the surrounding test module's
scope. Outside — where your `assert` patterns live — the same alias
resolves at the top level. They become different atoms, and your patterns
will never match.
Force top-level resolution with the explicit `Elixir.X` form:
```elixir
errors =
dsl_errors do
defmodule Elixir.BadPerson do # <-- note the Elixir. prefix
...
end
end
assert [{BadPerson, _}] = errors # <-- bare alias resolves to Elixir.BadPerson, matches
```
Always use the `Elixir.X` form for any module you intend to assert on.
## Testing a verifier that produces errors
Anything that returns `{:error, %Spark.Error.DslError{}}` from
`Spark.Dsl.Verifier.verify/1` — your own verifiers and Spark's built-ins
(`VerifyEntityUniqueness`, `VerifySectionSingletonEntities`).
### Match a specific error
`assert_dsl_error/2` is the most common shape. It takes a pattern, runs the
do-block, and returns the first matching error so you can do follow-up
assertions on it:
```elixir
defmodule MyApp.PersonValidatorTest do
use ExUnit.Case, async: true
import Spark.Test
test "rejects an invalid email" do
err =
assert_dsl_error %Spark.Error.DslError{path: [:fields, :email]} do
defmodule Elixir.BadPerson do
use MyApp.PersonValidator
fields do
field :email, :string do
check &String.contains?(&1, "@")
end
end
end
end
assert err.message =~ "invalid email"
end
end
```
The pattern is matched via `match?/2`, so any valid Elixir pattern works.
On no match the test fails with a message that includes the pattern source
and the inspected list of actual errors.
When the pattern doesn't matter, use the `/1` form:
```elixir
err =
assert_dsl_error do
defmodule Elixir.BadPerson do
# ...
end
end
```
### Assert clean compilation
For happy-path tests where no error should be produced:
```elixir
test "valid configuration is accepted" do
refute_dsl_errors do
defmodule Elixir.GoodPerson do
use MyApp.PersonValidator
fields do
field :email, :string
end
end
end
end
```
Returns `:ok` on success. On failure the message lists the offending
modules and their errors.
### Work with the full collected list
Use `dsl_errors/1` directly when you need to compare error counts, assert
on multiple distinct errors at once, or otherwise inspect the full result.
The shape is `[{module, [%Spark.Error.DslError{}]}, ...]` in definition
order:
```elixir
errors =
dsl_errors do
defmodule Elixir.HasTwoIssues do
# ...
end
end
assert [{HasTwoIssues, errors_list}] = errors
assert length(errors_list) == 2
```
## Testing a verifier that produces warnings
The same three helpers exist (`dsl_warnings/1`, `assert_dsl_warning/2,/1`,
`refute_dsl_warnings/1`) with one shape difference:
**Warnings are normalized to `{message, location | nil}` tuples**, not
`Spark.Error.DslError` structs. A bare-string warning `{:warn, "msg"}`
becomes `{"msg", nil}`; a tuple-form warning `{:warn, {"msg", anno}}`
preserves the location. Source annotations require `debug_info` to be
enabled for tests — see [Asserting on source
annotations](#asserting-on-source-annotations) below.
```elixir
test "deprecated option emits a warning" do
{message, _location} =
assert_dsl_warning {_, _} do
defmodule Elixir.UsesDeprecatedOption do
use MyApp.Validator
fields do
field :legacy_option, :string
end
end
end
assert message =~ "deprecated"
end
```
For literal-message matching put the string in the pattern:
```elixir
{_, _} =
assert_dsl_warning {"legacy_option is deprecated", _} do
# ...
end
```
`refute_dsl_warnings/1` and `dsl_warnings/1` work identically to their
errors counterparts, just with the warning payload shape.
## What the helpers don't capture
- **`Spark.Warning.warn/3` called directly** — for example by
`Spark.Options` for schema-level deprecation warnings, or by
`Spark.Dsl.Extension.__after_verify__/1` for `__spark_metadata__`
deprecation. These bypass the verifier-callback path. Use
`capture_io(:stderr, ...)` to assert on them.
- **Transformer warnings** — `{:warn, dsl_state, warnings}` returns from
`Spark.Dsl.Transformer.transform/1` flow through a separate code path
and are not collected.
## Asserting on source annotations
Spark only captures `:erl_anno.anno()` when `debug_info` is enabled. To
enable it for runtime modules compiled inside the helpers, add the
following to your `mix.exs`:
```elixir
test_elixirc_options: [debug_info: true]
```
Without it, `location` in `{message, location}` payloads is `nil` even when
the verifier passes the anno correctly.
## Async safety
The helpers are safe in `async: true` ExUnit tests. The collection
mechanism uses per-process state (`Process.put/2`) and point-to-point
`send/2`; nothing crosses process boundaries.
One caveat: any `defmodule` defined inside a process you spawn yourself
won't be captured, because the spawned process has its own dictionary.
Either keep `defmodule` calls in the same process as the helper, or set
the collector flag explicitly inside the spawned process:
```elixir
parent = self()
Task.async(fn ->
Process.put({Spark.Dsl, :test_collector}, parent)
defmodule Elixir.SomeModule do
# ...
end
end)
|> Task.await()
```
## See also
- `Spark.Test` — the helper module
- `Spark.Dsl.Verifier` — the verifier callback contract
- `Spark.Error.DslError` — the error struct returned by the error helpers