Packages

Autonoma SDK — automate the Autonoma Environment Factory endpoint

Current section

Files

Jump to
autonoma docs validation.md
Raw

docs/validation.md

<!-- Generated by scripts/build-sdk-docs.mjs from docs/languages/elixir/validation.md. Edit the source, not this copy. -->
# Validating scenarios (Elixir)
Scenario data is often generated by an LLM, so validate it against a real database before it reaches production. `Autonoma.Check.check_scenario/2` runs the full `up` then `down` cycle through the same handler the platform hits, using your real factories, and reports exactly where it failed.
## Autonoma.Check.check_scenario(scenario, opts)
The first argument is the **scenario map** (a map with a `"create"` key). Your factory registry goes in `opts` under `:factories` - `check_scenario` calls the real factories, so they must point at a real (test) database.
```elixir
# test/autonoma/scenarios_test.exs
alias Autonoma.Check
scenario = %{
"create" => %{
"Organization" => [%{"_alias" => "org", "name" => "Test Org", "slug" => "test-org"}],
"User" => [%{"_alias" => "admin", "name" => "Admin", "email" => "admin@test.com", "organizationId" => %{"_ref" => "org"}}],
"Member" => [%{"role" => "owner", "organizationId" => %{"_ref" => "org"}, "userId" => %{"_ref" => "admin"}}]
}
}
result =
Check.check_scenario(scenario,
factories: MyAppWeb.Autonoma.Factories.all(),
scope_field: "organizationId"
)
unless result.valid do
IO.puts("Failed at phase: #{result.phase}")
for err <- result.errors, do: IO.puts("[#{err.phase}] #{err.message}")
end
```
### opts
| Key | Default | Meaning |
|-----|---------|---------|
| `:factories` | `%{}` | Your factory registry. Required for a real run. |
| `:scope_field` | `"organizationId"` | The scope column name. |
| `:auth` | a stub returning a bearer header | Override to exercise your real auth callback. |
| `:shared_secret` | `"autonoma-check-shared"` | Rarely overridden for a dry run. |
| `:signing_secret` | `"autonoma-check-signing"` | Must differ from the shared secret. |
| `:sdk` | `%{}` | Extra `sdk` metadata. |
### The result
`check_scenario` returns an `%Autonoma.Check.CheckResult{}` struct:
| Field | Meaning |
|-------|---------|
| `:valid` | `true` if the full create + teardown cycle succeeded. |
| `:phase` | `"ok"` if it passed, else `"up"` (create rejected) or `"down"` (teardown failed). |
| `:errors` | A list of `%Autonoma.Check.CheckError{phase, message, fix}`. `message` is the underlying SDK or database error; `fix` is a hint when one is known. |
| `:timing` | `%{up_ms: ..., down_ms: ...}` in milliseconds. |
## Running every scenario in an ExUnit test
The reliable way to validate locally is a real database your factories already talk to (your Ecto test Repo, backed by Postgres). Drop every scenario file into a folder and run one test over all of them.
```elixir
# test/autonoma/scenarios_test.exs
defmodule MyAppWeb.Autonoma.ScenariosTest do
use MyApp.DataCase, async: false
alias Autonoma.Check
@scenarios_dir Path.join([File.cwd!(), "priv", "autonoma", "scenarios"])
for path <- Path.wildcard(Path.join(@scenarios_dir, "*.json")) do
@path path
test "validates #{Path.basename(path)}" do
scenario = @path |> File.read!() |> Jason.decode!()
result =
Check.check_scenario(scenario,
factories: MyAppWeb.Autonoma.Factories.all(),
scope_field: "organizationId"
)
for err <- result.errors, do: IO.puts("[#{err.phase}] #{err.message}")
assert result.valid
end
end
end
```
```bash
# shell
mix test test/autonoma/scenarios_test.exs
```
`Autonoma.Check.check_all_scenarios(scenarios, opts)` takes a list of scenario maps and returns a list of results, if you prefer to load them yourself.
## Validating without the helper
`check_scenario` is just a thin driver over `Autonoma.Handler.handle/2`. If you need finer control, call the handler directly the way the test suite does: build a config, sign the body with `Autonoma.HMAC.sign_body/2`, and assert on the response.
```elixir
# test/autonoma/handler_direct_test.exs
config = %{
scope_field: "organizationId",
shared_secret: "shared-secret-a",
signing_secret: "signing-secret-b",
factories: MyAppWeb.Autonoma.Factories.all(),
auth: fn user, _ctx -> %{"headers" => %{"Authorization" => "Bearer #{user["id"]}"}} end
}
body = Jason.encode!(%{"action" => "up", "create" => scenario["create"]})
req = %{body: body, headers: %{"x-signature" => Autonoma.HMAC.sign_body(body, "shared-secret-a")}}
result = Autonoma.Handler.handle(config, req)
assert result.status == 200
```
## The fix loop
Validation is iterative, especially for LLM-generated data:
1. Run `check_scenario`.
2. If it fails, read `result.errors` (each has a `:message` and sometimes a `:fix`).
3. Edit the scenario JSON and re-run.
4. Repeat until `:valid` is `true`.
Common failures and fixes:
| Message contains | Cause | Fix |
|------------------|-------|-----|
| `missing required fields` | A field the factory's `input_fields` marks required is absent | Add the field with a value. |
| `references unknown alias(es)` | A `_ref` targets an alias no record declares | Add the target record with that `_alias`, or fix the name. |
| `no factory registered for model` | A model in `create` has no factory | Register a factory for it and add it to the map. |
| `unique constraint` / DB error | Two records share a unique value | Make each unique field distinct across records. |
| `cycle detected` | Circular `_ref` dependency between models | Break the cycle so ordering is possible. |