sagents

# Agent Evals - How-To Guide

How we measure non-deterministic agent behaviour in Houston: run a defined
operation against a Sagents agent, assert on **what it answered** *and* **the
path it took**, and (in later phases) repeat it N times to get a success rate.

The rail lives under the `Connect.AI.Evals` namespace and pairs three things:

| Layer | Owns | Where |
|---|---|---|
| [`tribunal`](https://hex.pm/packages/tribunal) | Assertion macros, dataset format + `mix tribunal.eval`, reporters, LLM-as-judge | dependency |
| [`LangChain.Trajectory`](https://hexdocs.pm/langchain/LangChain.Trajectory.html) | Trajectory capture + tool-call matching | dependency |
| **Host harness** | Driving the real agent path, fixtures, HITL interrupt handling | `test/support/evals/` |

> **Evals are test-env only.** The harness modules (`Provider`, `Fixtures`,
> `Interrupts`, …) live in `test/support/evals/` and compile only under
> `MIX_ENV=test` (see `elixirc_paths/1` in [`mix.exs`](../../mix.exs)). They are
> **not** part of the dev/prod build. The eval *cases* are ordinary ExUnit files
> co-located under `lib/connect/ai/evals/` (Houston discovers tests via
> `test_paths: ["lib", "checks"]`).

## Running evals

Eval cases that hit a live model are tagged `@moduletag :eval`, which is
**excluded from the default `mix test` run** (configured in
[`lib/test_helper.exs`](../../lib/test_helper.exs)). Run them on demand, in an
environment with the agent model configured:

```bash
# All eval cases
mix test --only eval

# One file
mix test --only eval lib/connect/ai/evals/deployment_agent_eval_test.exs
```

The deterministic harness unit tests (provider orchestration, trajectory
capture, interrupt classification) are **not** tagged `:eval` and run in the
normal suite — they need no LLM.

## Production fidelity (the most important rule)

Evals exist to measure what we ship. `Connect.AI.Evals.Provider.run/1` reaches
the agent the **same way production does — through `Sagents.AgentServer`**, built
from the real `*Factory.create_agent/2`. It does **not** call the lower-level
`Sagents.Agent.execute/3` or build a bare `LLMChain`. That means the
message-preprocessor, the middleware lifecycle, HITL interrupts, persistence and
the PubSub event path all run exactly as they do in the app. An eval that
shortcuts that wiring measures an agent we never run.

(Agents that production runs via one-shot `Sagents.Extract` should be modeled
with `Sagents.Extract` in their own provider — match production, don't force
everything through `AgentServer`.)

## The four kinds of eval — and what each is best for

You will usually combine the first two in a single case.

### 1. Deterministic string assertions — *"did it answer?"*

Tribunal's `assert_contains`, `assert_regex`, `assert_json`, `assert_equals`,
etc. (imported via `use Tribunal.EvalCase`). No LLM call, instant, exact.

- **Best for:** factual content the answer must contain ("names Edge", "reports
  a failure", returns valid JSON, stays under a token budget).
- **Reach for this first.** It's the cheapest signal and never flakes.

### 2. Trajectory assertions — *"did it take the right path?"*

Assert which tools were called, in what order, with what arguments — using
`LangChain.Trajectory` (see the [Trajectory pattern](#the-trajectory-pattern)
below).

- **Best for:** distinguishing *right answer, right path* from *right answer,
  wrong path* (lucky guess, skipped a required tool, called a forbidden one).
- Also cheap and deterministic — it inspects captured calls, no extra LLM call.

### 3. LLM-as-judge — *"is it good?"*

Tribunal's subjective assertions (`assert_faithful`, `assert_correctness`,
`assert_relevant`, custom `Tribunal.Judge` rubrics). Requires `req_llm` (present:
`~> 1.10` in [`mix.exs`](../../mix.exs)) and a configured judge model.

- **Best for:** qualities you can't pin down deterministically (tone, faithfulness
  to context, "did it satisfy the request").
- **Use sparingly.** Judges are themselves non-deterministic and add noise. Prefer
  deterministic/trajectory conditions; if you must judge, run the judge K times
  and take a majority vote, and record the judge model with the run.

### 4. HITL interrupt tests — *"does the pause/approve/answer flow work?"*

Drive the agent through human-in-the-loop pauses using
`Connect.AI.Evals.Interrupts` (see [HITL interrupts](#hitl-interrupts) below).

- **Best for:** cases whose *subject* is the interrupt — a gated tool that must
  pause for approve/reject, or an `ask_user` question with specific options/flags.
- For incidental interrupts that aren't the point of the case, let the Provider
  auto-drive past them (it already does — `on_review: :approve_all`).

### 5. N-iteration success rate — *"how often does it work?"*

`Connect.AI.Evals.run/2` runs a case K times and aggregates to a success rate +
mean tokens/duration (see [the N-iteration loop](#the-n-iteration-loop) below).
That's the tool for measuring flakiness and confirming a prompt change moved the
rate the right way (70% → 90%). A single `Provider.run/1` is still fine when you
just need one pass through the agent.

## Writing a case

A complete example lives in
[`deployment_agent_eval_test.exs`](../../lib/connect/ai/evals/deployment_agent_eval_test.exs).
The skeleton:

```elixir
defmodule Connect.AI.Evals.MyAgentEvalTest do
  # async: false + DataCase so the SQL sandbox runs in shared mode — the agent's
  # tools may execute in a spawned task and must see the seeded fixtures.
  use Houston.DataCase, async: false
  use Tribunal.EvalCase

  alias Connect.AI.Evals.Provider
  alias LangChain.Trajectory

  @moduletag :eval

  # Background middleware (e.g. conversation-title) fires its own model call —
  # stub it so it can't escape to a live API. The turn under test stays live.
  setup :set_mimic_global

  setup do
    Mimic.stub(ReqLLM, :generate_text, fn _m, _ctx, _opts -> {:ok, title_response()} end)
    # ... seed DB state, build the agent from its real factory ...
    %{agent: agent}
  end

  test "answers and takes the expected path", %{agent: agent} do
    test_case =
      Tribunal.test_case(
        input: "What happened with @[Edge](package:#{id})?",
        metadata: %{agent: agent, fixture: :empty}
      )

    output = Provider.run(test_case)

    # (1) Right answer — deterministic string assertions.
    assert_contains(output, "Edge")
    assert_contains(String.downcase(output), "fail")

    # (2) Right path — the trajectory captured by the provider.
    %{trajectory: trajectory} = Provider.last_run()

    assert Trajectory.matches?(
             trajectory,
             [%{name: "get_deployments_list", arguments: nil}],
             mode: :superset
           )
  end
end
```

Pass the agent to the case as `metadata[:agent]`. The Provider accepts a
`%Sagents.Agent{}`, a factory return (`{:ok, agent}` / `{:ok, agent,
session_opts}`), a 0-arity builder, or an `{m, f, a}` tuple. Passing the
factory's full `{:ok, agent, session_opts}` return is preferred — the Provider
forwards `session_opts[:supervisor_opts]` into `AgentServer`, so things like the
`@-mention` `ChatMessagePreprocessor` wire up the production way rather than being
hand-applied.

## The Trajectory pattern

A trajectory is the flat, ordered list of tool calls the agent made, plus
aggregated token usage. The Provider captures it for you — **never reinvent
capture or matching**, lean on `LangChain.Trajectory`.

### Getting the trajectory

```elixir
output = Provider.run(test_case)
%{trajectory: traj, token_usage: usage, interrupt_log: log} = Provider.last_run()
```

`last_run/0` reads from the process dictionary, so call it **in the same process**
that called `run/1`. `interrupt_log` records any HITL decisions the Provider
auto-made (see below).

### Asserting on it

`LangChain.Trajectory.matches?/3` is the core matcher. `expected` is a list of
`%{name: ..., arguments: ...}` maps (`arguments: nil` = "any args"):

```elixir
# Was a tool called at all? (order-independent containment)
Trajectory.matches?(traj, [%{name: "get_deployments_list", arguments: nil}], mode: :superset)

# Exact full sequence, in order, with exact args
Trajectory.matches?(traj, [
  %{name: "search",       arguments: %{"query" => "weather"}},
  %{name: "get_forecast", arguments: nil}
])  # mode: :strict (default), args: :exact (default)
```

Options:

- `:mode` - several modes are defined.
  - `:strict` (same calls, same order, same count)
  - `:unordered` (same multiset)
  - `:superset` (actual contains *at least* all expected — **order
  independent**)
- `:args` — `:exact` · `:subset` (expected args are a subset of actual)

There's an ExUnit assertion wrapper with a nice diff on failure:

```elixir
use LangChain.Trajectory.Assertions

assert_trajectory traj, [%{name: "get_deployments_list", arguments: nil}], mode: :superset
refute_trajectory traj, [%{name: "delete_everything",   arguments: nil}], mode: :superset
```

And slicing helpers: `Trajectory.calls_by_name(traj, "search")`,
`Trajectory.calls_by_turn(traj)`.

### Ordering ("A before B") — `called_before?/4`

`:superset` checks containment **without** enforcing order, and `:strict`
requires the *entire* sequence and count. For the middle ground — "tool A was
called *before* tool B, ignoring everything in between" (e.g. "listed deployments
before describing one") — use `Trajectory.called_before?/4` (LangChain ≥ 0.8.13):

```elixir
# True if any get_deployments_list call precedes any get_deployment_details call.
Trajectory.called_before?(traj, "get_deployments_list", "get_deployment_details")

# Surface a missing tool as an error instead of a silent false.
Trajectory.called_before?(traj, "search", "answer", require_both: true)
```

Semantics: *any A before any B* (`min(index of A) < max(index of B)`), tolerant
of interleaving. By default a missing tool returns `false` (so
`refute_called_before` passes vacuously); `require_both: true` raises
`ArgumentError` when either tool is absent — use it when a missing tool means a
broken eval. Accepts a `Trajectory`, an `LLMChain`, or a bare tool-call list.

Companion ExUnit macros (via `use LangChain.Trajectory.Assertions`):

```elixir
assert_called_before traj, "search", "answer"
refute_called_before traj, "write_file", "read_file"
```

### Golden trajectories (Phase 5, not built)

`Trajectory.to_map/1` / `from_map/1` serialize a trajectory for storage. Once a
case passes convincingly you can record its trajectory as a "golden" and assert
later runs `matches?` it with a chosen mode — a cheap "did not deviate" check
that needs no judge. Not yet wired into the harness.

## The N-iteration loop

A single agent run is non-deterministic, so the honest measurement is a *rate*.
`Connect.AI.Evals.run/2` ([source](../../test/support/evals/runner.ex)) drives
the **same case** K times and aggregates to a `Connect.AI.Evals.RunResult` —
success rate, mean tokens, mean duration — which is what you compare across a
prompt or tool-wording change.

```elixir
import Connect.AI.Evals.Assertions   # assert_success_rate/2
alias LangChain.Trajectory

result =
  Connect.AI.Evals.run(
    Tribunal.test_case(
      input: "What happened with @[Edge](package:#{package.id})?",
      metadata: %{agent: agent, fixture: :empty}
    ),
    iterations: :release,
    pass: fn %{output: text, metadata: meta} ->
      assert_contains(text, "Edge")
      Trajectory.matches?(meta.trajectory, [%{name: "get_deployments_list", arguments: nil}], mode: :superset)
    end
  )

assert_success_rate(result, 0.8)
```

### The `:pass` predicate

Called once per iteration with `%{output: String.t(), metadata: map() | nil}`
(metadata is `Provider.last_run/0`'s output — `%{trajectory:, token_usage:, …}`).
An iteration:

- **passes** when the predicate returns truthy *and* doesn't raise;
- **fails** when it returns `false`/`nil`, **or** raises an
  `ExUnit.AssertionError` — so Tribunal's `assert_*` and the `Trajectory`
  assertion macros work straight inside it (a failed assertion is a failed
  iteration, not a crash);
- is an **error** when the provider raises, the iteration times out, or the
  predicate raises a non-assertion exception. Errors are reported separately
  (`RunResult.errored`) so a broken harness/model run can't masquerade as agent
  flakiness — but they still count against the rate.

### Options

| Option | Meaning |
|---|---|
| `:pass` (required) | the predicate above |
| `:iterations` | positive integer, or `:quick` (5) / `:release` (20). Default `:quick` |
| `:provider` | `{module, function}` returning the text + exposing `last_run/0`. Default `{Provider, :run}` |
| `:max_concurrency` | iterations run at once (default 4). For DB-backed evals on the shared SQL sandbox, prefer `1` |
| `:iteration_timeout` | per-iteration kill bound (ms). Default 2× the case's `metadata[:timeout]` |

> **Low N has wide error bars.** At a true 50% rate, N=20 has a 95% CI of roughly
> ±22pp — too coarse to tell 70% from 80%. Use `:quick` while iterating, then
> `:release` (or a larger explicit N) before trusting a rate. Mean **cost** is not
> yet aggregated (needs per-model pricing — Phase 4); mean tokens is the proxy.

## HITL interrupts

`Connect.AI.Evals.Interrupts` ([source](../../test/support/evals/interrupts.ex))
drives a `Sagents.AgentServer` through human-in-the-loop pauses. Two kinds share
the `:interrupted` lifecycle but differ in payload:

- **Tool review** — a gated tool awaits approve / reject / edit.
- **Ask-user question** — the `ask_user` tool asks a question with options,
  awaiting an answer / cancel.

### Driving the AgentServer in a HITL case

A HITL case drives the `Sagents.AgentServer` **itself** — the Provider's
auto-approve/forbid policy is fixed for *incidental* interrupts (see
[below](#auto-driving-past-incidental-interrupts)) and `run/1` only returns the
final text, so it can't expose or assert on the pause. Build the agent the same
way, but start and drive it yourself.

`Connect.AI.Evals.Session` owns the start/subscribe/stop lifecycle so a case
doesn't hand-roll `AgentSupervisor.start_link_sync/1`, the `Process.unlink/1`, the
monitor-and-wait shutdown, or the easy-to-miss subscribe-snapshot drain:

```elixir
import Connect.AI.Evals.Interrupts
alias Connect.AI.Evals.{Provider, Session}
alias LangChain.{Message, Trajectory}
alias Sagents.{AgentResult, AgentServer}

# Start the agent the production way (forwarding the factory's supervisor_opts).
{agent, session_opts} = Session.resolve!(MyAgentFactory.create_agent(id, config))
sup_pid = Session.start!(agent, session_opts)
on_exit(fn -> Session.stop(sup_pid) end)

# subscribe! subscribes AND drains the idle snapshot. Do it BEFORE the triggering
# message — otherwise await_outcome/2 reads the snapshot as an immediate completion.
:ok = Session.subscribe!(agent.agent_id)
:ok = AgentServer.add_message(agent.agent_id, Message.new_user!("Delete the Edge package."))

# Drive to the pause, respond with the decision under test, require a clean finish.
outcome = await_outcome(agent.agent_id)
assert_review(outcome, tool: "delete_package")

final_state =
  agent.agent_id
  |> reject_review(outcome, tool: "delete_package")
  |> refute_interrupt()

# Assert how it handled the rejection — answer AND path (reuse the Provider's capture).
{:ok, text} = AgentResult.to_string(final_state)
assert text =~ "cancel"

refute Trajectory.matches?(
         Provider.build_trajectory(final_state, agent),
         [%{name: "delete_package", arguments: nil}],
         mode: :superset
       )
```

The agent must have a HITL-gated tool wired (a `review_configs` entry) or use
`ask_user`; a read-only tool like `get_deployments_list` won't pause. Every
waiting helper reads `{:agent, _}` events from the **current process mailbox**,
which is why `subscribe!/1` runs before the message.

### The primitive + assertions

`await_outcome/2` drives to the next terminal state and classifies it as
`{:review, data}`, `{:question, data}`, `{:completed, state}`, `{:error, reason}`,
or `{:interrupt, data}` (unmodeled kinds). Build assertions on top:

```elixir
import Connect.AI.Evals.Interrupts

outcome = await_outcome(agent_id)

# Tool review — assert a gate, then approve/reject and require a clean finish.
state = outcome |> then(&approve_review(agent_id, &1, tool: "write_file")) |> refute_interrupt()
state = outcome |> then(&reject_review(agent_id, &1, tool: "delete_file"))  |> refute_interrupt()

# Ask-user question — assert its shape, then answer one of three ways.
assert_question(outcome, response_type: :single_select, allow_other: true,
                         allow_cancel: false, options: ["postgresql", "mongodb"])
state = outcome |> then(&answer_question(agent_id, &1, "postgresql")) |> refute_interrupt()
state = outcome |> then(&answer_other(agent_id, &1, "Use DuckDB"))    |> refute_interrupt()
state = outcome |> then(&cancel_question(agent_id, &1))               |> refute_interrupt()
```

The helpers validate up front (review decisions against `allowed_decisions`;
answers against offered values, arity, and the `allow_other`/`allow_cancel`
flags) so mistakes surface as a readable failure, not an opaque `resume/2` error.

### Auto-driving past incidental interrupts

When interrupts aren't the point of the case, `run_to_completion/2` drives to a
terminal state with a per-kind policy:

```elixir
{:ok, state, log} = run_to_completion(agent_id, on_review: :approve_all)

{:ok, state, log} =
  run_to_completion(agent_id,
    on_review: :approve_all,
    on_question: fn %{options: [first | _]} -> %{type: :answer, selected: [first.value]} end)
```

Tool gates have a safe universal default (`:approve_all`); questions have **no**
generic right answer, so `on_question` defaults to `:forbid` and must be set
explicitly to drive through. The returned `log` is the audit trail of what was
auto-decided — auto-handling changes what is measured, so it's persisted with the
run. **This is exactly what `Provider.run/1` uses** (`on_review: :approve_all`),
which is why ordinary content/trajectory evals don't have to think about HITL.

## Module reference

| Module | Role |
|---|---|
| [`Connect.AI.Evals.Provider`](../../test/support/evals/provider.ex) | Drives one case through the real `AgentServer`; returns final text, stashes trajectory/tokens/interrupt-log in `last_run/0`. |
| [`Connect.AI.Evals.Session`](../../test/support/evals/session.ex) | Start/subscribe/stop lifecycle (`resolve!`, `start!`, `subscribe!`, `stop`) shared by the Provider and HITL cases. |
| [`Connect.AI.Evals.Runner`](../../test/support/evals/runner.ex) | The N-iteration loop (`run/2`); aggregates pass/fail/error to a `RunResult` (success rate, mean tokens/duration). |
| [`Connect.AI.Evals.RunResult`](../../test/support/evals/run_result.ex) | The aggregate of an N-iteration run + per-iteration `Iteration` detail. |
| [`Connect.AI.Evals.Assertions`](../../test/support/evals/assertions.ex) | `assert_success_rate/2` over a `RunResult`. |
| [`Connect.AI.Evals.Fixtures`](../../test/support/evals/fixtures.ex) / `Fixture` | Named, reproducible seeded state a case runs against (currently `:empty`). |
| [`Connect.AI.Evals.Interrupts`](../../test/support/evals/interrupts.ex) | HITL driving + assertions (`await_outcome`, `assert_review`/`assert_question`, `run_to_completion`). |
| `LangChain.Trajectory` / `.Assertions` | Trajectory capture + matching (`matches?/3`, `assert_trajectory`). |
| `Tribunal` / `Tribunal.EvalCase` | String + judge assertions, dataset runner, reporters. |

## Conventions checklist

- Tag live cases `@moduletag :eval`; keep harness unit tests untagged.
- `use Houston.DataCase, async: false` for DB-backed agent evals (shared sandbox
  so spawned tool tasks see fixtures).
- Stub background model calls (e.g. conversation-title `ReqLLM.generate_text`)
  with Mimic so only the turn under test goes live.
- Assert **answer** (string) **and** **path** (trajectory) — the combination is
  the whole point.
- Build the agent from its real factory; pass the factory's full return as
  `metadata[:agent]` so production wiring flows through.