jido_gralkor

# jido_gralkor

Drop-in long-term memory for a [Jido](https://hex.pm/packages/jido) agent. One Hex package: the Jido plugin and ReAct tools on top of an embedded Gralkor memory adapter — Graphiti driven directly from the BEAM via [Pythonx](https://github.com/livebook-dev/pythonx), with no separate Gralkor service to deploy. Storage uses either an embedded FalkorDB child or a remote FalkorDB deployment.

You write your agent's prompt, model, and business tools. `jido_gralkor` covers session identity, recall, capture, the `memory_search` / `memory_add` ReAct tools, a small helper that pins `tool_choice` to `memory_search` on the first ReAct iteration so the agent itself authors its memory queries, a graceful-shutdown flush, a context-rotation primitive for long-running agents, **Destinations** for named graphs, **Lenses** for ingestion, and **Reflections** for consumer-invoked synthesis.

This is the canonical home for new Gralkor development: Gralkor is Jido-first. As of `3.0.0` the former `:gralkor_ex` Hex package is folded into this one, and the legacy `:gralkor` and `:gralkor_ex` packages direct consumers here. Consumers need only `{:jido_gralkor, "~> 8.0"}` for the whole memory stack.

## Install

```elixir
def deps do
  [
    {:jido_gralkor, "~> 8.0"}
  ]
end
```

Then fetch:

```bash
mix deps.get
```

The package requires Elixir `~> 1.18` and has runtime dependencies on `:jido`, `:jido_ai`, `:pythonx`, and `:jason`. On the first native-runtime boot, Pythonx materialises a managed Python 3.12 environment with `graphiti-core` and `falkordblite`; consumers do not install Python themselves, but the boot needs package-download access and a writable cache.

## Required configuration

Four things the consumer must set up.

**1. A FalkorDB backend.** Graphiti runs in-process via Pythonx and connects to FalkorDB either as an embedded `falkordblite` child or over the network. Pick one:

```bash
# Embedded — falkordblite spawns a redis-server grandchild under this dir
export GRALKOR_DATA_DIR=/var/lib/<your-app>/gralkor   # writable
export GOOGLE_API_KEY=...                             # the default LLM and embedder are Google
```

Native Graphiti supports `google:` and `openai:` models, and each of its two
roles picks its provider independently. `GRALKOR_LLM_MODEL` selects the LLM
(default `google:gemini-3.1-flash-lite`) and `GRALKOR_EMBEDDER_MODEL` selects
the embedder (default `google:gemini-embedding-2-preview`). The
cross-encoder/reranker has no spec of its own and follows the LLM role's
provider.

Mixing the two roles is supported. An OpenAI LLM with a Google embedder builds
an OpenAI LLM client, an OpenAI reranker, and a Google embedder:

```bash
export GRALKOR_LLM_MODEL=openai:gpt-4.1-mini
export GRALKOR_EMBEDDER_MODEL=google:gemini-embedding-2-preview
export OPENAI_API_KEY=...   # the llm role selected openai
export GOOGLE_API_KEY=...   # the embedder role selected google
```

Set only the credential(s) for the providers your two specs actually select: an
all-Google pair needs `GOOGLE_API_KEY` alone, an all-OpenAI pair needs
`OPENAI_API_KEY` alone. Startup raises `ArgumentError` before any inference
client is constructed when a spec names a provider outside `:openai` / `:google`
(naming both specs and the supported providers), or when the credential for a
provider a spec selects is missing or blank (naming the variable and the role,
`"llm"` or `"embedder"`). Nothing checks that an LLM and an embedder from
different providers are otherwise compatible — embedding dimensions and the like
are yours to keep consistent.

```elixir
# Remote — point at a managed FalkorDB. config/runtime.exs
config :jido_gralkor,
  falkordb: [
    host: System.fetch_env!("FALKORDB_HOST"),
    port: String.to_integer(System.fetch_env!("FALKORDB_PORT")),
    username: System.get_env("FALKORDB_USERNAME"),
    password: System.get_env("FALKORDB_PASSWORD"),
    ssl: System.get_env("FALKORDB_SSL") == "true"
  ]
```

Remote wins when both are set. `:ssl` defaults to `false`; set `true` for FalkorDB Cloud or any TLS-fronted endpoint. Misconfigured `:falkordb` (non-keyword, missing host/port, blank host, non-positive port) raises `ArgumentError` at app start.

Embedded runtimes admit one `add_episode` mutation at a time through their shared local connection while searches remain concurrent. The remote backend retains concurrent writes. To allow a long embedded query more than the 60-second default read timeout, configure the timeout in milliseconds:

```elixir
config :jido_gralkor,
  embedded_falkordb_socket_timeout_ms: 120_000
```

The setting is passed to the embedded Redis client as `socket_timeout`; it is ignored by remote FalkorDB. A zero, negative, or non-integer value raises `ArgumentError` when the embedded runtime starts.

**2. In-memory client in tests.** Swap the adapter for the in-memory twin:

```elixir
# config/test.exs
config :jido_gralkor,
  client: Gralkor.Client.InMemory,
  destination_storage: Gralkor.Destination.Storage.InMemory,
  lens_storage: Gralkor.Lens.Storage.InMemory
```

Start the legacy client twin once in `test/test_helper.exs`:

```elixir
{:ok, _} = Gralkor.Client.InMemory.start_link()
ExUnit.start()
```

Lens tests should also start a fresh storage process in setup so state is isolated:

```elixir
setup do
  start_supervised!(Gralkor.Lens.Storage.InMemory)
  start_supervised!(Gralkor.Destination.Storage.InMemory)
  :ok
end
```

When the client and storage layers use these in-memory adapters, the native supervision tree (Pythonx → GraphitiPool → CaptureBuffer) does not start and Lens or Reflection storage calls do not reach Graphiti. No FalkorDB backend is required.

**3. `Jido.Thread.Plugin` on your `use Jido` supervisor.** The plugin reads `session_id` from `agent.state[:__thread__].id`, so the thread plugin must be active:

```elixir
defmodule MyApp.Jido do
  use Jido,
    otp_app: :my_app,
    default_plugins: [Jido.Thread.Plugin, Jido.Identity.Plugin]
end
```

**4. A non-blank human name in agent state.** Before any completed or failed turn is captured, populate `agent.state[:user_name]` with the current human's name (for example, from the request's tool context in `on_before_cmd/2`). The plugin deliberately has no generic `"User"` fallback: a missing or blank value raises `ArgumentError` before capture.

`:jido_gralkor` auto-supervises its shared storage runtime (Python → GraphitiPool → CaptureBuffer) when a FalkorDB backend is configured — no separate `Gralkor.Server` to wire into your supervision tree, and no readiness gate to add. Each `JidoGralkor.Plugin` also contributes one linked `JidoGralkor.Runtime` child beneath its consuming `Jido.AgentServer`; that child owns the agent's domain configuration and admitted Reflection work. Graceful application shutdown waits for active Lens flush work and flushes buffered capture before CaptureBuffer stops.

## Configuration reference

Everything `:jido_gralkor` reads, in one place. Nothing else is configurable — Python, the venv, and the Graphiti client are internal concerns with no consumer-facing knobs.

### Application environment (`config :jido_gralkor, …`)

| Key | Type | Default | What it does |
| --- | --- | --- | --- |
| `:falkordb` | keyword: `:host`, `:port`, optional `:username`, `:password`, `:ssl` | unset | Remote FalkorDB connection. Wins over the embedded backend when both are set. `:ssl` defaults to `false`. Invalid shape raises `ArgumentError` at app start. See [Required configuration](#required-configuration). |
| `:embedded_falkordb_socket_timeout_ms` | positive integer | `60_000` | Socket read timeout for the embedded FalkorDB connection, converted to seconds for `AsyncFalkorDB`. Ignored by remote FalkorDB. Invalid values raise `ArgumentError` when the embedded runtime starts. |
| `:client` | module implementing `Gralkor.Client` | `Gralkor.Client.Native` | The adapter. Set to `Gralkor.Client.InMemory` in tests; that value also suppresses the native supervision tree (Pythonx → GraphitiPool → CaptureBuffer). |
| `:lens_storage` | module | `Gralkor.Lens.Storage.Graphiti` | Physical storage behind `Gralkor.Lens.Store`. Set to `Gralkor.Lens.Storage.InMemory` in tests — pinning `:client` alone does **not** intercept `Client.ingest/1`, `replace/1`, or `search/1`. |
| `:destination_storage` | module implementing `Gralkor.Destination.Storage` | `Gralkor.Destination.Storage.Graphiti` | The single memory boundary behind `Client.search/1` and Reflection Destination outputs. Artefact writes create-or-confirm by stable `artefact.id`. Set to `Gralkor.Destination.Storage.InMemory` in tests. |
| `:destinations` | list of Destination definitions | `[]` | Application compatibility registry for direct client calls without an AgentServer target. Mounted agents use their own `:runtime_config`. |
| `:lenses` | list of Lens definitions | `[]` | Application compatibility registry for direct client calls without an AgentServer target. An omitted write mode defaults to `:append`; mounted-agent Lens definitions declare their write mode explicitly. |
| `:recall_deadline_ms` | positive integer | `12_000` | Wall-clock budget for a whole recall search and presentation. On expiry the recall task is killed and `recall/4` returns `{:error, :recall_deadline_expired}`. |
| `:test` | boolean | `false` | Verbose diagnostic logging: recall queries, returned facts, and flushed capture bodies are written to the log. Debugging aid — leave it off in production, where it would log memory contents. |

```elixir
# config/runtime.exs — everything optional, shown with its default
config :jido_gralkor,
  embedded_falkordb_socket_timeout_ms: 60_000,
  recall_deadline_ms: 12_000
```

Mounted-agent Destinations, Lenses, and Reflections come only from each plugin mount's `:runtime_config`, described below. Legacy direct client arities retain application-environment Destination and Lens registries as compatibility surfaces, but mounted agents never fall back to them. `:destination_storage` remains the single storage boundary for both search and Reflection output.

The retired `:reflection_storage` setting causes startup to fail. Configure `:destination_storage` for artefact persistence and declare each Reflection's Destination output in the agent's `:runtime_config`.

The implicit `"operator"` Lens and legacy `capture/5`, `memory_add/3`, and `recall/4` need no ontology configuration. The Lens uses the packaged operator Destination and the library-owned `Gralkor.DefaultOntology`. Application-specific extraction schemas belong on appending Lenses or Reflections.

`recall/4` presents every fact returned by memory search verbatim and in order inside an untrusted memory block, retaining any source wording carried by each fact. Recall makes no second inference call: the consuming agent decides how to interpret the returned memory with its own model.

### Environment variables

| Variable | Default | What it does |
| --- | --- | --- |
| `GRALKOR_DATA_DIR` | unset | Writable directory for the embedded `falkordblite` backend (it spawns a `redis-server` grandchild there). Ignored when `:falkordb` is configured. With neither set, no native runtime starts. |
| `GOOGLE_API_KEY` / `OPENAI_API_KEY` | — | Provider credentials for Graphiti's Python-side clients. Which one you need follows from the two model specs: each of `GRALKOR_LLM_MODEL` and `GRALKOR_EMBEDDER_MODEL` selects a provider, and only a provider some role selects needs its key. A provider selected by neither role needs no key at all. When the native runtime starts, a missing or blank key for a selected provider raises `ArgumentError` before any inference client is constructed, naming the variable and the role (`"llm"` or `"embedder"`). |
| `GRALKOR_LLM_MODEL` | `google:gemini-3.1-flash-lite` | `"provider:model"` spec for the Graphiti LLM. `google:` and `openai:` are supported; another provider raises at native startup, naming both specs and the supported providers. This role's provider also builds the cross-encoder/reranker. GPT-5.5 and GPT-5.6 clients receive `reasoning: "none"` explicitly so Graphiti writes do not inherit an incompatible reasoning tier. |
| `GRALKOR_EMBEDDER_MODEL` | `google:gemini-embedding-2-preview` | Same form and same supported providers, for the embedder — chosen independently of the LLM role, so `openai:` LLM + `google:` embedder is a valid pair (it needs both keys). A Google embedder is constructed with `batch_size: 1`; the OpenAI embedder takes no batch size. |

### Plugin mount options

```elixir
{JidoGralkor.Plugin,
 %{
   agent_name: "Susu",
   ingestion_lens: "observations",
   runtime_config: %{destinations: [], lenses: [], reflections: []}
 }}
```

| Option | Required | Default | What it does |
| --- | --- | --- | --- |
| `:agent_name` | yes | — | Non-blank string naming the agent in captured transcripts. Anything else raises at mount. |
| `:ingestion_lens` | no | unset (implicit-operator mode) | Packaged or `:runtime_config` Lens name receiving `memory_add` and automatic capture. An application-compatibility Lens is not available to a mounted agent unless it is also declared in that mount's runtime configuration. The removed `:default_lens` option raises and identifies this replacement. |
| `:runtime_config` | no | empty consumer collections | The complete consumer-owned Destination, Lens, and Reflection configuration for this agent. Packaged definitions are installed alongside it. Invalid startup configuration prevents the plugin from mounting. |

Per-turn, `tool_context[:lens]` overrides `:ingestion_lens` for that query; the plugin retains the selection on the request's thread entry so later capture stays bound to it.

Search selection is invocation-local, not a plugin mount option. `memory_search` accepts optional `destinations` and `lenses`; the removed `:search_destinations` mount option raises with migration guidance.

Replace the complete configuration for one running agent with `JidoGralkor.Runtime.replace(agent_server_pid, runtime_config)`. Runtime-targeted APIs accept the owning AgentServer PID. Validation and resolution complete before the three collections become active as one snapshot; an error leaves the old snapshot untouched. If the target has no available Gralkor runtime, the operation raises instead of falling back to application compatibility configuration. If that runtime fails, its linked AgentServer terminates and the consumer's supervisor must start a replacement agent with the current durable configuration.

### `JidoGralkor.ContextRotator.rotate_now/2`

| Option | Default | What it does |
| --- | --- | --- |
| `:flush_timeout_ms` | `30_000` | How long the synchronous pre-rotation flush may take. |
| `:keep_last_n` | `4` | Most-recent pre-flush thread entries seeded into the rotated thread. `0` drops everything that existed before the flush; turns that land during the flush are always carried over. |

## A complete configuration

Everything above, in one deployment. Three files.

**Ontologies are modules referenced by writers.** Define an ontology as ordinary compiled Elixir in your own `lib/`, then select it on each appending Lens or Reflection Destination output that should extract with it. Destinations only name graphs.

```elixir
# lib/my_app/ontologies.ex — compiled code. Named by writer definitions below.
defmodule MyApp.Ontology do
  use Gralkor.Ontology, entities: :strict, relationships: :scoped

  entity Teammate, "A person the agent works with." do
    field :handle,   :string, required: true, doc: "stable login handle"
    field :timezone, :string,                 doc: "IANA tz"
  end

  entity WorkingPreference, "A way a teammate prefers to work." do
    field :description, :string, required: true
  end

  from Teammate do
    prefers WorkingPreference do
      field :since, :string, doc: "date first observed"
    end
  end
end
```

```elixir
# config/runtime.exs
import Config

config :jido_gralkor,
  # Backend — pick one. Remote wins if both are present.
  falkordb: [
    host: System.fetch_env!("FALKORDB_HOST"),
    port: String.to_integer(System.fetch_env!("FALKORDB_PORT")),
    username: System.get_env("FALKORDB_USERNAME"),
    password: System.get_env("FALKORDB_PASSWORD"),
    ssl: System.get_env("FALKORDB_SSL") == "true"
  ],

  # Tuning — optional, shown at its default.
  recall_deadline_ms: 12_000

```

```elixir
# lib/my_app/chat_agent.ex — the mount owns this agent's complete domain configuration.
plugins: [
  {JidoGralkor.Plugin,
   %{
     agent_name: "Susu",
     ingestion_lens: "observations",
     runtime_config: %{
       destinations: [],
       lenses: [
         %{
           name: "observations",
           destination: "global",
           write: :append,
           ontology: MyApp.Ontology,
           ingestion: Gralkor.Lens.Ingestion.Store
         },
         %{
           name: "decisions",
           destination: "global",
           write: :append,
           ontology: MyApp.Ontology,
           ingestion: MyApp.DecisionIngestion
         }
       ],
       reflections: [
         %{
           name: "release-review",
           outputs: [
             %{kind: :destination, destination: "global"}
           ],
           chain_of_thought: %{
             steps: [
               %{
                 label: "review",
                 directions: "Review the supplied release evidence.",
                 output: %{"assessment" => "string", "approved" => "boolean"}
               }
             ]
           }
         }
       ]
     }
   }}
]
```

That mount writes captured turns and `memory_add` calls through the `"observations"` Lens to `global`. Memory search independently defaults to every accessible Destination registered for this agent and may narrow each call by Destination and Lens. Ingestion does not invoke Reflections; a consumer or consumer-owned scheduled job submits a named Reflection asynchronously when its workflow requires synthesis.

**Ontology placement.** Appending Lenses and Reflection Destination outputs default to Jido Gralkor's open `Gralkor.DefaultOntology`. Packaged ERL explicitly uses `Gralkor.Reflection.ERLOntology`. Applications attach custom ontology modules to their writers. If an older deployment set `config :jido_gralkor, :ontology`, remove it and select the module on each Lens or Reflection Destination output that needs it.

## Wire it on your agent

```elixir
defmodule MyApp.ChatAgent do
  use Jido.Agent,
    name: "my_chat",
    schema: [user_name: [type: :string, required: true]],
    strategy:
      {Jido.AI.Reasoning.ReAct.Strategy,
       tools: [
         JidoGralkor.Actions.MemorySearch,
         JidoGralkor.Actions.MemoryAdd,
         # ... your other tools
       ],
       system_prompt: """
       You are a helpful assistant with long-term memory.

       Use memory_search when answering benefits from past context.
       Use memory_add to record explicit insights you want to preserve
       beyond the conversation that's already being auto-captured.
       """,
       request_transformer: MyApp.ChatAgent.RequestTransformer},
    default_plugins: %{__memory__: false},
    plugins: [
      {JidoGralkor.Plugin,
       %{
         agent_name: "Susu",
         ingestion_lens: "observations",
         runtime_config: %{
           destinations: [],
           lenses: [
             %{
               name: "observations",
               destination: "global",
               write: :append,
               ontology: MyApp.Ontology,
               ingestion: Gralkor.Lens.Ingestion.Store
             }
           ],
           reflections: []
         }
       }}
    ]

  # Optional: pin tool_choice to memory_search on iteration 1 so the agent
  # itself authors a focused recall query in-thread.
  defmodule RequestTransformer do
    @behaviour Jido.AI.Reasoning.ReAct.RequestTransformer

    @impl true
    def transform_request(_request, state, _config, _runtime_context) do
      {:ok, JidoGralkor.ReAct.maybe_force_memory_search(%{}, state)}
    end
  end
end
```

The plugin claims Jido's `:__memory__` slot. On `ai.react.query`, it plants `:session_id` (when a thread is committed), `:agent_name`, and the selected ingestion `:lens` on the signal's `tool_context`. Search selectors come from each `memory_search` call. Recall itself is the LLM's job — `JidoGralkor.ReAct.maybe_force_memory_search/2` is the cheapest way to force it on iteration 1. Capture runs automatically on completion and failure: the ReAct event trace is normalised into Gralkor's canonical `[%Gralkor.Message{role, content}]` shape via `JidoGralkor.Canonical` — `user` for the user query, `behaviour` for intermediate thinking / tool calls / tool results, `assistant` for the final answer on completed turns, or a terminal `"request failed: …"` `behaviour` on failed turns so the failure stays visible to downstream distillation.

Set `tool_context[:lens]` on an individual query to override `ingestion_lens` for that turn. The plugin retains the selection on the request's Jido thread entry, making it authoritative for both `memory_add` and later completion or failure capture after ReAct has released its transient tool context.

The plugin reads `user_name` per-turn from `agent.state[:user_name]`. Populate it before each request (for example, via `on_before_cmd/2` from the signal's `tool_context`) so distill renders user lines under the correct human identity. Missing and blank names raise; there is no generic fallback.

## What happens at runtime

**Session identity.** `session_id` is the current Jido thread id (read from `agent.state[:__thread__].id`, populated by `Jido.Thread.Plugin`). The plugin does not mint its own identifier — Jido's thread lifecycle is the single source of truth.

**Destinations.** Every Lens and every Reflection Destination output references a registered Destination, which resolves to one logical graph ID: `global`, `operator/<operator id>`, or an application Destination's exact shared name. Application names beginning `operator/` are reserved for operator-local graphs and rejected. At the Graphiti boundary, each logical ID is encoded exactly once as `g_` followed by the lowercase hexadecimal encoding of every original byte. Appending Lenses and Destination outputs govern their own extraction. Multiple writers may save to the same Destination. Replacement writes inject `_gralkor_lens` into supplied nodes and relationships so a replaceable Lens changes only its own content there.

The physical encoding replaces the former lossy `-` and `/` to `_` normalisation. Graphs stored under old physical names are not read or migrated automatically; migrate them only from known logical Destination/operator IDs, or re-ingest their source content, because underscores cannot recover the original ID.

**Reflection invocation.** Lens-aware capture requires a non-blank operator before buffering, and CaptureBuffer assigns each buffered ingestion one cryptographically collision-resistant ID that it reuses across flush retries. Capture and ordinary ingestion stop after Lens ingestion; neither invokes a Reflection. The consuming application owns the event, job, or schedule that selects a configured Reflection and submits it with `Gralkor.Client.reflect/5`. Submission returns its replay-stable invocation ID immediately; the agent-owned runtime independently produces and delivers the artefact and reports the terminal outcome through the invocation callback.

**First-turn bootstrap.** On the very first query of a fresh agent, the thread isn't yet committed (the ReAct strategy's `ThreadAgent.append` runs after the plugin hook). The plugin plants `:agent_name` plus the configured ingestion `:lens`, but no `:session_id`; completed and failed turn capture are both skipped with a warning until a committed thread supplies that identity. `memory_search` still searches for the current operator because public Search does not depend on conversation-session identity.

**Death-triggered flush.** `JidoGralkor.Lifecycle` is an optional `Jido.AgentServer.Lifecycle` implementation. When wired as `lifecycle_mod:` on the agent, graceful termination schedules the configured client's `flush/1` for the active thread before termination returns. Lens definitions are resolved from one runtime snapshot before the ingestion worker starts, so that worker no longer depends on the agent runtime remaining alive; termination does not wait for ingestion itself. The plugin mount alone does not enable this lifecycle. No idle-timer machinery — Jido's `AgentServer` owns `:idle_timeout` directly.

```elixir
{:ok, pid} =
  MyApp.Jido.start_agent(
    MyApp.ChatAgent,
    id: "operator-42",
    initial_state: %{user_name: current_user.name},
    lifecycle_mod: JidoGralkor.Lifecycle
  )
```

**Context rotation.** `JidoGralkor.ContextRotator.rotate_now/2` synchronously flushes the active session via `flush_and_await/2`, installs a fresh Jido thread, and seeds the rotated thread with the most-recent `:keep_last_n` pre-flush entries plus any turns that landed during the flush. It returns `:ok` when there is no committed thread and `{:error, reason}` when state reading, flushing, or thread installation fails. The agent process is never stopped. Use it from a `/new` chat command or a small wrapper GenServer that fires on an interval.

**Error contracts.** Invalid configuration, invalid Lens requests, automatic plugin-capture failures, non-PID runtime targets, and unavailable targeted runtimes raise. Runtime-targeted operations never redirect to application compatibility configuration. Valid runtime-targeted `Gralkor.Client.ingest/2`, `replace/2`, `search/2`, `reflect/5`, and adapter operations return tagged success/error tuples; the ReAct search action propagates those errors. The asynchronous `memory_add` action logs background failures and still returns immediately, as described below.

**`memory_add` is async.** The tool returns `"Ingesting."` immediately and does the storage call in a background `Task`. Graphiti's entity/edge extraction can take tens of seconds; you don't want the agent waiting. Failures are logged; best-effort storage is the contract.

## Configure Lenses

A Lens is an application-owned memory ingestion channel that targets a Destination. An appending Lens selects its extraction ontology; its write mode sends content through an ingestion process. A whole-graph replacement Lens replaces its own graph content at the Destination.

An appending Lens declares `write: :append`, its Destination, and the ingestion process Gralkor invokes when content is sent through it.

The ontology is a module you compile into your own application — declared once in `lib/`, then named by each appending Lens that should extract with it:

```elixir
# lib/my_app/ontology.ex
defmodule MyApp.Ontology do
  use Gralkor.Ontology, entities: :strict, relationships: :scoped

  entity Teammate, "A person the agent works with." do
    field :handle,   :string, required: true, doc: "stable login handle"
    field :timezone, :string,                  doc: "IANA tz"
  end

  entity WorkingPreference, "A way a teammate prefers to work." do
    field :description, :string, required: true
  end

  from Teammate do
    prefers WorkingPreference do
      field :since, :string, doc: "date first observed"
    end

    trusts Teammate
  end
end
```

Point as many Lenses as your application needs at `global`, `operator`, or an application Destination. Several Lenses may use the same Destination with different ontologies:

```elixir
runtime_config = %{
  destinations: [],
  lenses: [
    %{
      name: "observations",
      destination: "global",
      write: :append,
      ontology: MyApp.Ontology,
      ingestion: Gralkor.Lens.Ingestion.Store
    },
    %{
      name: "decisions",
      destination: "global",
      write: :append,
      ontology: MyApp.Ontology,
      ingestion: MyApp.DecisionIngestion
    }
  ],
  reflections: []
}
```

A replaceable Lens declares `write: :replace_graph` and a Destination instead of `:ingestion` or `:ontology`:

```elixir
runtime_config = %{
  destinations: [],
  lenses: [
    %{
      name: "systems",
      destination: "global",
      write: :replace_graph
    }
  ],
  reflections: []
}
```

Destination names control visibility: `operator` resolves a separate `operator/<operator id>` graph for each operator; `global` and application Destination names resolve to one shared graph each.

`Gralkor.Lens.Ingestion.Store` is the built-in straight-through process. A consumer can define any other ingestion process by implementing one callback:

```elixir
defmodule MyApp.DecisionIngestion do
  @behaviour Gralkor.Lens.Ingestion

  @impl true
  def ingest(request, store) do
    with {:ok, decisions} <- MyApp.Decisions.extract(request.content) do
      Enum.reduce_while(decisions, :ok, fn decision, :ok ->
        case Gralkor.Lens.Store.add(store, decision, request.source_description) do
          :ok -> {:cont, :ok}
          {:error, reason} -> {:halt, {:error, reason}}
        end
      end)
    end
  end
end
```

The callback receives the original `%Gralkor.Ingest{}` request and a Lens-bound `%Gralkor.Lens.Store{}`. It decides whether to make zero, one, or many writes and can use `Gralkor.Lens.Store.add/3` and `search/3`. The selected Destination supplies the graph; the Lens supplies the ontology. Runtime-targeted `Client.ingest/2` accepts appending Lenses and raises for replaceable Lenses; `Client.replace/2` accepts replaceable Lenses and raises for appending Lenses.

The plugin mount chooses how an agent uses the registered Lenses:

```elixir
{JidoGralkor.Plugin,
 %{
   agent_name: "Susu",
   ingestion_lens: "observations",
   runtime_config: runtime_config
 }}
```

- `ingestion_lens` receives `memory_add` calls and automatic capture unless a turn supplies `tool_context[:lens]`.

Search is independent of that mount. With only a query, `memory_search` searches episodes in every accessible registered Destination. Optional selectors narrow one invocation; names are ORed within each list and the two lists intersect:

```elixir
%{
  query: "What did earlier rollouts teach us?",
  destinations: ["global", "release-notes"],
  lenses: ["observations", "decisions"]
}
```

This includes episodes only when their Destination is either selected Destination and their originating Lens is either selected Lens. Selecting a Lens never adds its Destination. A Lens selector applies only to episode results; direct callers may still explicitly request facts, nodes, or Reflection artefacts without a Lens selector.

Consumers pass the owning AgentServer PID whose runtime configuration should resolve each named operation. A search resolves all selected Lenses and Destinations from one atomic snapshot:

```elixir
:ok =
  Gralkor.Client.ingest(agent_server, %Gralkor.Ingest{
    id: "release-planning-2026-08-28",
    operator_id: "operator-42",
    lens: "decisions",
    source_kind: :conversation,
    content: "We chose Friday.",
    source_description: "release planning"
  })

{:ok, memories} =
  Gralkor.Client.search(agent_server, %Gralkor.Search{
    operator_id: "operator-42",
    query: "When should we release?",
    destinations: ["global"],
    lenses: ["decisions"],
    max_results: 20
  })

:ok =
  Gralkor.Client.replace(agent_server, %Gralkor.Replace{
    operator_id: "operator-42",
    lens: "systems",
    graph: %Gralkor.Graph{
      nodes: [
        %{id: "payments", labels: ["System"], properties: %{name: "Payments"}},
        %{id: "ledger", labels: ["System"], properties: %{name: "Ledger"}}
      ],
      relationships: [
        %{
          from: "payments",
          to: "ledger",
          type: "DEPENDS_ON",
          properties: %{protocol: "events"}
        }
      ]
    }
  })
```

Every ingestion requires a non-blank `operator_id`, a non-blank replay-stable `id`, and deterministic provenance through `source_kind`. Both identifiers are validated before Lens storage begins. A consumer may use that ingestion ID as a related Reflection invocation ID; `Gralkor.Reflection.Runner` derives the artefact ID from the operator ID, invocation ID, and Reflection name. Reusing an ingestion ID does not deduplicate the Lens episode write.

The supported source kinds are:

- `:conversation` accepts speaker-attributed text and becomes a Graphiti message episode.
- `:document` accepts text and becomes a Graphiti text episode.
- `:structured_record` accepts a JSON-compatible map or list, which Gralkor JSON-encodes for a Graphiti JSON episode.

`source_kind` says where the information came from; it is not a credibility, truth, opinion, or speculation rating. Gralkor validates the kind and its content shape before invoking a Lens or Graphiti. Automatic turn capture declares `:conversation`; callers using `Gralkor.Ingest` or `memory_add/4` declare the kind themselves, while legacy `memory_add/3` remains a document-text compatibility call. The same Graphiti extraction call receives static guidance to retain attribution and epistemic wording such as uncertainty or speculation—Gralkor does not run a second presentation-classification inference. Fact results resolve their originating episodes and append the episode identifier, source kind, and source description on recall.

Appending Lens episodes record writer provenance by suffixing their source description with ` [lens: <Lens name>]`; Reflection episodes use the exact source description `reflection:<Reflection name>`. Both registries reject names containing the Lens delimiter so these forms stay unambiguous. This episode-writer provenance is separate from `_gralkor_lens`, which owns replacement-graph nodes and relationships.

Search defaults to stored episodes across every accessible registered Destination: the current operator's private `operator/<operator id>` logical graph plus every shared Destination. Supplying `destinations` narrows the graphs; supplying `lenses` narrows writers; OR applies within either list and both dimensions must match when both are present. Destination searches run concurrently while results retain selected Destination order. `max_results` defaults to `20`, must be a positive integer, and applies independently after writer filtering in every Destination. Each episode identifies its Destination plus its originating Lens or declaring Reflection; raw legacy episodes that carry neither trusted writer marker are omitted before that limit rather than assigned invented provenance. Direct callers may explicitly request `:facts`, `:nodes`, or `:artefacts` without Lens selectors; node searches accept `entity_types`, fact searches accept `edge_types`, and artefact searches may narrow by `artefact_id`. The `memory_search` action returns attributed episode results as JSON.

`%Gralkor.Graph{nodes:, relationships:}` is the sole replacement representation. Every node requires a unique, non-blank string `:id`, a list of non-blank string `:labels`, and a `:properties` map. Every relationship requires `:from` and `:to` identifiers naming supplied nodes, a non-blank string `:type`, and a `:properties` map. This payload is the whole current graph for the Lens; partial node and relationship operations are not supported.

Replacement changes only content owned by that Lens at its Destination. Gralkor overwrites any supplied `_gralkor_lens` property with the selected Lens name on every inserted node and relationship. Content saved through another Lens or Reflection, or carrying no Lens ownership, remains unchanged. An empty graph removes all graph content owned by the selected Lens. No graph-format option exists.

Invalid Lens names, write modes, and graph data raise `ArgumentError`; graph data is fully validated before storage mutation begins. Once a valid replacement starts, deletion and insertion are not transactional: an import error is returned, and content already removed or inserted is not rolled back.

Runtime and plugin configuration fail fast for blank, duplicate, reserved, retired, or malformed Lens definitions and for unknown Lens names. The retired `"default"` Lens name raises with guidance to use `"operator"`; it is not an alias. If no Lens configuration is used, the implicit `"operator"` Lens writes to `operator/<operator id>` and uses Jido Gralkor's built-in generic extraction contract.

### Ontology DSL

Each writer ontology is a module declared with `Gralkor.Ontology`:

- `entity Foo do field … end` declares an entity. `field :name, :type, opts` supports `:string | :integer | :float | :boolean`, plus `required: true` and `doc:` (rendered as the Pydantic field description).
- `entity Foo, "when to extract one" do … end` adds a description, rendered as the extracted type's own description. Graphiti's extractor reads it to decide when to mint the entity, so a type whose name alone is ambiguous — `Preference`, `Pattern`, `Learning` — is extracted far more reliably with one. The description must be a literal string.
- `from Source do verb Target [do field … end] end` declares outgoing relationships from `Source`. The verb's name becomes the edge type in graphiti (`prefers` → `"PREFERS"`, `relates_to` → `"RELATES_TO"`). The optional `do` block carries edge properties.
- Same verb in multiple `from` blocks becomes one edge type with multiple endpoint pairs.
- `entities: :strict` excludes graphiti's generic `Entity` extraction — only your declared types survive. `entities: :open` lets graphiti extract generic Entity nodes alongside yours.
- `relationships: :scoped` populates graphiti's `edge_type_map` from your declared `(src, dst)` pairs, so named edges only fire between declared endpoints. `relationships: :open` drops the map; graphiti's default applies. Either way, graphiti always extracts edge candidates — generic fall-through edges between unconstrained pairs are not closed off.
- Both opts are required at `use` — no defaults; pick deliberately.

**Protected names.** Runtime configuration and the application-compatibility Lens registry reject custom entity kinds named `Entity`, `Episodic`, or `Community`, which are Graphiti core labels; `Person` is not reserved. Graphiti also rejects a custom entity attribute whose field name collides with its own `EntityNode`: `uuid`, `name`, `group_id`, `labels`, `created_at`, `summary`, `attributes`, and `name_embedding`. The DSL does not currently catch field collisions at compile time, so `field :name, :string` compiles and then raises `EntityTypeValidationError` from Python on the first write through the Lens that selected the ontology. Name fields for what they hold — `handle`, `title`, `statement` — rather than reaching for `name` or `summary`.

On each store write, graphiti receives the selected Lens or Reflection ontology's `entity_types`, `edge_types`, `edge_type_map`, and `excluded_entity_types`, translated from the module's compile-time payload.

## Configure Reflections

A Reflection is a named synthesis process declared in one agent's runtime configuration with an inline structured Chain of Thought and exactly one Destination output. The consumer decides when it runs; after admission, that agent's JidoGralkor runtime owns production, Destination delivery, retry, abandonment, and the invocation callback.

The package supplies two declarations by default:

- `generalisations` declares a `global` Destination output using `Gralkor.DefaultOntology`.
- `erl` declares an `operator` Destination output using `Gralkor.Reflection.ERLOntology`, whose `Learning` entity declares optional `problem_kind`, `approach`, `success`, and `lesson` fields.

Package-owned definitions are always installed alongside the consumer's `runtime_config.reflections`; consumers cannot replace them. A Destination output's optional `:ontology` defaults to `Gralkor.DefaultOntology`.

```elixir
runtime_config = %{
  destinations: [],
  lenses: [],
  reflections: [
    %{
      name: "release-review",
      outputs: [
        %{
          kind: :destination,
          destination: "global",
          ontology: MyApp.ReleaseOntology
        }
      ],
      chain_of_thought: %{
        steps: [
          %{
            label: "review",
            directions: "Review the supplied release evidence.",
            output: %{"assessment" => "string", "approved" => "boolean"}
          }
        ]
      }
    }
  ]
}
```

A consumer submits a configured Reflection through the AgentServer that owns it. Submission returns the replay-stable invocation ID without waiting for inference, storage, or callback delivery:

```elixir
operator_id = "operator-42"
invocation_id = "release-2026-09-02"

invocation = %{
  id: invocation_id,
  operator_id: operator_id,
  invocation_context: %{reason: "release decision"},
  representations: completed_representations
}

review_consumer = self()
callback = fn result -> send(review_consumer, {:release_review, result}) end

{:ok, ^invocation_id} =
  Gralkor.Client.reflect(
    agent_server,
    "release-review",
    invocation,
    callback,
    tools: host_tools,
    tool_context: host_tool_context
  )
```

The callback eventually receives a map with `:invocation_id`, the terminal `:outcome`, and `:artefact` when production succeeded. A successful delivery reports `outcome: :delivered`. A non-retryable production failure reports `{:production_failed, reason}`. Abandonment reports `{:abandoned, %{stage: :production | :delivery, reason: reason}}`; delivery abandonment still includes the produced artefact.

Each admitted invocation progresses independently. A 5xx failure from inference, packaged-generalisation related-memory retrieval, or Destination delivery retries with exponential backoff until success or twenty-four hours from the first failed attempt. A 4xx failure is abandoned immediately. Unfinished work terminates with the agent without invoking its callback; the consumer owns durable scheduling and may resubmit after its supervisor starts a replacement agent.

Each inline Chain of Thought contains an ordered, non-empty `steps` list. A step declares a non-blank `label`, natural-language `directions`, and an exact non-empty structured `output` schema. Output types are `string`, `boolean`, `integer`, recursively typed arrays such as `Array<string>`, and exact objects such as `{ content: string; level: integer }`. Later directions may interpolate only prior outputs with `{{output_name}}`. Each step receives the invocation identity and context, completed lensed representations, host tools, and tool context. The final step becomes one `%Gralkor.Artefact{}` whose fields are exactly `id` and `payload`.

Before packaged generalisation inference begins, Gralkor performs one default episode search through the targeted agent runtime across every accessible registered Destination. The same search supplies related Lens-authored observations and prior Reflection-authored generalisations. The packaged prompt directs a new generalisation with no lineage to use level 1 and an evolved generalisation to use one level above its highest lineage snapshot. Generalisation lineage is validated only by the structured output contract; Gralkor preserves the model-produced values without comparing them with the related-memory input.

Programmatic requests and consumer-owned scheduled jobs use the same `Client.reflect/5` path. Gralkor does not own cron, calendar scheduling, or durable job state.

Multiple Reflections and Lenses may save to the same Destination. Search selects Destinations directly:

```elixir
{:ok, artefacts} =
  Gralkor.Client.search(agent_server, %Gralkor.Search{
    operator_id: "operator-42",
    query: "What release approaches have worked?",
    destinations: ["operator"],
    result_type: :artefacts,
    max_results: 20
  })

{:ok, [artefact]} =
  Gralkor.Client.search(agent_server, %Gralkor.Search{
    operator_id: "operator-42",
    query: "",
    destinations: ["operator"],
    result_type: :artefacts,
    artefact_id: "reflection-123"
  })
```

`result_type: :artefacts` returns artefacts from the selected Destinations, and `artefact_id` optionally narrows the lookup to one exact artefact. The deterministic UUID derives from the operator ID, invocation ID, and Reflection name, but the artefact itself contains no producer identity. Before extraction, Graphiti waits for a graph uniqueness constraint and acquires a graph-backed UUID claim with a renewable lease, so independent application runtimes serialize equal work and reject conflicting immutable content before Graphiti's upsert can overwrite it. Lease expiry and renewal use the graph server's clock; every ownership transfer advances a generation. For a claimed deterministic UUID, the episode, every extracted entity and edge, and its extraction-completion marker persist in one generation-fenced FalkorDB query, so loss of ownership aborts the complete graph-effect set before it commits. Until that marker exists, canonical lookup retains the episode only as resumable storage state and artefact search excludes it. Repeating an equal marked write is a successful confirmation without extraction; an equal unmarked deterministic episode re-enters normal extraction without rerunning the Runner. Artefact search uses ranked completed episodes to select up to the requested number of artefact identifiers, then enumerates every completed episode carrying those identifiers independently of BM25. It collapses equal historical duplicates by artefact ID, preserves the selected ranking, and reports an artefact conflict if any completed duplicate under a selected ID disagrees.

Reflection artefact episodes written before the extraction-completion marker existed remain hidden because an absent marker cannot distinguish a fully extracted legacy write from a partial write. Invoke the original Reflection again with the same stable invocation ID and deliver its artefact to the same Destination to establish the marker; unmarked legacy episodes remain non-searchable. An operator-authored migration may mark a legacy episode only after independently verifying that its extraction completed and its immutable artefact content is the intended canonical value. This does not migrate an episode left under the former physical graph-ID encoding.

## Testing against the in-memory twin

`Gralkor.Client.InMemory` is a real implementation of `Gralkor.Client` (not a mock) that stores canned responses and records every call. Your agent's integration tests can hit it without any network:

For Lens-aware ingest and replacement calls, pair it with `Gralkor.Lens.Storage.InMemory` as shown in Required configuration. Destination search uses `Gralkor.Destination.Storage.InMemory` instead.

```elixir
setup do
  Gralkor.Client.InMemory.reset()
  :ok
end

test "agent recalls stored context" do
  Gralkor.Client.InMemory.set_recall({:ok, "<gralkor-memory>known fact</gralkor-memory>"})
  Gralkor.Client.InMemory.set_capture(:ok)
  # ... exercise your agent, assert on responses, inspect recorded calls
end
```

The shared `Gralkor.ClientContract` macro suite exercises the in-memory twin. The production `Gralkor.Client.Native` adapter proves the same public port separately in its adapter tests.

Maintainers can use the project test aliases according to the feedback speed and runtime boundary they need:

```bash
mix test              # Unit and Integration; excludes Functional and Journey
mix test.unit         # Unit
mix test.integration  # Integration
mix test.functional   # Functional; may call real model providers
mix test.journey      # Journey
mix test.fast         # stale/affected Unit and Integration
mix test.changed      # stale/affected Unit, Integration, and Functional; excludes Journey
mix test.all          # Unit, Integration, Functional, Journey, and Node tests
```

Functional tests require their documented provider credentials and can send test inputs to external model providers. `mix test.fast` is the routine local feedback command; use `mix test.changed` only when the affected Functional boundary is intentionally available.

## What's in the library

The Jido glue:

- `JidoGralkor.Plugin` — `use Jido.Plugin, state_key: :__memory__, singleton: true`. Handles `ai.react.query` (planting session, agent, and selected ingestion Lens) and `ai.request.completed` / `ai.request.failed` (capture).
- `JidoGralkor.ReAct` — `maybe_force_memory_search/2` helper. Folds `tool_choice: %{type: "function", function: %{name: "memory_search"}}` into ReAct overrides on iteration 1; passes through unchanged on iterations 2+.
- `JidoGralkor.Canonical` — normalises a Jido/ReAct turn into the canonical `[%Gralkor.Message{role, content}]` shape.
- `JidoGralkor.Lifecycle` — `Jido.AgentServer.Lifecycle` impl whose sole job is the death-triggered flush.
- `JidoGralkor.ContextRotator` — synchronous `rotate_now/2` for in-life context consolidation.
- `JidoGralkor.Actions.MemorySearch` — the ReAct tool that always calls runtime-targeted `Gralkor.Client.search/2` for the current operator, using optional Destination and Lens selectors from that invocation. It works before a thread is committed and short-circuits only a blank query.
- `JidoGralkor.Actions.MemoryAdd` — fire-and-forget ReAct tool.
- `JidoGralkor.Actions.MemoryBuildIndices` — admin tool. Description tells the LLM `DO NOT CALL` unless the user asked. Whole-graph index rebuild.
- `JidoGralkor.Actions.MemoryBuildCommunities` — admin tool. Same `DO NOT CALL` guard. Runs Graphiti community detection on this agent's `operator/<operator id>` graph.

The embedded Gralkor adapter (under `lib/gralkor/`):

- `Gralkor.Client` — adapter behaviour plus runtime-targeted `ingest/2`, `replace/2`, `search/2`, capture, and asynchronous `reflect/5` boundaries; legacy application-registry arities remain compatibility surfaces.
- `Gralkor.Client.Native` — production adapter; wires `Recall`, `CaptureBuffer`, and `GraphitiPool`.
- `Gralkor.Client.InMemory` — test twin.
- `Gralkor.Destination` and `Gralkor.Destination.Registry` — first-class named graphs shared by Lenses and Reflections. The full agreed model is in [DESTINATIONS.md](DESTINATIONS.md).
- `Gralkor.Lens`, `Gralkor.Lens.Replaceable`, `Gralkor.Ingest`, `Gralkor.IngestedRepresentation`, `Gralkor.Replace`, `Gralkor.Graph`, `Gralkor.Search` — resolved ingestion models, completed-ingestion representation, and consumer request values.
- `Gralkor.Lens.Store` / `Gralkor.Lens.Storage.Graphiti` — append, replacement, and search capabilities for exact Destination graph identities.
- `Gralkor.Lens.Ingestion.Store` — the built-in straight-through ingestion process.
- `JidoGralkor.Runtime` — the per-AgentServer atomic configuration snapshot and supervisor for admitted Reflection production, Destination delivery, retry, and callback.
- `Gralkor.Reflection`, `Gralkor.Reflection.ChainOfThought`, and `Gralkor.Reflection.Runner` — structured declarations validated by the agent runtime and the inner ordered-inference engine.
- `Gralkor.Artefact` and `Gralkor.Destination.Storage` — producer-independent artefacts and their canonical Destination storage boundary.
- `Gralkor.Ontology` — compile-time DSL for declaring graphiti custom-entity ontologies (`entity`/`field`/`from`/verb macros).
- `Gralkor.Application`, `Gralkor.Python`, `Gralkor.GraphitiPool`, `Gralkor.CaptureBuffer`, `Gralkor.Recall`, `Gralkor.Distill`, `Gralkor.Format`, `Gralkor.Config`, and `Gralkor.Message` — the embedded capture, recall, and Graphiti pipelines.

The behavioural contract lives in [`test-trees/`](https://github.com/elimydlarz/jido_gralkor/tree/main/test-trees). Functional trees describe each application-visible feature, and the Journey tree describes the broad whole-application workflow. [`CLAUDE.md`](https://github.com/elimydlarz/jido_gralkor/blob/main/CLAUDE.md) carries the maintainer-facing mental model and project guidance.

## Publishing (maintainers)

`:jido_gralkor` is published to the public Hex registry as a package owned by the `elimydlarz` Hex user. Releases use that user's API key (`HEX_TOKEN`) loaded from the workspace `.env`; see the workspace `publish` skill for the full release flow.

```bash
$publish patch   # or minor | major | current
```

The skill loads `.env` and runs the full suite before changing release state, verifies or transfers the package to personal ownership, lets trunk-sync synchronize the version commit and default branch, publishes through the personal Hex token, and creates the lightweight `jido-gralkor-v<version>` tag through GitHub's API. Final verification requires that immutable tag to resolve to the published commit and the remote default branch to contain it, so later trunk-sync bookkeeping commits do not invalidate a completed release. Copy `.env.example` to `.env` and provide the configured inference credential, `HEX_TOKEN`, and a repository-scoped `GH_TOKEN` with Contents write permission.

## License

MIT.