Packages
LLM wire-protocol codecs with a canonical IR, a one-call client on Finch, and a replaceable JSON layer.
Current section
Files
Jump to
Current section
Files
README.md
# LlmCall
A call-oriented SDK for LLM APIs: a canonical IR, dialect codecs, and a one-call
client on Finch whose JSON layer you can replace.
```elixir
defmodule MyApp.OpenAI do
use LlmCall.Provider,
base_url: "https://api.openai.com",
auth: :bearer,
codecs: [chat: LlmCall.Dialect.OpenAI.Chat]
end
client = MyApp.OpenAI.client(api_key: System.fetch_env!("OPENAI_API_KEY"))
request = %LlmCall.Request{
model: "gpt-4o",
messages: [%LlmCall.Message{role: :user, content: [%LlmCall.Content.Text{text: "Hello"}]}]
}
{:ok, response} = LlmCall.complete(client, request)
LlmCall.Response.text(response)
```
`complete/3` validates, encodes, sends, and decodes. `stream/3` does the same and
returns a lazy `Enumerable` of events. `generate/3` and `poll/3` cover dedicated
image and video endpoints, including long-running jobs.
## Installation
```elixir
def deps do
[
{:llm_call, "~> 0.1.0"}
]
end
```
Requires Elixir 1.18+ on OTP 27+. Runtime dependencies are Finch and
`mint_web_socket`; everything else is dev/test only.
## Providers
A provider is a module you write: endpoint, auth shape, and the codecs it serves.
There is no built-in catalog โ `examples/` carries reference definitions.
```elixir
defmodule MyApp.Groq do
use LlmCall.Provider,
base_url: "https://api.groq.com/openai",
auth: :bearer,
codecs: [chat: LlmCall.Dialect.OpenAI.Chat]
end
client = MyApp.Groq.client(api_key: key)
```
`codecs:` is a named map because one endpoint can serve several request shapes.
An endpoint that returns text and images is still **one** chat codec
(`modalities: [:text, :image]`); a different request shape โ Imagen's `predict`,
OpenAI Images โ is a second codec selected per call:
```elixir
{:ok, %LlmCall.Media.Response{assets: assets}} =
LlmCall.generate(client, media_request, codec: :images)
```
Auth shapes: `:none`, `:bearer`, `:x_api_key`, `{:header, name}`, or
`{module, function}`. The key is always passed in; nothing reads the environment.
`query:` adds URL parameters to every request, for vendors that key on one โ
MiniMax's `GroupId`, Azure OpenAI's `api-version`. A codec's own parameter still
wins on the same key, and the Responses WebSocket handshake gets them too. A
query already written into `base_url` is folded into `query:` when the client is
built, so pasting `https://โฆ/openai?api-version=2024-02-01` works.
```elixir
defmodule MyApp.MiniMax do
use LlmCall.Provider,
base_url: "https://api.minimax.io",
auth: :bearer,
codecs: [video: LlmCall.Media.Dialect.MiniMax.Video]
end
client =
MyApp.MiniMax.client(
api_key: System.fetch_env!("MINIMAX_API_KEY"),
query: %{"GroupId" => System.fetch_env!("MINIMAX_GROUP_ID")}
)
```
The declaration only takes literals, so a value read at runtime belongs on
`client/1`.
## Streaming
```elixir
{:ok, stream} = LlmCall.stream(client, %{request | stream: true})
stream
|> Stream.each(&IO.inspect({&1.type, &1.delta}))
|> Stream.run()
```
The library parses SSE and provider terminators. A failure while the body is being
read raises `LlmCall.HTTP.Error`, whose `error/1` returns the canonical
`%LlmCall.Error{}` with the HTTP status; `{:halt, acc}` from a reducer closes the
connection. `complete/3` rejects `stream: true` rather than silently flipping it.
To observe events without owning the reduce, pass `on_event:`:
```elixir
{:ok, stream} =
LlmCall.stream(client, request, on_event: &MyApp.Audit.stream_event/1)
```
It runs before the reducer, including for `finish_stream/2` output; the return
value is ignored, and it is not called after the reducer halts. `LlmCall.HTTP` is
the only HTTP transport, so raw bytes are not exposed as a hook.
A reasoning block spans several events. The text arrives incrementally, and a
provider that signs its thinking blocks sends the signature in its own event, with
an empty `text` โ Anthropic on every block, Gemini on the thought parts before a
function call. Keep both when you fold the stream into an assistant turn, or the
block cannot be replayed in the next request:
```elixir
def merge(%LlmCall.Content.Thinking{} = block, %LlmCall.StreamDelta.Thinking{} = delta) do
%{block | text: block.text <> delta.text, signature: delta.signature || block.signature}
end
```
### Responses over WebSocket
The Responses protocol also has a multiplexed WebSocket connection:
named `stream_id` lanes, decoded events, and reconnect before the 60-minute
limit.
```elixir
{:ok, connection} = LlmCall.Dialect.OpenAI.Responses.Connection.start_link(client: client)
{:ok, stream} = LlmCall.Dialect.OpenAI.Responses.Connection.create(connection, request, stream_id: "main")
Enum.to_list(stream)
```
An in-flight response cannot resume across a reconnect; its stream raises
`LlmCall.WS.Error`. The connection reconnects with exponential backoff and does
not stop on failure: while disconnected, `create/3` returns a `:transport` error
and `Connection.reconnect/1` retries immediately. `Connection.response_id/2`
returns the provider id for `previous_response_id`.
Tool-call fragments are assembled into complete calls by a pure helper, which never
runs them:
```elixir
assembler = LlmCall.StreamAssembler.new(json_decoder: &JSON.decode/1)
{:ok, assembler, calls} =
Enum.reduce(events, {:ok, assembler, []}, fn event, {:ok, acc, calls} ->
{:ok, acc, new_calls} = LlmCall.StreamAssembler.feed(acc, event)
{acc, calls ++ new_calls}
end)
```
## Media generation
A completion can ask for image output; generated parts arrive in
`Response.content`:
```elixir
request = %LlmCall.Request{
model: "gemini-2.5-flash-image",
modalities: [:text, :image],
messages: [%LlmCall.Message{role: :user, content: [%LlmCall.Content.Text{text: "a cat"}]}]
}
{:ok, response} = LlmCall.complete(client, request)
# response.content may contain %LlmCall.Content.Image{}
```
Dedicated endpoints take a prompt and media parameters instead of messages:
```elixir
media = %LlmCall.Media.Request{
modality: :image,
model: "gpt-image-1",
prompt: "a cat, watercolor",
params: %{n: 2, size: "1024x1024"}
}
{:ok, %LlmCall.Media.Response{assets: assets}} =
LlmCall.generate(client, media, codec: :images)
```
Asynchronous video returns an `%LlmCall.Operation{}` whose `next` is the request
to execute; `poll/3` runs it and returns the next state:
```elixir
{:ok, %LlmCall.Media.Response{operation: op}} = LlmCall.generate(client, video, codec: :video)
{:ok, %LlmCall.Operation{status: :completed} = done} = LlmCall.poll(client, op, codec: :video)
```
`next_kind: :poll` means "execute `next` and poll"; `next_kind: :asset` means the
body is media bytes โ use `LlmCall.fetch/2` for it. The library never polls,
waits, or downloads on its own.
`LlmCall.media_capabilities/3` reports what a codec accepts: `max_n:
:unsupported` means no `n`, `max_references` is the reference count. A knob the
codec cannot express is a validation error, never a silent drop.
## Replaceable JSON layer
HTTP is Finch, one implementation. JSON is a module resolved at client-build
time: the call option, the provider declaration, the compile-time application
default, then Elixir's built-in `JSON`.
| Layer | Module | Default |
| --- | --- | --- |
| JSON | `LlmCall.JSON` facade | Elixir `JSON` |
```elixir
# config/config.exs of the host application
config :llm_call,
json_library: Torque,
capture_raw: :response,
http_opts: [name: MyApp.Finch],
timeout: 30_000
```
`Torque`, `Jason` and `Poison` expose tagged `encode/1`/`decode/1`; Elixir
`JSON` exposes `encode!/1`/`decode!/1`. The `LlmCall.JSON` facade accepts either
shape and normalizes it, so no adapter module is needed. `json_library:` is the
single configuration key, matching Phoenix.
`json_library` and `capture_raw` are read with `Application.compile_env/3`;
changing them requires recompiling the library. Per-client `json_library:`,
`capture_raw:` (through `LlmCall.Config.new/1`) and provider declarations still
override them without touching the compiled default. An explicit
`%LlmCall.Config{}` uses the compiled default for both.
`LlmCall.Application` starts a Finch pool named `LlmCall.Finch`; pass
`http_opts: [name: MyPool]` to use your own. If that name is already taken when
`:llm_call` starts, the existing pool is used and no second one is started.
`http_opts` accepts only `:name`, `:pool_timeout`, `:request_timeout` and
`:pool_strategy`; `timeout:` is the read timeout, and `receive_timeout` in
`http_opts` is rejected instead of ignored.
`timeout:` is Finch's `:receive_timeout`: each body chunk must arrive within it,
not the whole call, so a large response on a slow link can outlive it. Add
`http_opts: [request_timeout: ...]` when you want a wall-clock cap.
## Tools
Tool data is wire data only: `LlmCall.ToolSpec`, `LlmCall.ToolCall` and
`LlmCall.ToolResult`. The library does not derive schemas or dispatch calls โ the
[`tool_call`](https://hex.pm/packages/tool_call) package does that well, so use it
directly and map its provider-neutral descriptions into canonical specs:
```elixir
defmodule MyApp.Tools.Weather do
use ToolCall
@doc """
Get the forecast for a city.
## Parameters
* `city` - city name
"""
@spec get_weather(city :: String.t()) :: {:ok, %{temp: float()}}
def get_weather(city), do: {:ok, %{temp: 21.5}}
end
tools =
for tool <- ToolCall.describe(MyApp.Tools.Weather) do
%LlmCall.ToolSpec{
name: tool.name,
description: tool.description,
parameters: tool.parameters
}
end
request = %{request | tools: tools}
{:ok, response} = LlmCall.complete(client, request)
for call <- LlmCall.Response.tool_calls(response) do
{:ok, result} = ToolCall.run(MyApp.Tools.Weather, String.to_existing_atom(call.name), call.arguments)
# build the %LlmCall.ToolResult{} and send the next request yourself
end
```
Approvals, audits and timeouts stay in `tool_call`, and the follow-up request stays
with you: the library does not run an agent loop.
What a decoded turn can be sent back as depends on the protocol: text and tool
calls always, thinking blocks and `Content.Raw` only where the request shape
allows it. `docs/design.md` ยง8.2 has the table, and
`LlmCall.Codec.Conformance.assert_replays/3` pins it for an adapter.
Provider-hosted tools (web search, grounding, server tools) are builtin tools whose
`name` is the vendor's own identifier; the codec forwards them verbatim when it
declares `builtin_tools: :passthrough`:
```elixir
%LlmCall.ToolSpec{kind: :builtin, name: "web_search_20250305", provider_options: %{"max_uses" => 5}}
```
## Dialect adapters
Four **reference adapters** โ the dialect families the IR is written against:
| Dialect | Module |
| --- | --- |
| OpenAI Chat Completions | `LlmCall.Dialect.OpenAI.Chat` |
| OpenAI Responses | `LlmCall.Dialect.OpenAI.Responses` |
| Anthropic Messages | `LlmCall.Dialect.Anthropic.Messages` |
| Gemini GenerateContent | `LlmCall.Dialect.Gemini.GenerateContent` |
And four **vendored media adapters**, which ship with the library but are not part
of the stable API surface (see `LlmCall.Media`):
| Dialect | Module |
| --- | --- |
| OpenAI Images | `LlmCall.Media.Dialect.OpenAI.Images` |
| Gemini Imagen `predict` | `LlmCall.Media.Dialect.Gemini.Predict` |
| Seedance (BytePlus ModelArk) | `LlmCall.Media.Dialect.Seedance.Video` |
| MiniMax Hailuo | `LlmCall.Media.Dialect.MiniMax.Video` |
The library does not promise provider coverage. An adapter outside this repository
is a module that `use`s `LlmCall.Dialect.Adapter` (or `LlmCall.Media.Dialect.Adapter`) and
supplies `@path`, `@headers`, a private `body/2`, and the protocol's own
`validate_request/2`; share validation through `LlmCall.Dialect.Helpers` instead
of reimplementing it. See [`CONTRIBUTING.md`](CONTRIBUTING.md).
OpenAI-compatible endpoints (Groq, DeepSeek, Together, vLLM, Ollama's OpenAI
surface, ...) use the OpenAI Chat codec with their own base URL and
`provider_options`; their capabilities are not uniform.
## Two levels
The one-call client is the default. The encode/decode functions stay available for
callers that already own a transport:
```elixir
{:ok, encoded} = LlmCall.encode(LlmCall.Dialect.Anthropic.Messages, request)
# transport: encoded.method to base_url <> encoded.path, JSON-encode encoded.body
{:ok, response} = LlmCall.decode_response(LlmCall.Dialect.Anthropic.Messages, payload)
```
## Design
See [`docs/design.md`](docs/design.md) for the canonical IR, the codec contract,
the streaming model, the client, and the failure and compatibility policy.
[`docs/roadmap.md`](docs/roadmap.md) records what 0.1 shipped and the order in which
further operation families (embeddings, rerank, speech, transcription) would arrive.
Adapters check themselves with `LlmCall.Codec.Conformance` or
`LlmCall.Media.Conformance` (test support published from `lib/`, not a runtime
dependency). The vendored media adapters run the media suite in this repository's
own `mix test`.