Current section
Files
Jump to
Current section
Files
guides/reference/configuration.cheatmd
# Configuration
Library mode uses ordinary Elixir application configuration. Standalone mode reads non-secret runtime and routing data from strict TOML, while environment variables are reserved for secrets and the TOML file location.
## Application Options
{: .col-2}
### Repo
```elixir
config :llm_proxy, repo: MyApp.Repo
```
Ecto repo used by the default storage adapter. Library hosts normally provide an already supervised repo. Standalone production uses `LLMProxy.Storage.Repo.QuackDB`.
### Storage Adapter
```elixir
config :llm_proxy, storage: MyApp.LLMProxyStorage
```
Module implementing `LLMProxy.Storage.Adapter`. Default: `LLMProxy.Storage.Ecto`.
### HTTP Enabled
```elixir
config :llm_proxy, http_enabled: false
```
Whether LLMProxy starts its own loopback Cowboy listener. Default: `true`. Disable in library mode unless a separate listener is intentional.
### HTTP Port
```elixir
config :llm_proxy, http: [port: 4000]
```
Port for the bundled listener. The listener binds `127.0.0.1`.
### Master Key
```elixir
config :llm_proxy, master_key: System.fetch_env!("LLM_PROXY_MASTER_KEY")
```
Bootstrap/operator credential. It bypasses ordinary model and quota checks.
### RPC Socket
```elixir
config :llm_proxy, rpc_socket: "/run/llm-proxy/rpc.sock"
```
Starts the [SafeRPC](https://hexdocs.pm/safe_rpc) server for chat, operations, and optional Incant admin calls.
### Public URL
```elixir
config :llm_proxy, public_url: "https://llm.example.com"
```
Public base URL used by setup helpers and provider headers.
### Request Body Limit
```elixir
config :llm_proxy, body_limit_bytes: 32_000_000
```
Maximum authenticated JSON request body. Must be a positive integer. Default: 32 MB.
### Remote Timeout
```elixir
config :llm_proxy, remote_timeout_ms: 30_000
```
Default SafeRPC request timeout. Default: 30 seconds.
### Provider Receive Timeout
```elixir
config :llm_proxy, provider_receive_timeout_ms: :timer.minutes(10)
```
Provider transport timeout and standalone Cowboy idle ceiling. Default: 10 minutes.
### Provider Cooldowns
```elixir
config :llm_proxy, providers: %{
"openrouter" => %{rate_limit_cooldown_ms: 30_000},
"openai-codex" => %{
rate_limit_cooldown_ms: 30_000,
quota_cooldown_ms: :timer.hours(4)
}
}
```
Equivalent standalone TOML:
```toml
[providers.openrouter]
rate_limit_cooldown_ms = 30000
[providers.openai-codex]
rate_limit_cooldown_ms = 30000
quota_cooldown_ms = 14400000
```
Both values are fallback durations, in milliseconds: positive integers no greater than 31 days.
`rate_limit_cooldown_ms` defaults to 30 seconds for ordinary throttling. `quota_cooldown_ms`
defaults to four hours and applies only when an adapter explicitly identifies subscription quota
exhaustion. The Codex adapter recognizes the structured `usage_limit_reached` error; a bare 429
is not sufficient. Neither OAuth authentication nor HTTP status alone implies exhausted quota.
Valid upstream `Retry-After` values (seconds or HTTP dates) take precedence over fallback durations;
Codex quota-reset timestamps take precedence for subscription exhaustion. A zero or elapsed retry
hint creates no new cooldown. Invalid or out-of-range hints fall back to configured durations.
Ordinary throttling is model-scoped when a model is known; Codex subscription exhaustion is
account-scoped. Configuration uses the named provider instance, even when routes share a token pool.
These settings do not change route circuit-breaker `cooldown_ms`, replay safety, or attempt budgets.
The old `config :llm_proxy, token_cooldown_ms: ...` is deprecated. An explicit value logs a warning
when used and supplies only the quota fallback if the provider has no `quota_cooldown_ms` override.
It no longer controls ordinary API throttling. Calling `TokenPool.Server.mark_rate_limited/1`
without a duration now uses the provider's short rate-limit fallback; subscription adapters should
classify quota errors explicitly rather than relying on that helper's former four-hour default.
Cooldown writes belong to provider execution and stream-failure handling, not HTTP rendering.
`Result.retry_after_ms` carries the selected delay and pre-stream HTTP errors include `Retry-After`.
Adapter error-conversion functions return metadata without persisting a cooldown; callers that use
adapters directly should use `Provider`/`Execution` or the shared `RateLimit.record/1` boundary.
No streams are replayed after output has started.
Existing cooldown rows are preserved across upgrades. Audit any old, long API cooldowns separately;
do not bulk-delete subscription quota blocks. This change does not add live provider-health probes.
### Provider Token Selection
```elixir
config :llm_proxy, token_selection_strategy: :fill_first
```
`:affinity` is the default and preserves stable user-to-token affinity.
`:fill_first` selects the highest-priority healthy token within the preferred
credential kind, breaking ties by token ID. OAuth remains preferred over API
keys. Standalone TOML:
```toml
[provider_tokens]
selection_strategy = "fill_first"
```
This non-secret setting is not read from the environment. Provider-token
priorities are non-negative integers and can be edited through Incant's
priority-only form. Set an initial priority when adding a token:
```elixir
LLMProxy.Storage.add_token("openai-codex", "oauth", access_token, %{priority: 100})
```
### Provider Token Codec
```elixir
config :llm_proxy,
provider_token_codec: {
LLMProxy.Provider.TokenCodec.AESGCM,
active_key_id: "2026-08",
keys: %{
"2026-08" => System.fetch_env!("PROVIDER_TOKEN_KEY_2026_08"),
"2026-02" => System.fetch_env!("PROVIDER_TOKEN_KEY_2026_02")
},
allow_plaintext: true
}
```
Codec for provider API keys and OAuth tokens. AES-GCM keys must decode to 32
bytes. This keyring is separate from `:master_key`. Writes use only the active
key ID. Reads accept all key IDs in the keyring. The default plaintext codec is
for compatibility and does not protect stored credentials.
### Deployment Failure Threshold
```elixir
config :llm_proxy, deployment_failure_threshold: 3
```
Default consecutive retryable failures before opening a deployment circuit. Default: 3.
### Deployment Cooldown
```elixir
config :llm_proxy, deployment_cooldown_ms: 30_000
```
Default open-circuit cooldown. Default: 30 seconds.
### Max Retries
```elixir
config :llm_proxy, max_retries: 1
```
Compatibility retry/fallback limit for configured fallback chains. Default: 1. The strict provider-dispatch budget is `max_retries + 1`; open-circuit and unsupported-protocol route skips do not consume it.
Standalone TOML:
```toml
[routing]
max_retries = 1
```
### Replay Policy
```elixir
config :llm_proxy, replay_policy: :safe_only
```
`:safe_only` permits fallback only when LLMProxy has evidence that the upstream did not accept work, such as unavailable credentials, a clear connection failure, or a 429 refusal. This is the default.
`:allow_uncertain` also permits replay after timeouts and 5xx responses. Use it only when duplicate cost and side effects are acceptable. It restores the earlier fallback behavior.
Standalone TOML:
```toml
[routing]
replay_policy = "safe_only"
```
### Fallbacks
```elixir
config :llm_proxy,
fallbacks: %{"primary" => ["secondary"]}
```
Compatibility map for model fallback chains. Catalog routes are preferred for new configurations.
### Cache Adapter
```elixir
config :llm_proxy, cache: MyApp.LLMCache
```
Module implementing `LLMProxy.Cache`. Unset by default.
### Cache Policy
```elixir
config :llm_proxy,
cache_policy: [
enabled: true,
ttl_ms: 60_000,
models: %{"fresh" => [enabled: false]}
]
```
Default and per-model deterministic cache policy.
### Guardrails
```elixir
config :llm_proxy, guardrails: [MyApp.RequestPolicy]
```
Ordered modules implementing `LLMProxy.Guardrail`.
## Provider Configuration
{: .col-2}
### Public Model Allowlist
```elixir
config :llm_proxy, public_models: ["fast", "codex"]
```
Filters model discovery and request admission. Unset by default, which exposes all registered models for compatibility. In standalone mode, configure visible catalog aliases in TOML; direct provider model IDs are not exposed by the allowlist:
```toml
[catalog]
public_models = ["fast", "codex"]
```
An explicit empty list exposes no models. This non-secret setting is not read from the environment.
### Built-in Provider Keys
```elixir
config :llm_proxy,
providers: %{
"openai" => %{api_keys: "sk-a,sk-b"},
"anthropic" => %{api_keys: "sk-ant-..."},
"openrouter" => %{api_keys: "sk-or-..."},
"openai-codex" => %{oauth_tokens: "access|refresh|expires_ms|account"}
}
```
Runtime bootstrap values are persisted into provider-token storage.
### Named Provider
```elixir
config :llm_proxy,
providers: %{
"example-service" => %{
adapter: "openai",
base_url: "https://api.example.com/v1",
token_pool: "example-production"
}
}
```
Uses an existing ReqLLM adapter without a custom provider module.
### Provider Base URL
```elixir
%{base_url: "https://api.example.com/v1"}
```
Default endpoint for a named or built-in provider.
### Provider Token Pool
```elixir
%{token_pool: "example-production"}
```
Default credential pool for provider routes.
### GLM Usage Source
```elixir
%{
adapter: "zai_coding_plan",
base_url: "https://api.z.ai/api/coding/paas/v4",
token_pool: "glm-production",
usage_adapter: "glm",
usage_auth_scheme: "raw",
usage_paths: ["/api/monitor/usage/quota/limit", "/api/monitor/usage"]
}
```
`usage_adapter` is optional for `zai`, `zai_coder`, and `zai_coding_plan`; when present it must be the string `"glm"`. `usage_auth_scheme` must be `"raw"` or `"bearer"`. `usage_paths` must be a non-empty list of at most three distinct absolute origin paths. Raw whitespace and control bytes are rejected; encode valid path characters when needed. Invalid values fail configuration loading rather than being coerced or ignored.
### Anthropic Defaults
```elixir
%{
api_version: "2023-06-01",
beta: "feature-a,feature-b",
conversion_defaults: %{max_tokens: 4096}
}
```
Provider-specific headers and request-conversion defaults.
## Model Configuration
{: .col-2}
### Single Route
```elixir
config :llm_proxy,
models: [
fast: [route: [to: :openai, model: "gpt-4.1-mini"]]
]
```
Public alias `fast` targets one upstream deployment.
### Multiple Routes
```elixir
config :llm_proxy,
models: [
fast: [
routing: :ordered,
routes: [
[to: :openai, model: "gpt-4.1-mini"],
[to: :anthropic, model: "claude-3-5-haiku-20241022", order: 2]
]
]
]
```
### Routing
```elixir
routing: :round_robin
```
`:ordered`, `:shuffle`, `:round_robin`, `:weighted_shuffle`, `:lowest_cost`, or
`:latency_aware`.
### Route Timeout
```elixir
timeout: 30_000
```
Per-attempt timeout in milliseconds. `timeout_ms` is also accepted.
### Route Circuit Breaker
```elixir
failure_threshold: 3,
cooldown_ms: 30_000
```
Overrides deployment defaults.
### Route Order
```elixir
order: 2
```
Fallback group. Lower groups are attempted first.
### Route Weight
```elixir
weight: 3
```
Relative weight for `:weighted_shuffle`.
### Route Token Pool
```elixir
token_pool: "special-production"
```
Overrides the provider's default pool.
## Standalone Environment
{: .col-2}
### Master Key
```text
MASTER_KEY=...
```
Bootstrap/operator credential.
### Provider API-Key Pools
```text
LLM_PROXY_PROVIDER_KEYS={"example-production":["secret"]}
```
Secret JSON object mapping token-pool names to API-key arrays. Use pool names
such as `openai`, `anthropic`, or `openrouter` for built-in providers. Codex OAuth
credentials are provisioned through the admin login flow and persisted in
provider-token storage.
### Provider Token Keyring
```text
LLM_PROXY_PROVIDER_TOKEN_KEYRING={"active_key_id":"2026-08","keys":{"2026-08":"base64-encoded-32-byte-key"}}
```
Secret JSON keyring for standalone provider-token encryption. Keys must decode
to 32 bytes. Keep this keyring separate from `MASTER_KEY`; writes use the active
key ID and reads accept every retained key.
### TOML Path
```text
LLM_PROXY_CONFIG_TOML=/etc/llm-proxy/config.toml
```
Standalone TOML data file. Default: `/etc/llm-proxy/config.toml`.
## Standalone TOML
{: .col-2}
### Server
```toml
[server]
port = 4000
public_url = "https://llm.example.com"
body_limit_bytes = 32000000
rpc_socket = "/run/llm-proxy/rpc.sock"
```
The listener remains bound to loopback. Omit `rpc_socket` to disable SafeRPC.
### Storage
```toml
[storage]
database = "/var/lib/llm-proxy/llm_proxy.duckdb"
quackdb_uri = "http://127.0.0.1:9494"
quackdb_endpoint = "quack:localhost:9494"
```
### Routing Runtime
```toml
[routing]
max_retries = 1
replay_policy = "safe_only"
provider_connect_timeout_ms = 10000
token_selection_strategy = "affinity"
```
Fallback deployments belong in model routes, not a separate fallback map.
### Provider Usage
```toml
[provider_usage]
auto_refresh = true
refresh_interval_ms = 300000
request_timeout_ms = 10000
stale_after_ms = 600000
```
The refresh interval must be from `60000` through `3600000`. The request timeout
must be from `1000` through `30000`. The stale time must be at least the refresh
interval and no more than `86400000`. Omitted values use the built-in defaults.
### Telemetry
```toml
[telemetry]
otlp_endpoint = "http://127.0.0.1:4318"
```
Omit the endpoint to keep trace export disabled.
### Provider Token Rollout
```toml
[provider_tokens]
allow_plaintext = true
```
Keep plaintext compatibility enabled only during an explicit encryption rollout.
After migration and verification, set it to `false`; a missing keyring then fails
startup.
### Provider
```toml
[providers.example-service]
adapter = "openai"
base_url = "https://api.example.com/v1"
token_pool = "example-production"
```
Optional provider usage keys are `usage_adapter`, `usage_auth_scheme`, and `usage_paths`.
### Model
```toml
[[models]]
name = "example/model"
routing = "ordered"
```
### Route
```toml
[[models.routes]]
to = "example-service"
model = "upstream-model"
timeout = 30000
failure_threshold = 3
cooldown_ms = 30000
order = 1
weight = 1
```
TOML contains data only. Unknown keys are rejected; arbitrary atoms are not created.