llm_proxy

# Changelog

## Unreleased

## 0.1.1 - 2026-08-01

### Fixed

- Streaming endpoints now wait for the first upstream event before committing HTTP 200, return immediate lazy-stream failures as HTTP errors, and render sanitized protocol error events when providers fail after streaming begins.
- OpenAI Codex stream failures now preserve safe upstream reasons, while trace-correlated diagnostics distinguish upstream failures from local storage errors without exposing request, query, or credential data.
- Token counters now use 64-bit database columns, and accounting exceptions are reported internally without turning a completed model stream into a client failure.
- OpenAI-compatible provider failures now return one normalized error object, forwarding structured upstream fields without duplicate `details.error` wrappers or inspected Elixir terms.
- Public HTTP and SSE errors now use protocol-native OpenAI or Anthropic envelopes across authentication, quotas, JSON parsing, draining, guardrails, moderation, routing, and provider failures; only bounded safe fields reach clients.
- Release deployment drains now work from clean release-eval VMs, advertise their bounded SafeRPC atom vocabulary, and restore request acceptance when a drain deadline expires.
- OpenAI Codex WebSockets now use a finite connection deadline with no default established-stream receive deadline, and timeout or handshake failures retain safe phase/status diagnostics.
- Named tool choices and function tool definitions now normalize across OpenAI Chat, OpenAI Responses, and Anthropic Messages routing boundaries.
- ReqLLM now uses the released Hex package at v1.18, replacing the temporary Git pin; Cowboy/Cowlib were updated to versions that resolve the newly published memory-exhaustion advisories.

### Security

- Require SafeRPC 0.1.15 or later for bounded frames, strict request validation, executable ETF rejection, and isolated listener failures.

## 0.1.0 - 2026-07-22

### Added

- In-process execution through `LLMProxy.Provider` and `LLMProxy.chat/2`.
- ReqLLM provider registration as `:llm_proxy`, including remote BEAM calls.
- Model catalog with aliases, ordered or shuffled deployments, per-deployment
  timeouts, circuit breakers, retries, and fallback across providers and models.
- Direct providers for OpenAI, Anthropic, OpenRouter, OpenAI Codex, and Kimi
  Code, plus an OpenAI-compatible provider helper for custom upstreams.
- Reasoning effort levels forwarded to models that support them.
- OpenAI Chat Completions (`/v1/chat/completions`) with streaming.
- Anthropic Messages (`/v1/messages`) with streaming.
- OpenAI Responses (`/v1/responses`) with streaming.
- OpenAI Moderations (`/v1/moderations`).
- Streaming heartbeats during upstream silence, bounded connection capacity,
  and terminal-failure classification.
- Configurable request body limits with authentication before body parsing.
- API key management with per-key token, message, and cache quotas plus
  composable budget limits.
- Provider token pools with stable user pinning and `Retry-After` cooldowns.
- Usage tracking for input, output, and cache tokens plus estimated USD cost.
- Request metadata and tags for cost attribution.
- Trace logging with request and response bodies, latency, and a feedback API.
- Guardrail hooks for request, response, and stream policy without a bundled
  policy engine.
- Deterministic cache hooks for pluggable non-stream response caching.
- Embeddable storage migrations with SQLite and DuckDB (QuackDB) adapters.
- Optional Incant admin surfaces for API keys, provider tokens, traces, and
  messages, plus an operations dashboard.
- OpenTelemetry instrumentation for HTTP, Ecto, and Req.
- Drain support for graceful deployments.

### Compatibility

- Requires Elixir 1.17 or later.
- Incant integration is optional and supports Incant 0.1.x.