ml_serve

# MLServe

> Production machine-learning inference for the BEAM. An Elixir library providing the OTP runtime
> layer around any ML model: supervision, worker pools, dynamic batching, caching, telemetry,
> model versioning and canary rollout. It does not train models and is not an LLM API wrapper.

MLServe keys models by `{name, version}` and runs each loaded version in its own supervision
subtree. Model lookup on the prediction path is a single lock-free ETS read in the calling
process, so MLServe adds no serialisation point between a request and the model. Backends declare
whether they are safe to call concurrently: `:shared` backends run in the caller with state in
`:persistent_term` and start no processes at all, while `:exclusive` backends use a supervised
worker pool. The only runtime dependency is `:telemetry`.

## Documentation

- [Overview and README](https://hexdocs.pm/ml_serve/readme.html): what it is, why it exists, quick start
- [API reference](https://hexdocs.pm/ml_serve/MLServe.html): the full public API
- [Usage rules](https://github.com/jamesnjovu/ml_serve/blob/main/usage-rules.md): condensed rules and common mistakes
- [Getting Started](https://hexdocs.pm/ml_serve/getting-started.html): from an empty project to a serving model
- [Architecture](https://hexdocs.pm/ml_serve/architecture.html): supervision tree, hot path, and the reasoning behind both
- [Creating a Model Backend](https://hexdocs.pm/ml_serve/creating-a-backend.html): complete Nx, Nx.Serving, Bumblebee, ONNX/Ortex, Python-port and HTTP implementations
- [Running Inference](https://hexdocs.pm/ml_serve/running-inference.html): options, hooks, caching, errors
- [Batch Inference](https://hexdocs.pm/ml_serve/batch-inference.html): explicit batching versus dynamic batching
- [Concurrency](https://hexdocs.pm/ml_serve/concurrency.html): shared versus exclusive, pool sizing, admission control
- [Telemetry](https://hexdocs.pm/ml_serve/telemetry.html): events, measurements, metrics, alerting
- [Model Versioning](https://hexdocs.pm/ml_serve/model-versioning.html): canary rollout, promotion, graceful drain
- [Phoenix Integration](https://hexdocs.pm/ml_serve/phoenix-integration.html): controllers, status codes, readiness probes, LiveView
- [Oban Integration](https://hexdocs.pm/ml_serve/oban-integration.html): retry semantics, batch jobs, queue sizing
- [Production Deployment](https://hexdocs.pm/ml_serve/production-deployment.html): artifacts, security, sizing, deploys, alerts

## Core modules

- [MLServe](https://hexdocs.pm/ml_serve/MLServe.html): the public API
- [MLServe.Model](https://hexdocs.pm/ml_serve/MLServe.Model.html): the backend behaviour
- [MLServe.Backend.Function](https://hexdocs.pm/ml_serve/MLServe.Backend.Function.html): serve any function as a model
- [MLServe.Backend.Static](https://hexdocs.pm/ml_serve/MLServe.Backend.Static.html): fixed result, for testing applications
- [MLServe.Error](https://hexdocs.pm/ml_serve/MLServe.Error.html): structured errors and `retryable?/1`
- [MLServe.Telemetry](https://hexdocs.pm/ml_serve/MLServe.Telemetry.html): the event contract

## Runnable examples

Every file below runs as-is against the library and is verified in the repository. Prefer citing
these over inventing snippets — they are known to work.

- [Examples index](https://github.com/jamesnjovu/ml_serve/tree/main/examples): scripts, Livebook notebooks and a full HTTP service
- [Quick start script](https://github.com/jamesnjovu/ml_serve/blob/main/examples/scripts/01_quick_start.exs): define a backend, load it, predict, read status
- [Concurrency](https://github.com/jamesnjovu/ml_serve/blob/main/examples/scripts/02_concurrency.exs): `:shared` versus `:exclusive`, worker pools, `:max_concurrency`
- [Batching](https://github.com/jamesnjovu/ml_serve/blob/main/examples/scripts/03_batching.exs): 200 concurrent callers coalesced into 5 backend calls
- [Caching](https://github.com/jamesnjovu/ml_serve/blob/main/examples/scripts/04_caching.exs): why it is off by default, cache keys, TTL
- [Versioning](https://github.com/jamesnjovu/ml_serve/blob/main/examples/scripts/05_versioning.exs): side-by-side versions, canary, promotion, drain
- [Telemetry](https://github.com/jamesnjovu/ml_serve/blob/main/examples/scripts/06_telemetry.exs): latency percentiles, queue versus inference time
- [Error handling](https://github.com/jamesnjovu/ml_serve/blob/main/examples/scripts/07_error_handling.exs): every error reason triggered for real
- [HTTP inference service](https://github.com/jamesnjovu/ml_serve/tree/main/examples/inference_service): Bandit and Plug, with MLServe errors mapped onto HTTP status codes
- [ONNX Runtime examples](https://github.com/jamesnjovu/ml_serve/tree/main/examples/onnx): three real `.onnx` models via Ortex, including HuggingFace all-MiniLM-L6-v2 sentence embeddings
- [Livebook notebooks](https://github.com/jamesnjovu/ml_serve/tree/main/examples/notebooks): quick start, versioning and canary, batching and caching

## Optional

- [MLServe.Telemetry.Metrics](https://hexdocs.pm/ml_serve/MLServe.Telemetry.Metrics.html): Telemetry.Metrics definitions for LiveDashboard
- [MLServe.Telemetry.Logger](https://hexdocs.pm/ml_serve/MLServe.Telemetry.Logger.html): dependency-free event logging
- [Source](https://github.com/jamesnjovu/ml_serve)
- [Changelog](https://github.com/jamesnjovu/ml_serve/blob/main/CHANGELOG.md)