Packages

llama_cpp_ex

0.4.3
0.8.36 0.8.35 0.8.34 0.8.33 0.8.32 0.8.31 0.8.28 0.8.27 0.8.26 0.8.25 0.8.24 0.8.23 0.8.22 0.8.21 0.8.20 0.8.19 0.8.18 0.8.17 0.8.16 0.8.15 0.8.14 0.8.13 0.8.12 0.8.11 0.8.10 0.8.9 0.8.8 0.8.7 0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.9 0.7.8 0.7.7 0.7.6 0.7.5 0.7.4 0.7.3 0.7.2 0.7.0 0.6.14 0.6.13 0.6.12 0.6.11 0.6.10 0.6.9 0.6.8 0.6.7 0.6.6 0.6.5 0.6.4 0.6.3 0.6.1 0.6.0 0.5.0 0.4.4 0.4.3 0.4.2 0.4.1 0.3.0 0.2.0

Elixir bindings for llama.cpp — run LLMs locally with Metal, CUDA, Vulkan, or CPU acceleration.

HexDocs

Current section

Files

Jump to
Readme
65 Versions
3 Dependencies
Dependants
Files
Activity
llama_cpp_ex CHANGELOG.md
Raw

CHANGELOG.md 

# Changelog



## v0.4.3



### Changed



- **llama.cpp submodule** — Updated to latest upstream (b8185).

  - Vulkan: improve partial offloading performance on AMD

  - CUDA: cap grid.y at 65535 in non-contiguous dequantize/convert kernels

  - ggml-cpu: optimise s390x multiply extend instructions

  - Vendors: update cpp-httplib to 0.35.0, miniaudio to 0.11.24



## v0.4.2



### Changed



- **llama.cpp submodule** — Updated to latest upstream (b8179).



## v0.4.1



### Improved



- **Error handling** — `Chat.apply_template/3`, `Tokenizer.encode/3`, and `Tokenizer.decode/2` now return `{:error, reason}` instead of crashing when NIFs raise.

- **Telemetry documentation** — Server moduledoc documents all telemetry events, measurements, and metadata.

- **Typespecs** — Added `@spec` to `Server.start_link/1`.



### Changed



- **llama.cpp submodule** — Updated to latest upstream (b8157).



## v0.4.0



### Added



- **Full model loading params** — `main_gpu`, `split_mode`, `tensor_split` for multi-GPU placement; `use_mlock` and `use_direct_io` for memory control; `vocab_only` for cheap model introspection without loading weights.

- **Server GPU forwarding** — `Server.start_link/1` now forwards `main_gpu`, `split_mode`, `tensor_split`, `use_mlock`, and `use_direct_io` to `Model.load/2`.



## v0.3.0



### Added



- **Jinja chat templates** — switched from `llama_chat_apply_template()` C API to the full Jinja-based `common_chat_templates_apply()` engine from llama.cpp's common library.

- **`enable_thinking` option** — pass `enable_thinking: false` to `Chat.apply_template/3`, `chat/3`, `stream_chat/3`, `chat_completion/3`, and `stream_chat_completion/3` to disable CoT reasoning for models like Qwen3/3.5.

- **`chat_template_kwargs` option** — pass arbitrary key-value pairs to the Jinja template engine.

- **Penalty parameters** — `penalty_repeat`, `penalty_freq`, and `penalty_present` options for repetition/frequency/presence penalties in sampling.

- **OpenAI-compatible response format** — `chat_completion/3` and `stream_chat_completion/3` return `ChatCompletion` and `ChatCompletionChunk` structs.

- **Qwen3.5 benchmark results** in README — Qwen3.5-27B and Qwen3.5-35B-A3B on Apple M4 Max.



### Changed



- `Chat.apply_template/3` now uses the Jinja engine and takes the model ref directly (no longer accepts `:template` option for raw template strings).

- Linked `libcommon.a` from llama.cpp build (previously excluded).

- `LlamaModel` RAII wrapper now caches `common_chat_templates` at model load time.



## v0.2.0



### Added



- **Continuous batching server** (`LlamaCppEx.Server`) — GenServer with slot pool for concurrent multi-sequence inference. One forward pass per tick with decode tokens and prefill chunks mixed in a single batch.

- **Embeddings** (`LlamaCppEx.Embedding`) — `embed/3` and `embed_batch/3` with L2 normalization and configurable pooling type.

- **Grammar-constrained generation** — GBNF grammar support via `grammar` and `grammar_root` options in `Sampler.create/2` and `generate/3`.

- **Batched inference primitives** — `prefill/3`, `decode_batch/3`, `decode_token/4`, `batch_eval/2`, `sampler_sample_at/3` NIFs for building custom inference loops.

- **Streaming via Server** — `LlamaCppEx.Server.stream/3` for token-by-token streaming through the batched server.

- **Telemetry events** — `[:llama_cpp_ex, :server, :tick]` and `[:llama_cpp_ex, :server, :request, :done]` for observability.

- **Benchmark suite** (`bench/`) — Benchee-based benchmarks for single-sequence and server generation, plus a custom continuous batching harness measuring throughput scaling.



### Changed



- `Sampler.create/1` now requires the model as the first argument: `Sampler.create(model, opts)`.

- `Context.create/2` accepts new options: `:embeddings`, `:pooling_type`, `:n_seq_max`.



## v0.1.0



Initial release.



- Model loading and introspection

- Text generation with configurable sampling

- Streaming token generation via `Stream.resource/3`

- Chat template support

- Tokenization and detokenization

- Metal, CUDA, Vulkan, and CPU backends

- RAII resource management via `fine`