Packages
llama_cpp_ex
0.8.1
0.8.36
0.8.35
0.8.34
0.8.33
0.8.32
0.8.31
0.8.28
0.8.27
0.8.26
0.8.25
0.8.24
0.8.23
0.8.22
0.8.21
0.8.20
0.8.19
0.8.18
0.8.17
0.8.16
0.8.15
0.8.14
0.8.13
0.8.12
0.8.11
0.8.10
0.8.9
0.8.8
0.8.7
0.8.6
0.8.5
0.8.4
0.8.3
0.8.2
0.8.1
0.8.0
0.7.9
0.7.8
0.7.7
0.7.6
0.7.5
0.7.4
0.7.3
0.7.2
0.7.0
0.6.14
0.6.13
0.6.12
0.6.11
0.6.10
0.6.9
0.6.8
0.6.7
0.6.6
0.6.5
0.6.4
0.6.3
0.6.1
0.6.0
0.5.0
0.4.4
0.4.3
0.4.2
0.4.1
0.3.0
0.2.0
Elixir bindings for llama.cpp — run LLMs locally with Metal, CUDA, Vulkan, or CPU acceleration.
Current section
Files
Jump to
Current section
Files
lib/llama_cpp_ex/server/batch_strategy.ex
defmodule LlamaCppEx.Server.BatchStrategy do
@moduledoc """
Behavior for batch building strategies.
A strategy decides how to allocate the token budget between
decode tokens (generation) and prefill chunks each tick.
## Built-in Strategies
* `LlamaCppEx.Server.Strategy.DecodeMaximal` - Decode tokens first, prefill fills
remaining budget. Best for interactive use (lowest generation latency). **Default.**
* `LlamaCppEx.Server.Strategy.PrefillPriority` - Prefill chunks first, decode fills
remaining budget. Best for batch processing (highest throughput).
* `LlamaCppEx.Server.Strategy.Balanced` - Equal budget split between decode and
prefill. Fair under mixed workloads.
## Custom Strategies
Implement the `c:build_batch/4` callback:
defmodule MyStrategy do
@behaviour LlamaCppEx.Server.BatchStrategy
@impl true
def build_batch(slots, budget, chunk_size, opts) do
# Return {entries, updated_slots}
end
end
"""
@type entry ::
{token_id :: integer(), pos :: integer(), seq_id :: integer(), logits :: boolean()}
@doc """
Build a batch of entries from the current slot state.
Returns `{entries, updated_slots}` where entries is a list of
`{token_id, pos, seq_id, logits}` tuples in forward order (will be
reversed by the caller).
## Parameters
* `slots` - Map of seq_id to slot state maps.
* `budget` - Maximum tokens allowed in this batch (`n_batch`).
* `chunk_size` - Maximum prefill tokens per slot per tick.
* `opts` - Additional context:
* `:queue_depth` - Number of requests waiting for a slot.
* `:model_ref` - Model reference for detokenization.
"""
@callback build_batch(
slots :: %{non_neg_integer() => map()},
budget :: pos_integer(),
chunk_size :: pos_integer(),
opts :: keyword()
) :: {entries :: [entry()], updated_slots :: %{non_neg_integer() => map()}}
end