whisper_cpp

# whisper_cpp

A thin Elixir wrapper around [`whisper-rs`](https://codeberg.org/tazz4843/whisper-rs),
the Rust bindings to [whisper.cpp](https://github.com/ggerganov/whisper.cpp).
It exposes whisper.cpp speech-to-text to the BEAM through a Rustler NIF: load a
model, hand it 16 kHz mono f32 PCM, get structured segments back. No subprocess,
no Python, no temporary files.

## Installation

```elixir
def deps do
  [{:whisper_cpp, "~> 0.5.0"}]
end
```

Installation downloads a precompiled NIF for your target from the project's
GitHub releases - no Rust toolchain needed. Requires Elixir 1.19+.

## Usage

```elixir
{:ok, model} = WhisperCpp.load_model("models/ggml-large-v3.bin")

# Decode upstream (ffmpeg, bumblebee, ...) into 16 kHz mono f32 PCM:
#   ffmpeg -i jfk.wav -f f32le -ac 1 -ar 16000 jfk.pcm
pcm = File.read!("jfk.pcm")

{:ok, %WhisperCpp.Transcription{text: text, segments: segs}} =
  WhisperCpp.transcribe(model, {:pcm_f32, pcm}, language: "en")

IO.puts(text)
for s <- segs, do: IO.puts("[#{s.start}-#{s.end}] #{s.text}")
```

Audio is always `{:pcm_f32, binary}` - little-endian f32 samples, mono, 16 kHz,
normalised to `[-1.0, 1.0]`. The library does **not** decode WAV/MP3/etc;
decode upstream. `transcribe_slice/4` runs a `[start_s, end_s)` window of a
master PCM buffer and shifts the returned times back into the source timeline.

Built-in silero voice activity detection strips silence before the encoder:
pass `vad_model_path:` (`ggml-silero-v5.1.2.bin`, ~0.85 MB, from
[ggml-org/whisper-vad](https://huggingface.co/ggml-org/whisper-vad)) and
timestamps stay on the original timeline.

See [the docs](https://hexdocs.pm/whisper_cpp) for the full option list
(`:translate`, `:initial_prompt`, `:word_timestamps`, `:beam_size`,
`:n_threads`, VAD tuning, cancellation, progress messages, ...) and error
handling.

## Backends

CPU is available in every build except `coreml` (see below). Pick one
accelerator per build; the precompiled Hex package ships CPU plus `cuda` /
`hipblas` variants for Linux and Metal on Apple Silicon, selected via
`WHISPER_CPP_VARIANT`:

```bash
WHISPER_CPP_VARIANT=cuda mix deps.compile whisper_cpp
```

The precompiled NIFs need this CPU baseline:

| Target                      | Minimum CPU                                                         |
| --------------------------- | ------------------------------------------------------------------- |
| `x86_64-unknown-linux-gnu`  | AVX2, FMA, F16C, BMI2 (Intel Haswell, AMD Zen, or newer)            |
| `aarch64-unknown-linux-gnu` | ARMv8.2-A with dotprod and fp16 (Neoverse N1, Cortex-A76, or newer) |
| `aarch64-apple-darwin`      | Apple M1 or newer                                                   |

The `cuda` and `hipblas` variants need the same CPU. On an older CPU, build
from source with `WHISPER_CPP_BUILD=1`. A source build tunes ggml for the CPU
it runs on.

This baseline applies from 0.5.0 on. Releases up to 0.4.1 were tuned
for the CPU of the release runner. Their `aarch64-unknown-linux-gnu` artefacts
need SVE and i8mm, and the 0.3.1 `x86_64-unknown-linux-gnu` artefact needs
AVX-512. On a CPU without these, build those versions from source.

To build from source with any whisper-rs backend (`cuda`, `hipblas`, `vulkan`,
`metal`, `coreml`, `intel-sycl`, `openblas`, `openmp`):

```bash
WHISPER_CPP_BUILD=1 WHISPER_CPP_FEATURES=cuda mix deps.compile whisper_cpp
```

Source builds need Rust 1.98 or later, `cmake`, a C++17 compiler, and the
backend's own SDK (CUDA toolkit, ROCm, Vulkan SDK, ...).

A `coreml` build uses the Core ML encoder whenever the model's
`-encoder.mlmodelc` is present and cannot turn it off per model. It rejects
`device: :cpu` and `use_gpu: false` with `:invalid_request`; build without
`coreml` for CPU-only inference.

## Testing

```bash
mix test                  # unit tests, no downloads
mix test --include integration  # downloads ggml-tiny.en + ggml-tiny, real inference
```

## License

MIT. whisper.cpp is MIT-licensed; `whisper-rs` is public domain (Unlicense)
and vendors whisper.cpp, linking it statically.