Packages

iree_tokenizers

0.8.1
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to
Readme
15 Versions
4 Dependencies
Dependants
Files
Activity
iree_tokenizers lib iree tokenizers.ex
Raw

lib/iree/tokenizers.ex 

defmodule IREE.Tokenizers do

  @moduledoc """

  Fast Hugging Face `tokenizer.json`, OpenAI `.tiktoken`, and SentencePiece

  `.model` bindings for Elixir backed by the IREE tokenizer runtime.



  The main entrypoint is `IREE.Tokenizers.Tokenizer`.



  Supported load formats:



  - Hugging Face `tokenizer.json`

  - OpenAI `.tiktoken`

  - SentencePiece `.model`



  Supported runtime capabilities:



  - one-shot encode/decode

  - batched encode/decode

  - streaming encode/decode

  - token offsets and type IDs

  - vocabulary lookup helpers



  The library is intentionally inference-focused. Pair-sequence encoding,

  tokenizer training, and full mutation parity with `elixir-nx/tokenizers`

  are not yet complete.

  """

end