Packages

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

Current section

Files

Jump to
iree_tokenizers test fixtures test_sentencepiece.model
Raw

test/fixtures/test_sentencepiece.model

File is too large to be displayed (0.3 MB).