Packages

iree_tokenizers

0.8.6
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to
Readme
15 Versions
4 Dependencies
Dependants
Files
Activity
iree_tokenizers native iree_tokenizers_native src lib.rs
Raw

native/iree_tokenizers_native/src/lib.rs 

mod error;

mod ffi;

mod sentencepiece;

mod stream;

mod tokenizer;



use rustler::{Env, Term};



fn on_load(_env: Env, _info: Term) -> bool {

    true

}



rustler::init!("Elixir.IREE.Tokenizers.Native", load = on_load);