Packages

iree_tokenizers

0.5.0
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to
Readme
15 Versions
4 Dependencies
Dependants
Files
Activity
iree_tokenizers native iree_tokenizers_native sources tokenizer_sources.txt
Raw

native/iree_tokenizers_native/sources/tokenizer_sources.txt 

vendor/iree_tokenizer_src/iree/tokenizer/decoder.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/byte_fallback.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/byte_level.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/ctc.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/metaspace.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/passthrough.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/replace.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/sequence.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/strip.c

vendor/iree_tokenizer_src/iree/tokenizer/decoder/wordpiece.c

vendor/iree_tokenizer_src/iree/tokenizer/format/huggingface/added_tokens_json.c

vendor/iree_tokenizer_src/iree/tokenizer/format/huggingface/decoder_json.c

vendor/iree_tokenizer_src/iree/tokenizer/format/huggingface/model_json.c

vendor/iree_tokenizer_src/iree/tokenizer/format/huggingface/normalizer_json.c

vendor/iree_tokenizer_src/iree/tokenizer/format/huggingface/postprocessor_json.c

vendor/iree_tokenizer_src/iree/tokenizer/format/huggingface/segmenter_json.c

vendor/iree_tokenizer_src/iree/tokenizer/format/huggingface/tokenizer_json.c

vendor/iree_tokenizer_src/iree/tokenizer/format/tiktoken/tiktoken.c

vendor/iree_tokenizer_src/iree/tokenizer/model.c

vendor/iree_tokenizer_src/iree/tokenizer/model/bpe.c

vendor/iree_tokenizer_src/iree/tokenizer/model/bpe_backtrack.c

vendor/iree_tokenizer_src/iree/tokenizer/model/bpe_cache.c

vendor/iree_tokenizer_src/iree/tokenizer/model/bpe_encode.c

vendor/iree_tokenizer_src/iree/tokenizer/model/bpe_heap.c

vendor/iree_tokenizer_src/iree/tokenizer/model/bpe_tables.c

vendor/iree_tokenizer_src/iree/tokenizer/model/bpe_window.c

vendor/iree_tokenizer_src/iree/tokenizer/model/unigram.c

vendor/iree_tokenizer_src/iree/tokenizer/model/wordpiece.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/bert.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/deferred.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/lowercase.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/nfc.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/nfd.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/nfkd.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/passthrough.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/precompiled.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/prepend.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/regex_replace.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/replace.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/sequence.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/strip.c

vendor/iree_tokenizer_src/iree/tokenizer/normalizer/strip_accents.c

vendor/iree_tokenizer_src/iree/tokenizer/postprocessor.c

vendor/iree_tokenizer_src/iree/tokenizer/regex/compile.c

vendor/iree_tokenizer_src/iree/tokenizer/regex/exec.c

vendor/iree_tokenizer_src/iree/tokenizer/regex/internal/dfa.c

vendor/iree_tokenizer_src/iree/tokenizer/regex/internal/lexer.c

vendor/iree_tokenizer_src/iree/tokenizer/regex/internal/nfa.c

vendor/iree_tokenizer_src/iree/tokenizer/regex/internal/parser.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/bert.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/digits.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/metaspace.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/passthrough.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/punctuation.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/sequence.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/split.c

vendor/iree_tokenizer_src/iree/tokenizer/segmenter/whitespace.c

vendor/iree_tokenizer_src/iree/tokenizer/special_tokens.c

vendor/iree_tokenizer_src/iree/tokenizer/tokenizer.c

vendor/iree_tokenizer_src/iree/tokenizer/vocab/vocab.c

vendor/iree_tokenizer_src/iree/tokenizer/vocab/vocab_builder.c

vendor/iree_tokenizer_src/iree/tokenizer/vocab/vocab_hash.c

vendor/iree_tokenizer_src/iree/tokenizer/vocab/vocab_merge_hash.c

vendor/iree_tokenizer_src/iree/tokenizer/vocab/vocab_trie.c