Packages

iree_tokenizers

0.8.2
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to
Readme
15 Versions
4 Dependencies
Dependants
Files
Activity
iree_tokenizers native iree_tokenizers_native sources base_sources.txt
Raw

native/iree_tokenizers_native/sources/base_sources.txt 

vendor/iree_tokenizer_src/iree/base/allocator_libc.c

vendor/iree_tokenizer_src/iree/base/allocator.c

vendor/iree_tokenizer_src/iree/base/bitfield.c

vendor/iree_tokenizer_src/iree/base/bitmap.c

vendor/iree_tokenizer_src/iree/base/internal/arena.c

vendor/iree_tokenizer_src/iree/base/internal/atomic_slist.c

vendor/iree_tokenizer_src/iree/base/internal/base64.c

vendor/iree_tokenizer_src/iree/base/internal/csprng.c

vendor/iree_tokenizer_src/iree/base/internal/dynamic_library_posix.c

vendor/iree_tokenizer_src/iree/base/internal/dynamic_library_win32.c

vendor/iree_tokenizer_src/iree/base/internal/fpu_state.c

vendor/iree_tokenizer_src/iree/base/internal/json.c

vendor/iree_tokenizer_src/iree/base/internal/memory.c

vendor/iree_tokenizer_src/iree/base/internal/mpsc_queue.c

vendor/iree_tokenizer_src/iree/base/internal/path.c

vendor/iree_tokenizer_src/iree/base/internal/shm_posix.c

vendor/iree_tokenizer_src/iree/base/internal/shm_win32.c

vendor/iree_tokenizer_src/iree/base/internal/spsc_queue.c

vendor/iree_tokenizer_src/iree/base/internal/sysfs.c

vendor/iree_tokenizer_src/iree/base/internal/time.c

vendor/iree_tokenizer_src/iree/base/internal/unicode_tables.c

vendor/iree_tokenizer_src/iree/base/internal/unicode.c

vendor/iree_tokenizer_src/iree/base/printf.c

vendor/iree_tokenizer_src/iree/base/status_stack_trace.c

vendor/iree_tokenizer_src/iree/base/status.c

vendor/iree_tokenizer_src/iree/base/string_builder.c

vendor/iree_tokenizer_src/iree/base/string_view.c

vendor/iree_tokenizer_src/iree/base/time.c

vendor/iree_tokenizer_src/iree/base/tracing/console.c

vendor/iree_tokenizer_src/iree/base/wait_source.c

vendor/iree_tokenizer_src/iree/base/threading/mutex.c