iree_tokenizers

0.3.0

0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to

Readme 15 Versions 4 Dependencies Dependants Files Activity

iree_tokenizers native iree_tokenizers_native Cargo.toml

Raw

native/iree_tokenizers_native/Cargo.toml

[package]
name = "iree_tokenizers_native"
version = "0.1.0"
edition = "2021"

[lib]
name = "iree_tokenizers_native"
path = "src/lib.rs"
crate-type = ["cdylib"]

[dependencies]
base64 = "0.22"
rustler = "0.37.3"
sentencepiece-model = "0.1.4"
serde_json = "1.0"
thiserror = "2"

[build-dependencies]
cc = "1"

Search filters

Type any of these into the search box. They can be combined with free text which searches package names and descriptions.

Operator	Description	Example
name:	Match package name. Supports * wildcards and repo/package form	name:phx* or name:hexpm/phoenix
description:	Full-text search of package descriptions	description:auth
depends:	Packages depending on a given package. Supports repo:package form	depends:ecto or depends:hexpm:ecto
build_tool:	Filter by build tool	build_tool:mix
updated_after:	Packages updated after an ISO8601 datetime	updated_after:2025-01-01T00:00:00Z
extra:	Match custom metadata (key,value). Nested keys are separated by commas	extra:license,MIT

Wrap values in double quotes to include spaces, e.g. name:"my package".