Packages

iree_tokenizers

0.1.0
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

15 Versions

Jump to
Readme
15 Versions
3 Dependencies
Dependants
Files
Activity
Hide whitespace

Compare versions

3 files changed
+13 additions
-9 deletions
  @@ -1,6 +1,7 @@
1 1 # IREE.Tokenizers
2 2
3 - Fast Hugging Face `tokenizer.json` bindings for Elixir backed by the IREE tokenizer runtime.
3 + Fast Hugging Face `tokenizer.json` bindings for Elixir backed by the IREE tokenizer runtime. I discovered [IREE Tokenizers](https://github.com/iree-org/iree-tokenizer-py) from the [ZML.ai blog](https://zml.ai/posts/iree-tokenizer/), a company I deeply admire!
4 +
4 5
5 6 ## Features
6 7
  @@ -87,7 +88,9 @@ package.
87 88
88 89 #### Model latency comparison
89 90
90 - The current checked-in local snapshot from [`bench/results/model_matrix.md`](bench/results/model_matrix.md) contains:
91 + The current checked-in local snapshot from
92 + [`bench/results/model_matrix.md`](https://github.com/goodhamgupta/iree_tokenizers/blob/main/bench/results/model_matrix.md?raw=1)
93 + contains:
91 94
92 95 | Model | Repo used | Tokenizers package (ms) | IREE oneshot / stream (ms) | Speedup |
93 96 | --- | --- | ---: | ---: | --- |
  @@ -109,15 +112,16 @@ The benchmark harness intentionally keeps only one representative repo per token
109 112
110 113 Latency chart:
111 114
112 - ![Model matrix latency](bench/results/model_matrix_latency.svg)
115 + ![Model matrix latency](https://github.com/goodhamgupta/iree_tokenizers/blob/main/bench/results/model_matrix_latency.svg?raw=1)
113 116
114 117 Speedup chart:
115 118
116 - ![Model matrix speedup](bench/results/model_matrix_speedup.svg)
119 + ![Model matrix speedup](https://github.com/goodhamgupta/iree_tokenizers/blob/main/bench/results/model_matrix_speedup.svg?raw=1)
117 120
118 121 ### Benchmark Harness
119 122
120 - The benchmark harness lives under [`bench/`](bench/README.md).
123 + The benchmark harness lives under
124 + [`bench/`](https://github.com/goodhamgupta/iree_tokenizers/tree/main/bench).
121 125
122 126 Set it up once:
  @@ -1,7 +1,7 @@
1 1 {<<"links">>,
2 - [{<<"GitHub">>,<<"https://github.com/shubhamgupta/iree_tokenizers">>}]}.
2 + [{<<"GitHub">>,<<"https://github.com/goodhamgupta/iree_tokenizers">>}]}.
3 3 {<<"name">>,<<"iree_tokenizers">>}.
4 - {<<"version">>,<<"0.1.0-dev">>}.
4 + {<<"version">>,<<"0.1.0">>}.
5 5 {<<"description">>,
6 6 <<"Fast Hugging Face tokenizer.json bindings for Elixir via the IREE tokenizer runtime">>}.
7 7 {<<"elixir">>,<<"~> 1.19">>}.
  @@ -1,8 +1,8 @@
1 1 defmodule IREETokenizers.MixProject do
2 2 use Mix.Project
3 3
4 - @source_url "https://github.com/shubhamgupta/iree_tokenizers"
5 - @version "0.1.0-dev"
4 + @source_url "https://github.com/goodhamgupta/iree_tokenizers"
5 + @version "0.1.0"
6 6
7 7 def project do
8 8 [