Packages

iree_tokenizers

0.1.0-dev
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to
Readme
15 Versions
3 Dependencies
Dependants
Files
Activity
iree_tokenizers test fixtures minimal_unigram.json
Raw

test/fixtures/minimal_unigram.json 

{

  "version": "1.0",

  "model": {

    "type": "Unigram",

    "vocab": [

      ["<unk>", -10.0],

      ["▁hello", -1.0],

      ["▁world", -1.0]

    ],

    "unk_id": 0

  },

  "pre_tokenizer": {

    "type": "Metaspace",

    "replacement": "▁",

    "prepend_scheme": "always",

    "split": true

  },

  "decoder": {

    "type": "Metaspace",

    "replacement": "▁",

    "prepend_scheme": "always",

    "split": true

  }

}