Packages

iree_tokenizers

0.8.2
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to
Readme
15 Versions
4 Dependencies
Dependants
Files
Activity
iree_tokenizers test fixtures sentencepiece_stream_split_minimal.json
Raw

test/fixtures/sentencepiece_stream_split_minimal.json 

{

  "normalizer": {

    "type": "Sequence",

    "normalizers": [

      {

        "type": "Prepend",

        "prepend": "▁"

      },

      {

        "type": "Replace",

        "pattern": {

          "String": " "

        },

        "content": "▁"

      }

    ]

  },

  "model": {

    "type": "BPE",

    "vocab": {

      "▁h": 0,

      "ello": 1,

      "▁hello": 2

    },

    "merges": [

      "▁h ello"

    ]

  }

}