Packages

viva_tensor

2.2.103
2.2.104 2.2.103 2.2.102 2.2.100 2.1.0 retired 1.3.0 retired 1.2.1 retired 1.2.0 retired 1.1.0 retired 1.0.0 retired

Tensor library for Gleam/BEAM with a pure Gleam API, zero-copy views, and optional native acceleration

HexDocs Report package

Current section

Files

Jump to
Readme
10 Versions
7 Dependencies
Dependants
Files
Activity
viva_tensor include viva_tensor@text@tokenizer_BpeTokenizer.hrl
Raw

include/viva_tensor@text@tokenizer_BpeTokenizer.hrl 

-record(bpe_tokenizer, {

    vocab :: gleam@dict:dict(binary(), integer()),

    inverse_vocab :: gleam@dict:dict(integer(), binary()),

    merges :: list({binary(), binary()}),

    unk_id :: integer()

}).