Packages

viva_tensor

2.2.102
2.2.104 2.2.103 2.2.102 2.2.100 2.1.0 retired 1.3.0 retired 1.2.1 retired 1.2.0 retired 1.1.0 retired 1.0.0 retired

Tensor library for Gleam/BEAM with a pure Gleam API, zero-copy views, and optional native acceleration

HexDocs Report package

Current section

Files

Jump to
Readme
10 Versions
6 Dependencies
Dependants
Files
Activity
viva_tensor include viva_tensor@text@unigram_UnigramTokenizer.hrl
Raw

include/viva_tensor@text@unigram_UnigramTokenizer.hrl 

-record(unigram_tokenizer, {

    vocab :: gleam@dict:dict(binary(), integer()),

    inverse_vocab :: gleam@dict:dict(integer(), binary()),

    scores :: gleam@dict:dict(binary(), float()),

    unk_id :: integer(),

    bos_id :: integer(),

    eos_id :: integer()

}).