Packages

viva_tensor

2.2.103
2.2.104 2.2.103 2.2.102 2.2.100 2.1.0 retired 1.3.0 retired 1.2.1 retired 1.2.0 retired 1.1.0 retired 1.0.0 retired

Tensor library for Gleam/BEAM with a pure Gleam API, zero-copy views, and optional native acceleration

HexDocs Report package

Current section

Files

Jump to
Readme
10 Versions
7 Dependencies
Dependants
Files
Activity
viva_tensor include viva_tensor@text@tokenizer_WordPieceTokenizer.hrl
Raw

include/viva_tensor@text@tokenizer_WordPieceTokenizer.hrl 

-record(word_piece_tokenizer, {

    vocab :: gleam@dict:dict(binary(), integer()),

    inverse_vocab :: gleam@dict:dict(integer(), binary()),

    unk_id :: integer(),

    cls_id :: integer(),

    sep_id :: integer(),

    pad_id :: integer(),

    max_input_chars_per_word :: integer()

}).