Packages
Tensor library for Gleam/BEAM with a pure Gleam API, zero-copy views, and optional native acceleration
Current section
Files
Jump to
Current section
Files
include/viva_tensor@text@tokenizer_WhitespaceTokenizer.hrl
-record(whitespace_tokenizer, {
vocab :: gleam@dict:dict(binary(), integer()),
inverse_vocab :: gleam@dict:dict(integer(), binary()),
unk_id :: integer(),
pad_id :: integer()
}).