Packages
Tensor library for Gleam/BEAM with a pure Gleam API, zero-copy views, and optional native acceleration
Current section
Files
Jump to
Current section
Files
include/viva_tensor@text@unigram_SentencePieceTokenizer.hrl
-record(sentence_piece_tokenizer, {
mode :: viva_tensor@text@unigram:sentence_piece_mode(),
unigram :: gleam@option:option(viva_tensor@text@unigram:unigram_tokenizer()),
bpe :: gleam@option:option(viva_tensor@text@tokenizer:bpe_tokenizer())
}).