Packages

iree_tokenizers

0.2.0
0.8.6 0.8.5 0.8.4 0.8.3 0.8.2 0.8.1 0.8.0 0.7.0 0.6.0 0.5.0 0.4.0 0.3.0 0.2.0 0.1.0 0.1.0-dev

Fast Hugging Face tokenizer.json, OpenAI tiktoken, and SentencePiece model bindings for Elixir via the IREE tokenizer runtime

HexDocs

Current section

Files

Jump to
Readme
15 Versions
4 Dependencies
Dependants
Files
Activity
iree_tokenizers lib iree tokenizers encoding transformation.ex
Raw

lib/iree/tokenizers/encoding/transformation.ex 

defmodule IREE.Tokenizers.Encoding.Transformation do

  @moduledoc """

  Helpers for building encoding transformation lists.

  """



  @type t ::

          {:pad, {non_neg_integer(), keyword()}}

          | {:truncate, {non_neg_integer(), keyword()}}

          | {:set_sequence_id, non_neg_integer()}



  @spec pad(non_neg_integer(), keyword()) :: t()

  def pad(target_length, opts \\ []), do: {:pad, {target_length, opts}}



  @spec truncate(non_neg_integer(), keyword()) :: t()

  def truncate(max_length, opts \\ []), do: {:truncate, {max_length, opts}}



  @spec set_sequence_id(non_neg_integer()) :: t()

  def set_sequence_id(id), do: {:set_sequence_id, id}

end