Packages

tinkex

0.2.0
0.4.0 0.3.4 0.3.3 0.3.2 0.3.1 0.3.0 0.2.2 0.2.1 0.2.0 0.1.20 0.1.19 0.1.18 0.1.17 0.1.16 0.1.15 0.1.14 0.1.13 0.1.12 0.1.11 0.1.10 0.1.9 0.1.8 0.1.7 0.1.6 0.1.5 0.1.4 0.1.3 0.1.2 0.1.1 0.1.0

Elixir SDK for Tinker: LoRA training, sampling, and future-based workflows with telemetry and HTTP/2.

HexDocs

Current section

Files

Jump to
Readme
30 Versions
8 Dependencies
Dependants
Files
Activity
tinkex lib tinkex regularizers orthogonality.ex
Raw

lib/tinkex/regularizers/orthogonality.ex 

defmodule Tinkex.Regularizers.Orthogonality do

  @behaviour Tinkex.Regularizer



  @moduledoc """

  Orthogonality regularizer adapter using NxPenalties constraints.

  """



  @impl true

  def compute(_data, logprobs, opts \\ []) do

    penalty = NxPenalties.Constraints.orthogonality(logprobs, opts)



    metrics =

      if tracing?(logprobs) do

        %{}

      else

        %{"orthogonality" => Nx.to_number(penalty)}

      end



    {penalty, metrics}

  end



  @impl true

  def name, do: "orthogonality"



  defp tracing?(%Nx.Tensor{data: %Nx.Defn.Expr{}}), do: true

  defp tracing?(_), do: false

end