Current section
5 Versions
Jump to
Current section
5 Versions
Compare versions
10
files changed
+604
additions
-81
deletions
| @@ -0,0 +1,94 @@ | |
| 1 | + # Changelog |
| 2 | + |
| 3 | + All notable changes to this project are documented in this file. |
| 4 | + |
| 5 | + ## [0.2.0] - 2026-06-23 |
| 6 | + |
| 7 | + ### Added |
| 8 | + |
| 9 | + - Added model version 7, `:conservative_cnn`, a conservative CNN architecture intended to reduce false positives. |
| 10 | + - Added false-positive-aware validation scoring for checkpoint selection and hyperparameter tuning. |
| 11 | + - Added threshold calibration for v7, including persisted `decision_policy` metadata in serialized classifiers. |
| 12 | + - Added v7 inference safeguards for low-signal inputs through configurable positive thresholds and minimum positive token counts. |
| 13 | + |
| 14 | + ### Changed |
| 15 | + |
| 16 | + - Changed the default training model to v7, `:conservative_cnn`. |
| 17 | + - Changed the default vector length to `512`. |
| 18 | + - Updated serialization, loading, serving, and compiled prediction paths to reuse persisted decision policies when available. |
| 19 | + |
| 20 | + ### Documentation |
| 21 | + |
| 22 | + - Updated README and generated documentation sources for v7, threshold calibration, production guidance, model versions, and the `0.2.0` dependency snippet. |
| 23 | + |
| 24 | + ## [0.1.3] - 2026-06-15 |
| 25 | + |
| 26 | + ### Added |
| 27 | + |
| 28 | + - Added model version 6, `:transformer`, with sequence-length logit bias for short or missing text inputs. |
| 29 | + - Added static sinusoidal positional embeddings to the transformer model. |
| 30 | + - Added transformer execution and logit-bias tests. |
| 31 | + |
| 32 | + ### Changed |
| 33 | + |
| 34 | + - Updated dependencies, including Nx `0.12.1` and ExDoc `0.40.3`. |
| 35 | + - Reduced JIT recompilation warnings during training and hyperparameter tuning. |
| 36 | + |
| 37 | + ### Documentation |
| 38 | + |
| 39 | + - Updated release documentation for the transformer model. |
| 40 | + |
| 41 | + ## [0.1.2] - 2026-05-17 |
| 42 | + |
| 43 | + ### Changed |
| 44 | + |
| 45 | + - Updated dependency versions. |
| 46 | + - Fixed the Nx dependency version. |
| 47 | + |
| 48 | + ## [0.1.1] - 2026-04-19 |
| 49 | + |
| 50 | + ### Added |
| 51 | + |
| 52 | + - Added model version 5 with a multi-scale architecture and made it the default at the time. |
| 53 | + - Added tests for model version 5. |
| 54 | + |
| 55 | + ### Changed |
| 56 | + |
| 57 | + - Refactored model architecture names to be more descriptive. |
| 58 | + - Updated CI to newer Elixir and OTP versions. |
| 59 | + |
| 60 | + ### Fixed |
| 61 | + |
| 62 | + - Fixed Axon deprecation warnings by using `Axon.ModelState.empty()`. |
| 63 | + - Fixed compiler warnings and formatted tests. |
| 64 | + |
| 65 | + ## [0.1.0] - 2026-02-06 |
| 66 | + |
| 67 | + ### Added |
| 68 | + |
| 69 | + - Added the initial binary text classification library built on Axon and Nx. |
| 70 | + - Added tokenizer, vectorizer, training, serving, saving, and loading workflows. |
| 71 | + - Added explicit label mapping support. |
| 72 | + - Added configurable compiler/backend options for training and serving. |
| 73 | + - Added model versioning for backward-compatible serialized classifiers. |
| 74 | + - Added model versions 1 through 4, including CNN, mixed-pooling CNN, multi-scale CNN, and Sep-SE-CNN. |
| 75 | + - Added hyperparameter auto-tuning for learning rate and dropout. |
| 76 | + - Added `BinClass.compile_predictor/2` for low-latency in-process inference. |
| 77 | + - Added examples for training, serving, configurable backends, and simple inference. |
| 78 | + - Added Hex package metadata, MIT license, CI workflow, and Dependabot configuration. |
| 79 | + |
| 80 | + ### Changed |
| 81 | + |
| 82 | + - Optimized model architecture and training performance before the first tagged release. |
| 83 | + - Reverted the experimental v5 work before `v0.1.0` and kept v4 as the default at release time. |
| 84 | + - Marked the temporary-file helper module as internal. |
| 85 | + |
| 86 | + ### Documentation |
| 87 | + |
| 88 | + - Added and refined README, examples documentation, and module documentation for the initial Hex release. |
| 89 | + |
| 90 | + [0.2.0]: https://github.com/preciz/bin_class/compare/v0.1.3...HEAD |
| 91 | + [0.1.3]: https://github.com/preciz/bin_class/compare/v0.1.2...v0.1.3 |
| 92 | + [0.1.2]: https://github.com/preciz/bin_class/compare/v0.1.1...v0.1.2 |
| 93 | + [0.1.1]: https://github.com/preciz/bin_class/compare/v0.1.0...v0.1.1 |
| 94 | + [0.1.0]: https://github.com/preciz/bin_class/releases/tag/v0.1.0 |
| @@ -13,7 +13,7 @@ The package can be installed by adding `bin_class` to your list of dependencies | |
| 13 13 | ```elixir |
| 14 14 | def deps do |
| 15 15 | [ |
| 16 | - {:bin_class, "~> 0.1.3"} |
| 16 | + {:bin_class, "~> 0.2.0"} |
| 17 17 | ] |
| 18 18 | end |
| 19 19 | ``` |
| @@ -35,19 +35,24 @@ data = [ | |
| 35 35 | ### 2. Train the model |
| 36 36 | |
| 37 37 | ```elixir |
| 38 | - # Labels can be a list (index 0 and 1) or an explicit map |
| 38 | + # Labels can be a list (index 0 and 1) or an explicit map. |
| 39 | + # The default model is v7, a conservative CNN optimized to reduce false positives. |
| 39 40 | classifier = BinClass.Trainer.train(data, |
| 40 41 | epochs: 10, |
| 41 42 | labels: %{0 => :negative, 1 => :positive} |
| 42 43 | ) |
| 43 44 | |
| 44 | - # Optional: Enable auto-tuning to find the best Learning Rate and Dropout |
| 45 | + # Optional: enable auto-tuning to search learning rate and dropout. |
| 46 | + # For production training, tune: false is usually preferred unless you have |
| 47 | + # a stable validation split and enough data. |
| 45 48 | classifier = BinClass.Trainer.train(data, tune: true) |
| 46 49 | ``` |
| 47 50 | |
| 51 | + By default, training uses a fixed token vector length of `512`. |
| 52 | + |
| 48 53 | ### 3. Save and Load |
| 49 54 | |
| 50 | - You can save the entire model (including tokenizer, parameters, and metadata) to a single file. |
| 55 | + You can save the entire model (including tokenizer, parameters, decision policy, and metadata) to a single file. |
| 51 56 | |
| 52 57 | ```elixir |
| 53 58 | BinClass.save(classifier, "my_model.bin") |
| @@ -89,10 +94,42 @@ Check out the `examples/` directory for scripts demonstrating various use cases: | |
| 89 94 | ## Features |
| 90 95 | |
| 91 96 | - **Production Ready**: Built on `Nx.Serving` for automatic batching and process isolation. |
| 92 | - - **Unified Serialization**: Save and load the entire classifier state from a single file. |
| 97 | + - **Unified Serialization**: Save and load the entire classifier state, including tokenizer, model parameters, and calibrated decision policy, from a single file. |
| 93 98 | - **Model Versioning**: Decouples model parameters from code changes by explicitly versioning architectures. |
| 94 | - - **Multiple Architectures**: Supports **Sep-SE-CNN** (Separable Convolutions + Squeeze-and-Excitation) with multi-scale kernels and a high-accuracy **Transformer Encoder** architecture with sequence length logit bias. |
| 99 | + - **Multiple Architectures**: Supports legacy CNN variants, **Sep-SE-CNN** (Separable Convolutions + Squeeze-and-Excitation), **Transformer Encoder**, and v7 **Conservative CNN**. |
| 100 | + - **Conservative v7 Default**: Uses a v1-style CNN backbone with false-positive-aware checkpoint selection, low-signal safeguards, and persisted threshold calibration. |
| 95 101 | - **Early Stopping**: Automatically halts training when validation loss stops improving. |
| 96 102 | - **Automatic Class Balancing**: Handles imbalanced datasets via automated oversampling. |
| 97 103 | - **Automated Tokenization**: Automatically builds vocabulary from training data or accepts custom streams. |
| 98 104 | - **Efficient**: Uses `EXLA` as the default compiler for high-performance training and inference, with support for other `Nx` backends and compilers. |
| 105 | + |
| 106 | + ## Model Versions |
| 107 | + |
| 108 | + Models are versioned so saved classifiers keep loading even when the library default changes: |
| 109 | + |
| 110 | + - `1` / `:cnn`: original CNN |
| 111 | + - `2` / `:cnn_mixed_pooling`: CNN with mixed pooling |
| 112 | + - `3` / `:multi_scale_cnn`: multi-scale CNN |
| 113 | + - `4` / `:sep_se_cnn`: separable CNN with squeeze-and-excitation |
| 114 | + - `5` / `:parallel_cnn`: parallel CNN |
| 115 | + - `6` / `:transformer`: transformer encoder |
| 116 | + - `7` / `:conservative_cnn`: conservative CNN default |
| 117 | + |
| 118 | + v7 is based on the original simple CNN backbone, with production behavior tuned for cases where false positives are more costly than false negatives. It selects checkpoints using a false-positive penalty and stores a calibrated positive threshold in the serialized classifier. |
| 119 | + |
| 120 | + ## Production Notes |
| 121 | + |
| 122 | + For production training, start with the defaults unless you have a specific validation strategy: |
| 123 | + |
| 124 | + ```elixir |
| 125 | + classifier = BinClass.Trainer.train(data, |
| 126 | + epochs: 10, |
| 127 | + tune: false, |
| 128 | + model_version: :conservative_cnn, |
| 129 | + vector_length: 512, |
| 130 | + false_positive_penalty: 0.5, |
| 131 | + calibrate_threshold: true |
| 132 | + ) |
| 133 | + ``` |
| 134 | + |
| 135 | + The calibrated `decision_policy` is saved with the model and reused by both `BinClass.load/2` and `BinClass.compile_predictor/2`. Older serialized models that do not include this field still load normally and fall back to their version defaults. |
| @@ -1,6 +1,6 @@ | |
| 1 1 | {<<"links">>,[{<<"GitHub">>,<<"https://github.com/preciz/bin_class">>}]}. |
| 2 2 | {<<"name">>,<<"bin_class">>}. |
| 3 | - {<<"version">>,<<"0.1.3">>}. |
| 3 | + {<<"version">>,<<"0.2.0">>}. |
| 4 4 | {<<"description">>, |
| 5 5 | <<"An easy-to-use library for building, training, and deploying binary text classifiers with Axon.">>}. |
| 6 6 | {<<"elixir">>,<<"~> 1.19">>}. |
| @@ -40,8 +40,10 @@ | |
| 40 40 | <<"lib/bin_class/model/multi_scale_cnn.ex">>, |
| 41 41 | <<"lib/bin_class/model/sep_se_cnn.ex">>, |
| 42 42 | <<"lib/bin_class/model/parallel_cnn.ex">>, |
| 43 | + <<"lib/bin_class/model/conservative_cnn.ex">>, |
| 43 44 | <<"lib/bin_class/model/transformer.ex">>,<<"lib/bin_class/serving.ex">>, |
| 44 45 | <<"lib/bin_class/tmp.ex">>,<<"lib/bin_class/tokenizer.ex">>, |
| 45 46 | <<"lib/bin_class/trainer.ex">>,<<"lib/bin_class/vectorizer.ex">>, |
| 46 | - <<".formatter.exs">>,<<"mix.exs">>,<<"README.md">>,<<"LICENSE">>]}. |
| 47 | + <<".formatter.exs">>,<<"mix.exs">>,<<"README.md">>,<<"LICENSE">>, |
| 48 | + <<"CHANGELOG.md">>]}. |
| 47 49 | {<<"build_tools">>,[<<"mix">>]}. |
| @@ -18,6 +18,10 @@ defmodule BinClass do | |
| 18 18 | labels: %{0 => :negative, 1 => :positive} |
| 19 19 | ) |
| 20 20 | |
| 21 | + # The default model is v7 (:conservative_cnn). It uses a 512-token |
| 22 | + # vector length, false-positive-aware checkpoint selection, and |
| 23 | + # persisted threshold calibration. |
| 24 | + |
| 21 25 | # Save the model |
| 22 26 | BinClass.save(classifier, "model.bin") |
| 23 27 | |
| @@ -49,6 +53,11 @@ defmodule BinClass do | |
| 49 53 | |
| 50 54 | @doc """ |
| 51 55 | Serializes the classifier to a binary. |
| 56 | + |
| 57 | + The serialized data includes tokenizer state, model parameters, labels, |
| 58 | + vector length, model version, training metadata, and any calibrated |
| 59 | + decision policy. Classifiers saved before decision policies existed remain |
| 60 | + readable via `deserialize_classifier/1`. |
| 52 61 | """ |
| 53 62 | def serialize(%BinClass.Classifier{} = classifier) do |
| 54 63 | # Ensure model params are on Binary backend for safe serialization |
| @@ -64,7 +73,8 @@ defmodule BinClass do | |
| 64 73 | epoch: classifier.epoch, |
| 65 74 | model_version: classifier.model_version, |
| 66 75 | learning_rate: classifier.learning_rate, |
| 67 | - dropout_rate: classifier.dropout_rate |
| 76 | + dropout_rate: classifier.dropout_rate, |
| 77 | + decision_policy: classifier.decision_policy |
| 68 78 | } |
| 69 79 | |
| 70 80 | :erlang.term_to_binary(data) |
| @@ -82,7 +92,8 @@ defmodule BinClass do | |
| 82 92 | vocab_size: classifier.vocab_size, |
| 83 93 | labels: classifier.labels, |
| 84 94 | model_version: classifier.model_version, |
| 85 | - dropout_rate: classifier.dropout_rate || 0.2 |
| 95 | + dropout_rate: classifier.dropout_rate || 0.2, |
| 96 | + decision_policy: classifier.decision_policy |
| 86 97 | ) |
| 87 98 | |
| 88 99 | BinClass.Serving.new(classifier.model_params, classifier.tokenizer, serving_opts) |
| @@ -100,7 +111,8 @@ defmodule BinClass do | |
| 100 111 | vocab_size: classifier.vocab_size, |
| 101 112 | labels: classifier.labels, |
| 102 113 | model_version: classifier.model_version, |
| 103 | - dropout_rate: classifier.dropout_rate || 0.2 |
| 114 | + dropout_rate: classifier.dropout_rate || 0.2, |
| 115 | + decision_policy: classifier.decision_policy |
| 104 116 | ) |
| 105 117 | |
| 106 118 | BinClass.Serving.new(classifier.model_params, classifier.tokenizer, serving_opts) |
| @@ -116,6 +128,9 @@ defmodule BinClass do | |
| 116 128 | |
| 117 129 | @doc """ |
| 118 130 | Deserializes a saved model from a binary and returns a BinClass.Classifier struct. |
| 131 | + |
| 132 | + Older binaries that do not contain `:decision_policy` load with |
| 133 | + `decision_policy: nil` and use version defaults at inference time. |
| 119 134 | """ |
| 120 135 | def deserialize_classifier(binary) when is_binary(binary) do |
| 121 136 | data = :erlang.binary_to_term(binary) |
| @@ -132,7 +147,8 @@ defmodule BinClass do | |
| 132 147 | epoch: Map.get(data, :epoch), |
| 133 148 | model_version: Map.get(data, :model_version, 1), |
| 134 149 | learning_rate: Map.get(data, :learning_rate), |
| 135 | - dropout_rate: Map.get(data, :dropout_rate, 0.2) |
| 150 | + dropout_rate: Map.get(data, :dropout_rate, 0.2), |
| 151 | + decision_policy: Map.get(data, :decision_policy) |
| 136 152 | } |
| 137 153 | end |
| 138 154 | |
| @@ -162,6 +178,11 @@ defmodule BinClass do | |
| 162 178 | |
| 163 179 | {_, predict_fn} = Axon.build(model, compiler: compiler) |
| 164 180 | |
| 181 | + policy = |
| 182 | + classifier.decision_policy || BinClass.Serving.decision_policy(classifier.model_version) |
| 183 | + |
| 184 | + use_decision_policy? = BinClass.Serving.decision_policy?(classifier.model_version) |
| 185 | + |
| 165 186 | # Compile the prediction function specifically for the given batch size |
| 166 187 | template = Nx.broadcast(0, {batch_size, classifier.vector_length}) |> Nx.as_type(:u16) |
| 167 188 | |
| @@ -181,12 +202,19 @@ defmodule BinClass do | |
| 181 202 | Enum.take(texts, batch_size) |
| 182 203 | end |
| 183 204 | |
| 184 | - batch_tensor = |
| 205 | + vectors = |
| 185 206 | batch_texts |
| 186 207 | |> Enum.map(fn text -> |
| 187 208 | BinClass.Vectorizer.build(classifier.tokenizer, text, classifier.vector_length) |
| 188 | - |> Nx.tensor(type: :u16) |
| 189 209 | end) |
| 210 | + |
| 211 | + active_lengths = |
| 212 | + vectors |
| 213 | + |> Enum.map(fn vector -> Enum.count(vector, &(&1 != 0)) end) |
| 214 | + |
| 215 | + batch_tensor = |
| 216 | + vectors |
| 217 | + |> Enum.map(&Nx.tensor(&1, type: :u16)) |
| 190 218 | |> Nx.Batch.stack() |
| 191 219 | |
| 192 220 | # Run inference directly |
| @@ -199,7 +227,20 @@ defmodule BinClass do | |
| 199 227 | batch_output |
| 200 228 | |> Nx.to_list() |
| 201 229 | |> Enum.take(length(texts)) |
| 202 | - |> Enum.map(fn probs -> BinClass.Serving.decode_prediction(probs, classifier.labels) end) |
| 230 | + |> Enum.zip(active_lengths) |
| 231 | + |> Enum.map(fn {probs, active_length} -> |
| 232 | + if use_decision_policy? do |
| 233 | + BinClass.Serving.decode_prediction( |
| 234 | + probs, |
| 235 | + classifier.labels, |
| 236 | + policy.positive_threshold, |
| 237 | + active_length, |
| 238 | + policy.min_positive_tokens |
| 239 | + ) |
| 240 | + else |
| 241 | + BinClass.Serving.decode_prediction(probs, classifier.labels) |
| 242 | + end |
| 243 | + end) |
| 203 244 | |
| 204 245 | if multi?, do: results, else: List.first(results) |
| 205 246 | end |
| @@ -1,6 +1,10 @@ | |
| 1 1 | defmodule BinClass.Classifier do |
| 2 2 | @moduledoc """ |
| 3 3 | A struct representing a trained binary classifier. |
| 4 | + |
| 5 | + The optional `:decision_policy` field stores calibrated inference settings |
| 6 | + such as `:positive_threshold` and `:min_positive_tokens`. It is persisted |
| 7 | + with newly serialized classifiers and may be `nil` for older saved models. |
| 4 8 | """ |
| 5 9 | |
| 6 10 | defstruct [ |
| @@ -13,7 +17,8 @@ defmodule BinClass.Classifier do | |
| 13 17 | :epoch, |
| 14 18 | :model_version, |
| 15 19 | :learning_rate, |
| 16 | - :dropout_rate |
| 20 | + :dropout_rate, |
| 21 | + :decision_policy |
| 17 22 | ] |
| 18 23 | |
| 19 24 | defimpl Inspect do |
| @@ -31,7 +36,8 @@ defmodule BinClass.Classifier do | |
| 31 36 | accuracy: classifier.accuracy, |
| 32 37 | epoch: classifier.epoch, |
| 33 38 | learning_rate: classifier.learning_rate, |
| 34 | - dropout_rate: classifier.dropout_rate |
| 39 | + dropout_rate: classifier.dropout_rate, |
| 40 | + decision_policy: classifier.decision_policy |
| 35 41 | }, |
| 36 42 | opts |
| 37 43 | ), |
Loading more files…