Packages
A comprehensive Natural Language Processing library for Elixir featuring stemming, tokenization, ranking algorithms (TF-IDF, BM25), similarity metrics, stopword filtering, n-grams, and text statistics. Inspired by NLTK and designed for idiomatic Elixir code.
Current section
Files
Jump to
Current section
Files
CHANGELOG.md
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.1.0] - 2025-01-XX
### Added
- Snowball stemmers for 7 languages (English, Spanish, Portuguese, French, German, Italian, Polish)
- Multiple tokenization strategies (standard, whitespace, regex, n-gram, keyword)
- TF-IDF ranking algorithm with customizable preprocessing
- BM25 ranking algorithm with configurable parameters
- Similarity metrics: Levenshtein, Jaccard, Dice, Jaro-Winkler, Hamming, LCS
- Stopword detection and filtering for 30+ languages
- Text filtering: case conversion, length filtering, pattern replacement
- N-gram generation (character and word n-grams)
- Corpus statistics: term frequency, document frequency, most frequent terms
- Co-occurrence analysis with matrix building
- Comprehensive test suite
- Benchmark suites for performance testing