search_core

# Changelog

## 0.2.0

Stemming moves from the `stemmers` Rust NIF to the pure-Elixir
[`text_stemmer`](https://hex.pm/packages/text_stemmer). Installing `search_core` no
longer needs a Rust toolchain or a precompiled-binary download, and stemming is now
preemptible by the BEAM scheduler. Stemmer output is unchanged (both are generated from
the canonical Snowball sources); language coverage grows from 18 to 33.

### Added

- `SearchCore.Language` — the languages the stack accepts, which is exactly the set the
  installed `text_stemmer` reports. `supported_languages/0`, `supported?/1`, `validate!/1`.
- `SearchCore.Language.base/1` — the language an algorithm variant belongs to
  (`:en_porter` → `:en`). Use it for anything keyed by language rather than by algorithm.

### Changed

- **The supported language set follows the installed `text_stemmer`, at call time.** It is
  never frozen into `search_core`, so upgrading `text_stemmer` makes any language it adds
  usable **immediately** — no `search_core` release, no recompile. `text_stemmer` is the
  single authority for what a language is; `search_core` carries no table that could
  disagree with it.
- Passing an unsupported language to `SearchCore.Pipeline.process/3` now raises
  `ArgumentError` naming the accepted shape, rather than failing inside the stemmer.

### Fixed

- **Snowball algorithm variants got no stopwords.** `Stopwords.get/1` was keyed by the
  exact code, so `:en_porter` matched no list and silently indexed every stopword —
  `process("the running dogs", :en_porter)` returned `["the", "run", "dog"]`. It is now
  keyed by `Language.base/1`, so a variant gets its language's list.

### Breaking

- **Languages are ISO 639-1 codes only: `:french` is no longer accepted — use `:fr`.**
  English-name aliases are gone entirely, along with `Language.canonical/1`,
  `canonical!/1`, `accepted/0`, `canonical_languages/0` and `name/1`. There is one
  spelling per language and one authority for it. Wanting a human-readable, *localized*
  language name for a UI is what the ISO code is for — hand it to
  [`ex_cldr_languages`](https://hex.pm/packages/ex_cldr_languages), which does it properly
  in every locale rather than returning an English atom.
- `SearchCore.Stopwords.available_languages/0` returns ISO codes (`[:fr, :en]`) instead of
  English names (`[:french, :english]`). `get/1` is keyed by `Language.base/1`.
- Stemming is ~11µs/word instead of ~0.5µs/word. Invisible on the query path; if you
  bulk-index large corpora and need the throughput, the `stemmers` NIF remains published
  and supported.

## 0.1.0

Initial release.

- `SearchCore.Pipeline` — normalize, tokenize, drop stopwords (FR/EN), stem (via the
  `stemmers` NIF) and optionally fold accents.
- `SearchCore.Tsvector` — `searchable_text/3` for indexing and `tsquery/3` for querying,
  both from the same pipeline (stemming symmetry), with `prefix:` and `combinator:`
  options. Short/all-stopword queries yield an empty tsquery.