search_core

# Changelog

## 0.3.0

### Added

- **`SearchCore.highlight/4`** (`SearchCore.Highlight`) — mark the words of a raw text
  that match a search query, for rendering result excerpts. Matching runs through the
  same pipeline as indexing and querying, so a word is highlighted exactly when it
  would have matched in Postgres: `"tomate"` highlights `"tomates"`, `"idee"`
  highlights `"idées"`. Returns renderer-agnostic `{:text, _}` / `{:match, _}` segments
  whose concatenation reproduces the input byte for byte; `prefix: true` mirrors a
  prefix tsquery.
- **`SearchCore.normalize/1`** (`SearchCore.Pipeline.normalize/1`) — the case- and
  accent-insensitive normal form of a string (trim + downcase + fold accents), for
  literal comparisons. Meant to back *both* sides of a comparison, the same way
  `process/3` backs both indexing and querying.
- **`SearchCore.weighted/3`** (`SearchCore.Tsvector.weighted/3`) — build a **weighted
  tsvector literal** from `{text, weight}` segments, to store instead of
  `searchable_text/3` output. Weights are Postgres' `:a`–`:d` classes, so `ts_rank` scores
  a hit in a reference or a title above the same hit in a body:

  ```elixir
  SearchCore.weighted([{"Chevaux", :a}, {"mangent du foin", :d}], :fr)
  #=> "'cheval':1A 'mangent':2 'foin':3"
  ```

  Positions are numbered across all segments, so term frequency still feeds the rank. The
  SQL side casts (`search_text::tsvector`) rather than calling `to_tsvector`. Because the
  pipeline restricts tokens to letters and digits, the generated literal needs no escaping
  — which is what makes producing it in Elixir safe.

### Changed

- **`SearchCore.Pipeline.process/3` drops tokens over 2046 bytes** (`:max_bytes`, tunable,
  forwarded by `tsquery/3`). That is Postgres' hard limit for one lexeme: `to_tsvector`
  merely warns and skips such a word, but a tsvector *literal* carrying it fails to parse
  — and a caller storing `weighted/3` output would see that failure inside their write
  transaction. Capping in the pipeline keeps the indexing and querying sides agreeing on
  which tokens exist at all.

## 0.2.0

Stemming moves from the `stemmers` Rust NIF to the pure-Elixir
[`text_stemmer`](https://hex.pm/packages/text_stemmer). Installing `search_core` no
longer needs a Rust toolchain or a precompiled-binary download, and stemming is now
preemptible by the BEAM scheduler. Stemmer output is unchanged (both are generated from
the canonical Snowball sources); language coverage grows from 18 to 33.

### Added

- `SearchCore.Language` — the languages the stack accepts, which is exactly the set the
  installed `text_stemmer` reports. `supported_languages/0`, `supported?/1`, `validate!/1`.
- `SearchCore.Language.base/1` — the language an algorithm variant belongs to
  (`:en_porter` → `:en`). Use it for anything keyed by language rather than by algorithm.

### Changed

- **The supported language set follows the installed `text_stemmer`, at call time.** It is
  never frozen into `search_core`, so upgrading `text_stemmer` makes any language it adds
  usable **immediately** — no `search_core` release, no recompile. `text_stemmer` is the
  single authority for what a language is; `search_core` carries no table that could
  disagree with it.
- Passing an unsupported language to `SearchCore.Pipeline.process/3` now raises
  `ArgumentError` naming the accepted shape, rather than failing inside the stemmer.

### Fixed

- **Snowball algorithm variants got no stopwords.** `Stopwords.get/1` was keyed by the
  exact code, so `:en_porter` matched no list and silently indexed every stopword —
  `process("the running dogs", :en_porter)` returned `["the", "run", "dog"]`. It is now
  keyed by `Language.base/1`, so a variant gets its language's list.

### Breaking

- **Languages are ISO 639-1 codes only: `:french` is no longer accepted — use `:fr`.**
  English-name aliases are gone entirely, along with `Language.canonical/1`,
  `canonical!/1`, `accepted/0`, `canonical_languages/0` and `name/1`. There is one
  spelling per language and one authority for it. Wanting a human-readable, *localized*
  language name for a UI is what the ISO code is for — hand it to
  [`ex_cldr_languages`](https://hex.pm/packages/ex_cldr_languages), which does it properly
  in every locale rather than returning an English atom.
- `SearchCore.Stopwords.available_languages/0` returns ISO codes (`[:fr, :en]`) instead of
  English names (`[:french, :english]`). `get/1` is keyed by `Language.base/1`.
- Stemming is ~11µs/word instead of ~0.5µs/word. Invisible on the query path; if you
  bulk-index large corpora and need the throughput, the `stemmers` NIF remains published
  and supported.

## 0.1.0

Initial release.

- `SearchCore.Pipeline` — normalize, tokenize, drop stopwords (FR/EN), stem (via the
  `stemmers` NIF) and optionally fold accents.
- `SearchCore.Tsvector` — `searchable_text/3` for indexing and `tsquery/3` for querying,
  both from the same pipeline (stemming symmetry), with `prefix:` and `combinator:`
  options. Short/all-stopword queries yield an empty tsquery.