search_core

# search_core

Multilingual full-text search building blocks for **plain Elixir/Ecto** — no Ash
required. Two pieces, both driven by the pure-Elixir
[`text_stemmer`](https://hex.pm/packages/text_stemmer) (Snowball, 33 languages, no NIF
and no toolchain to install):

- `SearchCore.Pipeline` — turn raw text into normalized, stemmed tokens.
- `SearchCore.Tsvector` — turn those tokens into the `search_text` you index and the
  `tsquery` you search with, both from the *same* pipeline so index and query never
  drift apart.

Part of the [search_ash monorepo](../).

```elixir
SearchCore.process("Les idées mangent les chevaux", :fr)
#=> ["ide", "mangent", "cheval"]

SearchCore.searchable_text("Les chevaux mangent", :fr)  #=> "cheval mangent"
SearchCore.tsquery("chevaux", :fr)                      #=> "cheval"
```

## Naming a language

Languages are **ISO 639-1 codes** (`:fr`, `:en`, `:de`) — one spelling per language, no
aliases:

```elixir
SearchCore.Language.supported_languages()  # the 36 codes, one per algorithm
SearchCore.Language.supported?(:fr)        #=> true
SearchCore.Language.validate!(:klingon)    # ** (ArgumentError) unsupported language …
```

### text_stemmer is the authority

Which languages exist is asked of `text_stemmer` at call time, never frozen in. Upgrading
`text_stemmer` makes any language it adds usable **immediately** — no `search_core`
release, no recompile, nothing to maintain here. `search_core` deliberately carries no
table of language names that could disagree with the stemmer.

Need a human-readable, *localized* language name for a UI? That's what the ISO code is
for — hand it to [`ex_cldr_languages`](https://hex.pm/packages/ex_cldr_languages), which
does it properly in every locale.

### Algorithm variants

Snowball ships more than one algorithm for a couple of languages. Those are addressed by
code only (`:en_porter`, `:en_lovins`, `:nl_porter`), while `:english` and `:dutch` map to
the current recommended algorithm. A variant still belongs to its language — `:en_porter`
takes the English stopword list — which is what `SearchCore.Language.base/1` gives you:

```elixir
SearchCore.Language.base(:en_porter)  #=> :en
```

## Postgres wiring

Tokens are already stemmed and accent-folded, so use the `'simple'` config:

```elixir
# indexing — store this in a column, GIN-index its tsvector
searchable = SearchCore.searchable_text(title <> " " <> body, :fr)

# querying
import Ecto.Query
tsquery = SearchCore.tsquery(user_input, :fr)

from a in Article,
  where: fragment("to_tsvector('simple', ?) @@ to_tsquery('simple', ?)", a.search_text, ^tsquery)
```

Because both sides go through the same pipeline, a search for "chevaux" finds a row
that stored "cheval" — no "search returns nothing" surprises.

See [`examples/basic`](examples/basic) for a runnable Ecto demo (multilingual seed,
search, and an `EXPLAIN` confirming GIN index usage).