torus

# v0.7.0

## New 🔥

**Hybrid Search** is now available via the new `Torus.hybrid/4` macro!

Hybrid search fuses several search strategies into a single ranked query using [Reciprocal Rank Fusion](https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking) - the industry-standard way to combine keyword and semantic search. Rows that rank high in several branches win; rows found by only one branch still compete.

```elixir
search_vector = Torus.to_vector("magic school")

Post
|> Torus.hybrid([p], [
     full_text: {[p.title, p.body], "magic school"},
     semantic: {p.embedding, search_vector, weight: 2.0}
   ], limit: 10)
|> Repo.all()
```

Key features:

- Fuses any combination of `full_text`, `bm25`, `similarity`, and `semantic` branches (the same type can appear more than once)
- Per-branch `:weight` and `:limit`, tunable RRF `:k`
- Everything runs in a single SQL query - the result is a regular Ecto query you can keep piping `select`, `preload`, or pagination onto
- The fused score is available via the `:torus_hybrid` named binding or the `:score_key` option

See the [Hybrid search guide](guides/hybrid_search.md) for details.

**Match highlighting** is now available via the new `Torus.highlight/3` macro!

Built on PostgreSQL's `ts_headline`, it wraps matches of the term in the selected text (in `<b>` tags by default) and can also return snippets instead of the full text. A `type: :substring` mode highlights matches inside words, pairing with `ilike`/`like` searches.

The simplest way to use it is the new `:highlight` option on `full_text`, `bm25`, `similarity`, `ilike`, `like`, and `hybrid` branches - just point it at the columns, the search's own term and options are reused:

```elixir
Post
|> Torus.full_text([p], [p.title, p.body], "shocker", highlight: [title: p.title])
|> Repo.all()
# => [%Post{title: "Hogwarts <b>Shocker</b>", ...}]
```

Or standalone in any `select`/`select_merge` via `Torus.highlight/3`:

```elixir
Post
|> Torus.full_text([p], [p.title, p.body], "shocker")
|> select([p], Torus.highlight(p.title, "shocker"))
|> Repo.all()
# => ["Hogwarts <b>Shocker</b>"]
```

## Improvements

- Test suite now runs semantic (pgvector) integration tests on CI.

## Breaking changes ⚠️

- Torus now requires `ecto`/`ecto_sql` `~> 3.10` (hybrid search relies on
  `selected_as/2` and ordering by select aliases). Upgrade Ecto before bumping Torus.
- `Torus.Embeddings.NebulexCache` now targets Nebulex 3.0+. Nebulex 3.0 moved
  `Nebulex.Adapters.Local` (the default adapter) to the separate `nebulex_local`
  package - if you use the cache, upgrade `nebulex` to `>= 3.0.0` and add
  `nebulex_local` to your dependencies.

## Fixes

- `Torus.QueryInspector.tap_sql/3` no longer crashes - it now prints the SQL and its
  parameters and returns the query.
- `Torus.similarity/5` and `Torus.full_text/5` with `order: :desc` (the default) now
  order with `NULLS LAST`, so rows whose search column is `NULL` no longer rank above
  actual matches. Hybrid branches rank the same way.
- `Torus.Embeddings.NebulexCache.embedding_model/1` no longer raises - it now passes
  the options through to the underlying embedding module.
- Empty search terms contribute no rows in `similarity` and `bm25` hybrid branches
  (matching `full_text` branches), instead of boosting arbitrary rows.

# v0.6.0

## New 🔥

**BM25 Full-Text Search** is now available via the new `Torus.bm25/5` macro!

[BM25](https://en.wikipedia.org/wiki/Okapi_BM25) is a modern ranking algorithm that generally provides superior relevance scoring compared to traditional TF-IDF (used by `full_text/5`). This integration uses the [pg_textsearch](https://github.com/timescale/pg_textsearch) extension by Timescale.

**See it in action [on the demo page](https://torus.dimamik.com/?method=bm25)**

Key features:

- State-of-the-art BM25 ranking with configurable index parameters (k1, b)
- Blazingly fast top-k queries via Block-Max WAND optimization (`Torus.bm25/5` + `limit`)
- Simple syntax: `Post |> Torus.bm25([p], p.body, "search term") |> limit(10)`
- Score selection with `:score_key` and post-filtering with `:score_threshold`
- Language/stemming configured at index creation via `text_config`

Requirements:

- PostgreSQL 17+
- pg_textsearch extension installed
- BM25 index on the search column (with `text_config` for language)

See the [BM25 Search Guide](https://dimamik.com/posts/bm25_search) for detailed setup instructions and examples.

**When to use BM25 vs full_text:**

- Use `bm25/5` for fast single-column search with modern relevance ranking
- Use `full_text/5` for multi-column search with weights or when using stored tsvector columns

# v0.5.3

## Fixes

- `Torus.Embeddings.OpenAI` now correctly parses updated response structure from OpenAI API.
- `exla` nx backend is now correctly supported in `Torus.Embeddings.LocalNxServing`.

# v0.5.2

## New 🔥

- New [demo page](https://torus.dimamik.com) where you can explore different search types and their options. It also includes semantic search, so if you're hesitant - go check it out!
- Other documentation improvements

## Fixes

- Correctly handles `order: :none` in `Torus.semantic/5` search.
- Updates `Torus.Embeddings.HuggingFace` to point to the updated feature extraction endpoint.
- Suppresses warnings for missing `ecto_sql` dependency by adding it to the required dependencies. Most of us already had it, but now it'll be explicit.
- Correctly parses an array of integers in `Torus.QueryInspector.substituted_sql/3` and `Torus.QueryInspector.tap_substituted_sql/3`. Now we should be able to handle all possible query variations.

# v0.5.1

- Adds `Torus.Embeddings.Gemini` to support Gemini embeddings.
- Extends semantic search docs on how to stack embedders
- Adds `:distance_key` option to `Torus.semantic/5` to allow selecting distance key to the result map. Later on we'll rely on this to support hybrid search.
- Correctly swaps `>` and `<` operators for pre-filtering when changing order in `Torus.semantic/5` search.

# v0.5.0

- Similarity search type now defaults to `:word_similarity` instead of `similarity`.
- Possible `Torus.similarity/5` search types are updated to be prefixed with `similarity` to replicate 1-1 these in `pg_trgm` extension.
- Extended optimization section in the docs

# v0.4.1

Minor doc updates

# v0.4.0

## Breaking changes:

- `Torus.full_text/5` - now returns all results when search term contains a stop word or is empty instead of returning none.

## Improvements:

- `Torus.full_text/5` - now supports `:empty_return` option that controls if the query should return all results when search term contains a stop word or is empty.
- `Torus.QueryInspector.tap_explain_analyze/3` - now correctly returns the query plan.
- Docs were grouped together by the search type.

## New 🔥

**Semantic search** is finally here! Read more about it in the [Semantic search with Torus](/guides/semantic_search.md) guide.
Shortly - it allows you to generate embeddings using a configurable adapters and use them to compare against the ones stored in your database.

Supported adapters (for now):

- `Torus.Embeddings.OpenAI` - uses OpenAI's API to generate embeddings.

- `Torus.Embeddings.HuggingFace` - uses HuggingFace's API to generate embeddings.

- `Torus.Embeddings.LocalNxServing` - generate embeddings on your local machine using a variety of models available on Hugging Face

- `Torus.Embeddings.PostgresML` - uses PostgreSQL [PostgresML extension](https://PostgresML.org/docs) to generate embeddings

- `Torus.Embeddings.Batcher` - a long‑running **GenServer** that collects individual embedding calls, groups them into a single batch, and forwards the batch to the configured `embedding_module` (any from the above or your custom one).

- `Torus.Embeddings.NebulexCache` - a wrapper around [Nebulex](https://hexdocs.pm/nebulex/readme.html) cache, allowing you to cache the embedding calls in memory, so you save the resources/cost of calling the embedding module multiple times for the same input.

And you can easily create your own adapter by implementing the `Torus.Embedding` behaviour.

# v0.3.0

## Breaking changes:

- `full_text_dynamic/5` is renamed to `full_text/5` and now supports stored columns.
- `similarity/5` - `limit` option is removed, use Ecto's `limit/2` instead.
- `full_text/5` - `:concat` option is renamed to `:coalesce`.

## Improvements:

- `full_text/5` now supports stored `tsvector` columns.
- `Torus.QueryInspector.substituted_sql/3` now correctly handles arrays substitutions.
- Docs are extended to guide through the performance and relevance.

And other minor performance/clearance improvements.

# v0.2.2

- `full_text_dynamic/5`: Replaced `:nullable_columns` with `:concat` option
- `similarity/5`: Fixed a bug where you weren't able to pass variable as a term
- `Torus.QueryInspector`: now is not tied with `Torus.Testing` and serves as a separate standalone module.

And other minor performance/clearance improvements.

# v0.2.1

`similarity/5` search is now fully tested and customizable. `full_text_dynamic/5` is up next.

# Changelog for Torus v0.2.0

Torus now supports full text search, ilike, and similarity search.