Current section
Files
Jump to
Current section
Files
dowser_opensearch
README.md
README.md
# Dowser.OpenSearch
[](https://hex.pm/packages/dowser_opensearch)
An Elixir client for the [OpenSearch](https://opensearch.org) API, built on top
of [`Dowser.Client`](https://hex.pm/packages/dowser_client).
The library follows the [OpenSearch OpenAPI
specification](https://github.com/opensearch-project/opensearch-api-specification):
each of its tags maps to one module, and each endpoint to one function named
after the operation.
## Installation
Add `dowser_opensearch` to your dependencies in `mix.exs`:
```elixir
def deps do
[
{:dowser_opensearch, "~> 0.3.0"}
]
end
```
**Upgrading from 0.2?** 0.3 replaces contexts with repos, moves the casting to
`dowser_client` and casts only nine date formats — see the
[upgrade guide](guides/upgrading/v0.3.md), and [CHANGELOG.md](CHANGELOG.md)
for everything that changed.
Define a repo — one module per cluster, configured under your own application
and started in your supervision tree, like an `Ecto.Repo`:
```elixir
defmodule MyApp.Search do
use Dowser.OpenSearch.Repo, otp_app: :my_app
end
```
```elixir
# config/runtime.exs
config :my_app, MyApp.Search, endpoint: System.get_env("OPENSEARCH_URL", "http://localhost:9200")
# the repo used when a call names none
config :dowser_opensearch, default_repo: MyApp.Search
```
```elixir
# application.ex
children = [MyApp.Search]
```
Every API function takes the cluster as its `:repo` option — a repo module, or
`[endpoint: ...]` for a script — and falls back to `:default_repo` without one.
See `Dowser.OpenSearch.Repo` and `Dowser.Repo` for the configuration (auth,
headers, TLS, `:httpc` profiles, an `init/2` callback for secrets), checking at
startup that the endpoint really is OpenSearch (`verify: true`), and the HTTP
verbs a repo exposes for endpoints this package doesn't cover.
## Usage
Every API function is reached through its own module — there are no shortcuts on
the top-level Dowser.OpenSearch namespace:
```elixir
{:ok, info} = Dowser.OpenSearch.Info.info()
info["version"]["distribution"]
#=> "opensearch"
```
Each function comes in two variants: the plain one returns `{:ok, body}` or
`{:error, exception}`, and the bang one returns the body directly or raises:
```elixir
Dowser.OpenSearch.Search.search!(%{query: %{match_all: %{}}}, index: "posts")
```
Required OpenSearch attributes are positional arguments; everything optional —
including the transport options forwarded to `Dowser.Client` — lives in the
trailing keyword list. A request body is always the first argument, so calls
pipe:
```elixir
%{query: %{term: %{"status" => "published"}}}
|> Dowser.OpenSearch.Search.search(index: "posts", params: [size: 50])
```
`HEAD` checks come as a pair where the `?` variant plays the bang role:
```elixir
Dowser.OpenSearch.Index.index_exists("posts") #=> {:ok, true}
Dowser.OpenSearch.Index.index_exists?("posts") #=> true
```
## Modules
| Module | Tag |
| ---------------------------------------- | ---------------------- |
| `Dowser.OpenSearch.Alias` | Aliases |
| `Dowser.OpenSearch.Cat` | CAT |
| `Dowser.OpenSearch.Cluster` | Cluster |
| `Dowser.OpenSearch.DanglingIndices` | Dangling Indices |
| `Dowser.OpenSearch.DataStream` | Data Streams |
| `Dowser.OpenSearch.Document` | Document |
| `Dowser.OpenSearch.Index` | Index |
| `Dowser.OpenSearch.IndexSettings` | Index Settings |
| `Dowser.OpenSearch.IndexStateManagement` | Index State Management |
| `Dowser.OpenSearch.IndexTemplate` | Index Templates |
| `Dowser.OpenSearch.Info` | Info |
| `Dowser.OpenSearch.List` | List |
| `Dowser.OpenSearch.Mappings` | Mappings |
| `Dowser.OpenSearch.Reindex` | Reindex |
| `Dowser.OpenSearch.Search` | Search |
## Type casting
Wire `Dowser.OpenSearch.Codec` in as the decoder and encoder, and documents are
cast against their own index mapping — dates become `DateTime`, IPs become
`:inet` tuples, and so on:
```elixir
config :my_app, MyApp.Search,
endpoint: "http://localhost:9200",
decoder: Dowser.OpenSearch.Codec,
encoder: Dowser.OpenSearch.Codec
```
```elixir
# with "published_at" mapped {"type": "date", "format": "strict_date_time"}
Dowser.OpenSearch.Document.get!("posts", "1")["_source"]["published_at"]
#=> ~U[2026-08-11 00:00:00.000Z]
```
Mappings are fetched once and cached by `Dowser.OpenSearch.MappingCacher`. This
package doesn't start it: add it to your supervision tree, next to the repo —
```elixir
children = [MyApp.Search, Dowser.OpenSearch.MappingCacher]
```
— or give the mappings yourself, and skip the cluster entirely:
`config :dowser_opensearch, mapping: MyApp.Mappings` (a `Dowser.Mapping` module),
or `struct:` on a call (a document module's own mapping). Casting against the
cluster's mappings with no cacher started fails with a `Dowser.MappingError`
naming it, rather than leaving every value uncast.
A `flat_object` field, and an object mapped `"enabled": false`, are returned
exactly as stored.
### What is cast
The casting is `dowser_client`'s (`Dowser.Type`), the same on Elasticsearch and
OpenSearch. Only the types JSON can't represent are cast; every other value
passes through as decoded JSON:
| mapping type | stored | Elixir |
| ----------------------------- | ------------------------------------------ | ------------------------------------------- |
| `date`, `date_nanos` | depends on the format — see below | `Date`, `Time`, `DateTime`, `NaiveDateTime` |
| `date_range` (`strict_date`) | `{"gte": "2019-03-01", "lte": "2019-03-31"}` | `Date.Range` |
| `integer_range`, `long_range` | `{"gte": 1, "lte": 10}` | `Range` |
| `ip` | `"127.0.0.1"` | `{127, 0, 0, 1}` |
| `binary` | Base64 | raw binary |
| `geo_point` | `{"lat": 1.2, "lon": 3.4}` | `{1.2, 3.4}` |
#### Dates
A date field's `format` decides what it is cast to. Nine formats are, each
read by the Elixir function that parses it:
| format | stored | Elixir |
| --------------------------------------- | --------------------------------- | ----------------------------------------- |
| `epoch_millis` | `1553391286000` | `DateTime.from_unix(value, :millisecond)` |
| `epoch_second` | `1553391286` | `DateTime.from_unix(value, :second)` |
| `strict_date` | `"2019-03-23"` | `Date.from_iso8601/1` |
| `strict_date_hour_minute_second` | `"2019-03-23T21:34:46"` | `NaiveDateTime.from_iso8601/1` |
| `strict_date_hour_minute_second_millis` | `"2019-03-23T21:34:46.123"` | `NaiveDateTime.from_iso8601/1` |
| `strict_date_time_no_millis` | `"2019-03-23T21:34:46-04:00"` | `DateTime.from_iso8601/1` |
| `strict_date_time` | `"2019-03-23T21:34:46.123-04:00"` | `DateTime.from_iso8601/1` |
| `strict_hour_minute_second` | `"21:34:46"` | `Time.from_iso8601/1` |
| `strict_hour_minute_second_millis` | `"21:34:46.123"` | `Time.from_iso8601/1` |
- **Any other format is not cast** — nor is a field with no format at all,
which OpenSearch reads as `strict_date_optional_time||epoch_millis`. Its
values come back as stored, and are written as given. Map a date field with
one of the nine to have it cast. A value its format can't read (a leap
second, a malformed string) comes back as stored too.
- **Several formats** (`"strict_date||epoch_millis"`): a value is read with
the first that recognizes it, and written with the first.
- **Precision:** a value is read at whatever precision it carries, and written
at exactly the format's — truncated to the second or the millisecond, and
padded to three digits for a `_millis` format
(`~U[2019-03-24 01:34:46Z]` is written `"2019-03-24T01:34:46.000Z"` to
`strict_date_time`).
- **Always UTC:** an offset in a stored value is applied and dropped, giving
one UTC `DateTime`; a value with no offset, or `-00:00`, is read as UTC, as
OpenSearch reads it. A `DateTime` in any zone is written as its UTC instant
(`Z`). No time-zone database is needed, and `tzdata` changes nothing. The
instant is exact, but the original offset isn't kept: store the zone in a
field of its own where the local time matters.
#### Ranges
A range is cast only when the Elixir struct holds it exactly: a `date_range`
mapped `strict_date` with `gte`/`lte` dates, an `integer_range` or
`long_range` with integer `gte`/`lte` bounds, `gte` no greater than `lte`.
Anything else — an open-ended range, `gt`/`lt` bounds, another date format —
comes back as stored. Only a range stepping by 1 is written back as a range.
### The casting options
Five options decide how documents are cast. Each can be passed on any API
call, given to the codec in a repo's configuration
(`decoder: {Dowser.OpenSearch.Codec, types: MyApp.Types}`), or set for the
application (`config :dowser_opensearch, types: MyApp.Types`) — most specific
first:
| Option | Decides | Default |
| --- | --- | --- |
| `:mapping` | where the mapping comes from: a map, a `Dowser.Mapping` module, or a document module | `config :dowser_opensearch, mapping:`, else the cluster's via `Dowser.OpenSearch.MappingCacher`, which you start |
| `:struct` | load each document into a document module's struct (`Dowser.Schema`), and dump it back — per call only | maps |
| `:types` | the `Dowser.Type` module leaf values go through | `Dowser.Type` |
| `:unknown_fields` | `:keep`, or `:raise` a `Dowser.UnknownFieldError` for a field the mapping doesn't declare | `:keep` |
| `:mapping_failure` | `:error`, `:warn` or `:ignore` when the mapping can't be fetched | `:error` |
```elixir
Dowser.OpenSearch.Document.get("posts", "1", struct: MyApp.Post) # {:ok, %{"_source" => %MyApp.Post{}}}
Dowser.OpenSearch.Document.get("posts", "1", mapping: MyApp.Post) # cast with its mapping, still a map
```
`:mapping` and `:struct` turn casting on for that call even on a repo with no
`:decoder`/`:encoder` configured.
## Testing
A repo is where a test takes over: every request goes through it, and nothing
else needs replacing.
### Stub the HTTP layer, keep the pipeline
`Dowser.Client.HTTP.Stub` answers every request the test process makes, while
everything else — the repo, URL building, JSON, casting — runs as in
production. Point the calls at an ad-hoc repo (or your own, configured in
`config/test.exs`); stubs are per process, so tests stay `async: true`:
```elixir
test "loads a post" do
Dowser.Client.HTTP.Stub.stub(fn :get, "http://localhost:9200/posts/_doc/1", _headers, _body, _opts ->
Dowser.Client.HTTP.Stub.json(200, %{
"_index" => "posts",
"_id" => "1",
"_source" => %{"title" => "hi", "published_at" => "2026-08-11T00:00:00.000Z"}
})
end)
assert {:ok, %{"_source" => %MyApp.Post{title: "hi"}}} =
Dowser.OpenSearch.Document.get("posts", "1",
repo: [endpoint: "http://localhost:9200"],
struct: MyApp.Post
)
end
```
Casting against the cluster's mappings needs mappings in tests too: give them
with `config :dowser_opensearch, mappings: %{"posts" => %{...}}` (no cacher
needed), `mapping:` / `struct:`, or start `Dowser.OpenSearch.MappingCacher` with
`start_supervised!/1` and `put/3` the mappings into it.
### Mock the repo
For a unit test that only needs a canned answer, mock the repo itself: a repo
is the `Dowser.Repo` behaviour, so [Mox](https://hexdocs.pm/mox) can stand in
for one.
```elixir
# test/test_helper.exs
Mox.defmock(MyApp.SearchMock, for: Dowser.Repo)
# in your test
Mox.expect(MyApp.SearchMock, :request, fn :post, "/posts/_search", _query, _opts ->
{:ok, %Dowser.Client.Response{status: 200, headers: %{}, body: %{"hits" => %{"hits" => []}}}}
end)
assert {:ok, %{"hits" => %{"hits" => []}}} =
Dowser.OpenSearch.Search.search(%{query: %{match_all: %{}}}, index: "posts", repo: MyApp.SearchMock)
```
The mock stands in for the whole transport — encoding, decoding and casting
included — so its body is exactly what the caller gets. Set
`config :dowser_opensearch, default_repo: MyApp.SearchMock` in `config/test.exs`
to have every call that names no repo go to it.
### Mock the repo with Mimic
[Mimic](https://hexdocs.pm/mimic) stubs the functions of an existing module,
per test process. A test-only repo module becomes the default repo in the test
environment, so the code under test — which names no repo — talks to it, and
each test says what `request/4` answers. `request/4` is the one function every
API call goes through.
```elixir
# mix.exs
defp deps do
[
# ...
{:mimic, "~> 1.10", only: :test}
]
end
defp elixirc_paths(:test), do: ["lib", "test/support"]
defp elixirc_paths(_env), do: ["lib"]
```
```elixir
# test/support/search_mock.ex — a repo like any other, compiled for tests only
defmodule MyApp.SearchMock do
use Dowser.OpenSearch.Repo, otp_app: :my_app
end
```
```elixir
# config/test.exs — the default repo is the mock in tests
config :my_app, MyApp.SearchMock, endpoint: "http://search.test"
config :dowser_opensearch, default_repo: MyApp.SearchMock
```
```elixir
# test/test_helper.exs
Mimic.copy(MyApp.SearchMock)
ExUnit.start()
```
```elixir
defmodule MyApp.PostsTest do
use ExUnit.Case, async: true
use Mimic
test "searches the posts" do
expect(MyApp.SearchMock, :request, fn :post, "/posts/_search", _query, _opts ->
{:ok, %Dowser.Client.Response{status: 200, headers: %{}, body: %{"hits" => %{"hits" => []}}}}
end)
# MyApp.Posts.search/1 calls Dowser.OpenSearch.Search.search(query, index: "posts"),
# with no :repo — so it goes to the default repo, the mock.
assert {:ok, %{"hits" => %{"hits" => []}}} = MyApp.Posts.search("hello")
end
end
```
A few things to know:
- The stub stands in for the whole transport, so the body it returns is what
the caller gets — nothing is decoded or cast. Let the request through with
`call_original/3` where the casting matters, or stub the HTTP layer instead.
- Mimic stubs are per process, which keeps `async: true` tests apart. A request
made from another process — a `Task`, a GenServer — needs
`Mimic.allow(MyApp.SearchMock, self(), pid)`, or `set_mimic_global` in a test
that isn't async.
- Mimic replaces calls *into* the module. The repo's own verbs
(`MyApp.SearchMock.get/2`, …) reach `request/4` from inside it, so a test
calling them directly stubs the verb it calls.
- Calls that name a repo (`repo: MyApp.Search`) bypass the default: point
`repo:` at the same mock in tests, or have the code read its repo from
configuration.
### Against a real cluster
Point a repo at a local cluster in `config/test.exs`, and start the mapping
cacher if the tests cast against the cluster's mappings. Tests that share an
index can't run concurrently: run them `async: false`, or give each test
indices of its own (`"posts_#{System.unique_integer([:positive])}"`), created
in `setup` and deleted in `on_exit`.
## Streaming
`Dowser.OpenSearch.Streamer` walks more documents than fit in one response,
over a point in time:
```elixir
%{query: %{match_all: %{}}, sort: [%{"_id" => "asc"}]}
|> Dowser.OpenSearch.Streamer.stream(index: "posts")
|> Stream.map(& &1["_source"])
|> Enum.each(&process/1)
```
The `sort` is required and its last key must be unique per document —
OpenSearch has no `_shard_doc` to supply a tiebreaker, and `search_after` on a
non-unique key silently skips rows. See the module documentation.
## Repositories
`Dowser.OpenSearch.Repository` binds the index-related functions to a fixed or
computed index:
```elixir
defmodule Posts do
use Dowser.OpenSearch.Repository, index: "posts", only: [search: [:search], document: [:get]]
end
Posts.search!(%{query: %{match_all: %{}}}) # searches /posts/_search
Posts.get_doc!("1") # GETs /posts/_doc/1
```
## Development
```sh
mix deps.get
mix test
mix lint # format --check-formatted, credo --strict, dialyzer
```
This package is developed alongside `dowser_client`. To build against a
working copy of it rather than the published release — needed whenever this
package relies on a `dowser_client` change not on Hex yet — set
`DOWSER_CLIENT_PATH`:
```sh
DOWSER_CLIENT_PATH=../dowser_client mix deps.get
DOWSER_CLIENT_PATH=../dowser_client mix test
```
## Documentation
The full API reference is on
[HexDocs](https://hexdocs.pm/dowser_opensearch).
## License
MIT — see
[LICENSE.txt](https://github.com/GRoguelon/dowser_opensearch/blob/main/LICENSE.txt).