Current section
Files
Jump to
Current section
Files
lib/hare_mq.ex
defmodule HareMq do
@moduledoc """
HareMq is an Elixir library for interacting with AMQP systems such as RabbitMQ.
It provides supervised connection management, queue/exchange topology declaration,
message publishing with optional deduplication, message consumption with automatic
retry/dead-letter routing, stream queue support, and dynamic consumer scaling with
an optional auto-scaler.
See the [README](https://github.com/Dimakoua/hare_mq) for full documentation.
---
## Publisher
```elixir
defmodule MyApp.MessageProducer do
use HareMq.Publisher,
routing_key: "routing_key",
exchange: "exchange"
def send_message(message), do: publish_message(message)
end
```
With deduplication:
```elixir
defmodule MyApp.MessageProducer do
use HareMq.Publisher,
routing_key: "routing_key",
exchange: "exchange",
unique: [
period: :infinity, # TTL in ms or :infinity
keys: [:project_id] # deduplicate by these map keys
]
def send_message(message), do: publish_message(message)
end
```
`publish_message/1` returns `:ok`, `{:error, :not_connected}`,
`{:error, {:encoding_failed, reason}}`, or `{:duplicate, :not_published}`.
---
## Consumer
```elixir
defmodule MyApp.MessageConsumer do
use HareMq.Consumer,
queue_name: "queue_name",
routing_key: "routing_key",
exchange: "exchange"
def consume(message) do
IO.puts("Received: \#{inspect(message)}")
:ok # :ok | {:ok, any()} to ack; :error | {:error, any()} to retry
end
end
```
---
## Batch Consumer
Batch processing allows processing multiple messages at once. Set `batch_size` and `batch_timeout_ms` to enable it.
```elixir
defmodule MyApp.BatchConsumer do
use HareMq.Consumer,
queue_name: "my_queue",
batch_size: 50,
batch_timeout_ms: 2000
def consume(messages, :batch) do
# messages is a list of decoded payloads
IO.puts("Received batch of \#{length(messages)} messages")
:ok
end
end
```
Batch processing is fully compatible with **retry**, **delay cascade**,
**auto-scaling**, and **stream queues**. Each message in a batch is
individually acknowledged or retried based on the return value of `consume/2`.
---
## Stream Consumer
Stream queues are persistent, append-only logs. Each consumer reads at its own
offset ā messages are never removed after consumption.
```elixir
defmodule MyApp.EventLog do
use HareMq.Consumer,
queue_name: "domain.events",
stream: true,
stream_offset: "first" # replay from the beginning
def consume(message) do
IO.inspect(message)
:ok
end
end
```
`stream_offset` options: `"next"` (default), `"first"`, `"last"`,
an integer offset, or a `%DateTime{}`.
When `stream: true`:
- Only the stream queue is declared (`x-queue-type: stream`).
- No delay or dead-letter queues are created.
- Messages are always acked regardless of the `consume_fn` return value.
> **Prerequisite:** enable the `rabbitmq_stream` plugin on your broker:
> `rabbitmq-plugins enable rabbitmq_stream`
---
## Dynamic Consumer
```elixir
defmodule MyApp.MessageConsumer do
use HareMq.DynamicConsumer,
queue_name: "queue_name",
routing_key: "routing_key",
exchange: "exchange",
consumer_count: 10
def consume(message) do
IO.puts("Received: \#{inspect(message)}")
:ok
end
end
```
With auto-scaling:
```elixir
defmodule MyApp.MessageConsumer do
use HareMq.DynamicConsumer,
queue_name: "queue_name",
routing_key: "routing_key",
exchange: "exchange",
consumer_count: 2,
auto_scaling: [
min_consumers: 1,
max_consumers: 20,
messages_per_consumer: 100,
check_interval_ms: 5_000
]
def consume(message), do: :ok
end
```
---
## Usage in Application
```elixir
defmodule MyApp.Application do
use Application
def start(_type, _args) do
children = [
MyApp.MessageConsumer,
MyApp.MessageProducer
]
opts = [strategy: :one_for_one, name: MyApp.Supervisor]
Supervisor.start_link(children, opts)
end
end
```
---
## Configuration
```elixir
config :hare_mq, :amqp,
url: "amqp://guest:guest@localhost:5672"
config :hare_mq, :configuration,
delay_in_ms: 10_000,
retry_limit: 15,
message_ttl_ms: 31_449_600,
reconnect_interval_ms: 10_000
config :hare_mq, :auto_scaler,
min_consumers: 1,
max_consumers: 20,
messages_per_consumer: 10,
check_interval_ms: 5_000
```
All values are read at runtime via `Application.get_env`, so
`Application.put_env` in tests takes effect without recompilation.
---
## Telemetry
HareMq emits [Telemetry](https://hexdocs.pm/telemetry) events throughout its lifecycle.
Attach handlers with `:telemetry.attach_many/4`:
:telemetry.attach_many(
"my-app-hare-mq",
[
[:hare_mq, :connection, :connected],
[:hare_mq, :connection, :disconnected],
[:hare_mq, :connection, :reconnecting],
[:hare_mq, :consumer, :message, :stop],
[:hare_mq, :retry_publisher, :message, :dead_lettered]
],
fn event, measurements, metadata, _config ->
require Logger
Logger.info("[hare_mq] \#{inspect(event)} \#{inspect(measurements)} \#{inspect(metadata)}")
end,
nil
)
### Connection events
| Event | When |
|---|---|
| `[:hare_mq, :connection, :connected]` | Broker connection opened |
| `[:hare_mq, :connection, :disconnected]` | Monitored connection process went down |
| `[:hare_mq, :connection, :reconnecting]` | Reconnect attempt scheduled |
### Consumer events
| Event | When |
|---|---|
| `[:hare_mq, :consumer, :connected]` | Channel open and `Basic.consume` called |
| `[:hare_mq, :consumer, :message, :start]` | `consume/1` callback is about to be invoked |
| `[:hare_mq, :consumer, :message, :stop]` | `consume/1` returned; stop metadata includes `:result` (`:ok`/`:error`) and `:duration` |
| `[:hare_mq, :consumer, :message, :exception]` | `consume/1` raised an exception |
The `:start`/`:stop`/`:exception` events follow the standard `:telemetry.span/3` contract.
### Publisher events
| Event | When |
|---|---|
| `[:hare_mq, :publisher, :connected]` | Publisher channel opened |
| `[:hare_mq, :publisher, :message, :published]` | Message published successfully |
| `[:hare_mq, :publisher, :message, :not_connected]` | Publish attempted without a channel |
### Retry publisher events
| Event | When |
|---|---|
| `[:hare_mq, :retry_publisher, :message, :retried]` | Failed message sent to a delay queue; measurements include `:retry_count` |
| `[:hare_mq, :retry_publisher, :message, :dead_lettered]` | Message exceeded `retry_limit` and moved to dead-letter queue |
See `HareMq.Telemetry` for the full measurements/metadata reference for each event.
## Rate Us:
If you enjoy using HareMq, please consider giving us a star on GitHub! Your feedback and support are highly appreciated.
[GitHub](https://github.com/Dimakoua/hare_mq)
"""
use Application
require Logger
@doc """
Starts the HareMq OTP application.
Brings up two supervised children:
- `HareMq.Connection` ā manages the default AMQP connection.
- `HareMq.DedupCache` ā ETS-backed deduplication cache used by publishers with `unique:` set.
Both are started under a `:one_for_one` supervisor.
If you need additional connections (e.g. multiple vhosts) start named
`HareMq.Connection` instances separately in your own supervision tree.
"""
def start(_type, _args) do
children = [
HareMq.Connection,
HareMq.DedupCache
]
opts = [strategy: :one_for_one, name: HareMq.Supervisor]
Supervisor.start_link(children, opts)
end
end