ex_abby

# ExAbby

**ExAbby** is a minimal A/B testing library for Elixir/Phoenix.  

*Caveat: This was created primarily over a weekend with the help of Chat GPT/Claude. The code is working but still needs a lot of cleanup and optimizations, which I'll do as I run into problems.   As it stands, it is working under low load in production settings providing a super easy way to AB test Phoenix and Liveview using assigns.*


## Why Ex Abby? 
I have found there are no super simple ways to get ab testing working for smaller sites in Elixir. You have to pay $$ or use a complex system.  And everything has moved to feature tagging. This experiment framework is based on something we built in-house for a previous company that reached virality co-efficients of 1.0 a few times.  And the goal is to make it super easy to use in Liveview environments. 

This is really early and the API is 100% likely to change. Feedback is appreciated! 

It supports:

- Ecto-based storage (Experiments, Variations, Trials)
- Session-based or User-based assignment
- **Linking session trials to users** - Track user performance from session-based experiments
- **Archive experiments** with optional winner declaration
- Flexible ID support: structs, integers, or strings
- Weighted randomization
- Recording success events
- LiveView helpers (checking `connected?/1` and storing assigned variation)
- Admin LiveViews with experiment filtering (Active/Archived/All)
- Upserting experiments/variations with optional weight updates
- Reviewing results over different time periods
- Control-relative p-values for experiments with multiple treatments
- Ability to toggle variations by user or session for testing

Coming in the future
- armed bandits
- optimizations / caching
- better UX of admin screens
- So much cleanup
- Likely changes to the API.


---

## Table of Contents
1. [Installation](#installation)
2. [Configuration](#configuration)
3. [Migrations](#migrations)
4. [Upgrading from v0.1 to v0.2](#upgrading-from-v01-to-v02)
5. [Upserting Experiments and Updating Weights](#upserting-experiments-and-updating-weights)
6. [Session Setup](#session-setup)
7. [Admin Routes](#admin-routes)
8. [Usage in Controllers](#usage-in-controllers)
9. [Usage in LiveView](#usage-in-liveview)
10. [Linking Sessions to Users](#linking-sessions-to-users)
11. [Archiving Experiments](#archiving-experiments)
12. [Troubleshooting](#troubleshooting)

---

## Installation

1. **Add** `ex_abby` as a dependency in your Phoenix (host) app’s `mix.exs`. 

   If it's a public Hex package (or if you plan to publish it):
   ```elixir
   defp deps do
     [
       {:ex_abby, "~> 0.4.0"}
     ]
   end
   ```

   If it's a **GitHub** repo (private or public):
   ```elixir
   defp deps do
     [
       {:ex_abby, github: "grahac/ex_abby", tag: "v0.4.0"}
     ]
   end
   ```

2. Run:
   ```bash
   mix deps.get
   ```

---

## Configuration

In your host app’s `config/config.exs` (or `dev.exs`, etc.), set:

```elixir
config :ex_abby,
  repo: MyApp.Repo,
  control_variation_name: "control",
  bot_detection: [
    enabled: true,
    detectors: [ExAbby.BotDetector.UserAgent],
    fallback_variation: "control"
  ]
```

Where `MyApp.Repo` is your **Ecto Repo** module.

`control_variation_name` identifies the variation used as the baseline for
significance comparisons in the admin results view. It defaults to `"control"`
when omitted.

Bot detection is enabled by default. The built-in detector classifies common
search, AI, social-preview, monitoring, SEO, and generic crawler user agents.
It is intentionally user-agent based: applications can add request-aware
detectors for Cloudflare or other signals, but custom detectors must return a
stable atom such as `{:bot, :cloudflare_verified_bot}` and must never return a
user agent, IP address, or other request-derived value.

---
## Migrations

ExAbby provides Ecto migrations that create three tables:

- `ex_abby_experiments`
- `ex_abby_variations`
- `ex_abby_trials`

In your host app, generate a new migration:

```bash
mix ecto.gen.migration create_ex_abby_tables
```

Open `priv/repo/migrations/2025xxxxxx_create_ex_abby_tables.exs`, and **add**:

```elixir
defmodule MyApp.Repo.Migrations.CreateExAbbyTables do
  use Ecto.Migration

  def up do
    ExAbby.Migrations.create_tables()
  end

  def down do
    ExAbby.Migrations.drop_tables()
  end
end
```

Then run:

```bash
mix ecto.migrate
```

---

## Upgrading from v0.1 to v0.2

Version 0.2.0 adds experiment archiving with optional winner declaration. To upgrade:

1. **Update your dependency** in `mix.exs`:
```elixir
{:ex_abby, "~> 0.2.0"}
```

2. **Create a new migration**:
```bash
mix ecto.gen.migration ex_abby_v2
```

3. **Add to the generated migration file**:
```elixir
defmodule MyApp.Repo.Migrations.ExAbbyV2 do
  use Ecto.Migration

  def up, do: ExAbby.Migrations.v1_to_v2()
  def down, do: ExAbby.Migrations.v2_to_v1()
end
```

4. **Run the migration**:
```bash
mix ecto.migrate
```

This adds two new columns to `ex_abby_experiments`:
- `archived_at` - Timestamp when experiment was archived
- `winner_variation_id` - Reference to the winning variation (optional)

---

## Upgrading to trial exclusion fields

Bot-exclusion audit fields are added with the next ExAbby migration:

```elixir
defmodule MyApp.Repo.Migrations.ExAbbyBotExclusion do
  use Ecto.Migration

  def up, do: ExAbby.Migrations.v2_to_v3()
  def down, do: ExAbby.Migrations.v3_to_v2()
end
```

The migration adds nullable `excluded_at` and `exclusion_reason` fields to
`ex_abby_trials`; it preserves rows for auditing and restoration.

---

## Upserting Experiments and Updating Weights

If you have a function like:

```elixir
ExAbby.upsert_experiment_and_update_weights(
  "landing_page_test",
  "Testing different landing pages",
  [
    {"Original", 1.0},
    {"Variation A", 1.0},
    {"Variation B", 2.0}
  ],
  success1_label: "Signup",
  success2_label: "Purchase"
)

```


Then:

- If `"landing_page_test"` **does not exist**, the library creates a new experiment with that name + description, and 3 variations with the specified weights.  
- If the experiment **already exists**, we do **not** change its weights. We update all the other info if
- you can optionally add labels to label success. This is just for readability and is optional.
---


## Seeding Experiments

Create a file `priv/repo/seeds/experiments.exs` to define your experiments:

```elixir
experiments = [
  {
    "button_color_test",
    "Testing different button colors for signup",
    [
      {"control", 0.33},
      {"green", 0.33}, 
      {"blue", 0.33}
    ],
    [success1_label: "Signup", success2_label: "Purchase", update_weights: false]
  }
]

# Seed or update experiments without modifying weights
Enum.each(experiments, fn {name, description, variations, opts} ->
  ExAbby.upsert_experiment_and_update_weights(name, description, variations, opts)
end)
```

Then in your `priv/repo/seeds.exs`, add:

```elixir
Code.require_file("seeds/experiments.exs", __DIR__)
```

You can run the seeds in different ways:

### Development:
```bash
mix run priv/repo/seeds.exs
```

---
### Session Setup

To enable session-based A/B testing, add `ExAbby.SessionPlug` to your endpoint or router pipeline:


```elixir
# In  your router pipeline (recommended):

pipeline :browser do
  # ... other plugs ...
  plug ExAbby.SessionPlug
end

# In lib/your_app_web/endpoint.ex
plug Plug.Session,
  store: :cookie,
  key: "_your_app_key",
  signing_salt: "your_signing_salt"

plug ExAbby.SessionPlug

```


This plug creates a unique `"ex_abby_session_id"` for tracking A/B test variations across requests. It also evaluates bot detection on every request and stores only compact `:human` or `{:bot, bot_name}` status in `conn.assigns[:ex_abby_bot]` and the session's `"ex_abby_bot"` key. It never persists a raw user agent or IP address.

### Bot behavior and custom detectors

For bot requests, the controller and LiveView assignment helpers return the
configured fallback variation without creating or changing a trial. Each such
assignment emits `[:ex_abby, :assignment, :excluded]` telemetry with the
experiment, `:bot` reason, and static bot name only.

To add a detector, implement `detect/1` against
`ExAbby.BotDetector.Context`, which includes transient `user_agent` and an
optional `conn`:

```elixir
defmodule MyApp.BotDetector do
  def detect(%ExAbby.BotDetector.Context{conn: conn}) do
    if conn.assigns[:verified_bot], do: {:bot, :verified_bot}, else: :human
  end
end

config :ex_abby,
  bot_detection: [detectors: [ExAbby.BotDetector.UserAgent, MyApp.BotDetector]]
```

The stored and emitted bot name must be a stable atom; invalid custom detector
responses are treated as `:human` so request evidence cannot leak into the
session or telemetry.

### Excluding historical trials

Use the batch APIs only from privileged server-side maintenance code. They are
transactional and idempotent, retain the original variation, and can be safely
reversed:

```elixir
{:ok, excluded_count} = ExAbby.exclude_session_trials(session_ids, reason: "bot")
{:ok, restored_count} = ExAbby.restore_session_trials(session_ids)
```

Excluded trials are omitted from experiment summaries, conversion rates, and
significance calculations, while the admin experiment view displays their
total separately. An eligible human request restores only the trial for the
experiment it revisits, preserving unrelated exclusions.

`ExAbby.Experiments` persistence functions and public raw session-ID/user-ID
overloads do not have request bot status and therefore bypass this eligibility
check. Normal Phoenix and LiveView integrations should use the `Plug.Conn` and
socket public APIs after `ExAbby.SessionPlug`; host code that uses low-level
assignment must perform bot-first eligibility itself.
---


## Admin Routes

ExAbby includes a simple admin interface for viewing and managing experiments. To use it:

1. Add the routes to your router:

```elixir
defmodule MyAppWeb.Router do
  use MyAppWeb, :router
  import ExAbby.Router  # add this line 

  scope "/admin", MyAppWeb do
    pipe_through [:browser, :admin_auth]
    ex_abby_admin_routes()
  end
end
```

2. Visit `/admin/ab_tests` to see a clean, Tailwind-styled interface showing:
   - List of all experiments
   - Experiment details and descriptions
   - Quick links to view individual experiments

### Statistical significance

The experiment results view compares every treatment with the variation named
by `control_variation_name`. It calculates two-sided, anytime-valid p-values by
inverting beta-binomial mixture confidence sequences for the binary conversion
rate in each arm. The calculation uses unique converters and trials in the
selected date range; excluded trials do not contribute to either count.

ExAbby applies Holm's adjustment to reduce false positives from making several
comparisons. Adjusted values below 0.05 are highlighted. The correction family
is **every configured treatment arm across every displayed success metric**,
including any arm that currently has no data. Either success metric can identify
a winner; both do not need to be significant. The family is deliberately fixed
by the experiment's design rather than by how many arms happen to have data at
the moment you look: sizing it from the arms that are currently measurable would
make the correction depend on the observed data, and would make p-values jump
whenever an unrelated empty arm recorded its first trial. The cost is that a
permanently empty arm — one with weight `0`, or one created after the selected
range began — still consumes a comparison for each displayed metric.

Each treatment also shows its absolute lift and a pairwise 95% anytime-valid
confidence interval. **These intervals are not Holm-adjusted.** They answer "how
large is this arm's lift" rather than "which arms survive the family-wise
decision", so with several treatments an interval can exclude zero while the
same row's adjusted p-value sits above 0.05. Read the highlighted p-value for
the decision and the interval for the magnitude.

Sparse conversion counts produce wide intervals rather than unreliable
approximations; `Not enough data` appears only when the control or treatment has
no eligible trials.

The p-values and confidence intervals remain valid when results are checked
repeatedly and the experiment is stopped based on them. This guarantee assumes
the experiment's start date, metric, and eligibility rules were fixed
independently of the observed results. Trying several date ranges or metrics and
reporting only the best result is a separate multiple-testing problem.


---

## Usage in Controllers

ExAbby supports multiple ways to identify users and sessions:

- **Plug.Conn** - For session-based experiments in controllers
- **Phoenix.LiveView.Socket** - For session-based experiments in LiveView
- **User struct** - Any struct/map with an `id` field (e.g., `%{id: 123}`)
- **Integer ID** - Pass user ID directly (e.g., `123`)
- **String ID** - Pass session ID directly (e.g., `"session_abc123"`)

### Session-based Example

In a controller action (e.g., `PageController`):

```elixir
def index(conn, _params) do
  # Single variation example
  {conn, _variation} = ExAbby.get_variation(conn, "landing_page_test")
  
  # Multiple variations example
  {conn, _variations} = ExAbby.get_variations(conn, ["landing_page_test", "button_color_test"])
  render(conn, "index.html")
end

def record_conversion(conn, _params) do
  # Single experiment success recording
  ExAbby.record_success(conn, "landing_page_test")
  
  # Multiple experiment success recording
  ExAbby.record_successes(conn, ["landing_page_test", "button_color_test"])
  
  # Record with options (works for both single and multiple)
  ExAbby.record_successes(conn, ["landing_page_test", "button_color_test"], 
    amount: 99.99,
    success_type: :success1
  )
  
  redirect(conn, to: "/thank_you")
end
```

### User-based Example

If you have a `current_user`:

```elixir
def show(conn, _params) do
  user = conn.assigns.current_user
  
  # Single variation
  variation = ExAbby.get_variation(user, "dashboard_experiment")
  
  # Multiple variations
  variations = ExAbby.get_variations(user, ["dashboard_experiment", "feature_test"])
  render(conn, "show.html", ab_variations: variations)
end

def record_dashboard_success(conn, _params) do
  user = conn.assigns.current_user
  
  # Record multiple successes
  ExAbby.record_successes(user, ["dashboard_experiment", "feature_test"])
  redirect(conn, to: "/thanks")
end
```

### Direct ID Usage

ExAbby now supports passing IDs directly without wrapping in a struct:

```elixir
# Using integer user IDs directly
user_id = 12345
variation = ExAbby.get_variation(user_id, "experiment_name")
variations = ExAbby.get_variations(user_id, ["exp1", "exp2"])

# Record success with user ID
ExAbby.record_success(user_id, "experiment_name")
ExAbby.record_successes(user_id, ["exp1", "exp2"], amount: 99.99)

# Using session IDs directly (strings)
session_id = "abc123xyz"
variation = ExAbby.get_variation(session_id, "experiment_name")
variations = ExAbby.get_variations(session_id, ["exp1", "exp2"])

# Record success with session ID
ExAbby.record_success(session_id, "experiment_name")
ExAbby.record_successes(session_id, ["exp1", "exp2"], success_type: :success2)
```

This is useful when:
- You only have the user ID (not the full user struct)
- You're working with session IDs from external systems
- You want to run A/B tests in background jobs or processes without full context


## Usage in LiveView

1. **Ensure** your endpoint/pipeline sets up a session and calls `ExAbby.SessionPlug` or something similar to create `"ex_abby_session_id"`.
2. In your LiveView:

```elixir
defmodule MyAppWeb.ButtonTestLive do
  use MyAppWeb, :live_view

  def mount(_params, session, socket) do
    # Get multiple variations at once
    socket = ExAbby.get_variations(socket, session, ["landing_page_test", "button_color_test"])
    {:ok, assign(socket, session: session)}
  end

  def render(assigns) do
    ~H"""
    <div class="max-w-md mx-auto mt-10 p-6 bg-white rounded-lg shadow-lg">
      <%= case @ex_abby_trials["landing_page_test"] do %>
        <% "hello_world" -> %>
          <div>Hello World!</div>
        <% _ -> %>
          <div>This is the control</div>
      <% end %>

      <button 
        phx-click="convert" 
        class={get_button_class(@ex_abby_trials["button_color_test"])}
      >
        Click Me!
      </button>
    </div>
    """
  end

  def handle_event("convert", _params, socket) do
    case ExAbby.record_successes(socket, ["landing_page_test", "button_color_test"]) do
      {:ok, _trial} ->
        {:noreply, put_flash(socket, :info, "Conversion recorded!")}
      {:error, _reason} ->
        {:noreply, put_flash(socket, :error, "Failed to record conversion")}
    end
  end

  # Helper function for button styling based on variation
  defp get_button_class("blue"), do: "bg-blue-500 text-white rounded hover:bg-blue-600"
  defp get_button_class("green"), do: "bg-green-500 text-white rounded hover:bg-green-600"
  defp get_button_class(_), do: "bg-gray-500 text-white rounded hover:bg-gray-600"
end
```

The variations are stored in `@ex_abby_trials` as a map where:
- Keys are experiment names (e.g., `"landing_page_test"`)
- Values are variation names (e.g., `"hello_world"`, `"control"`)

### Recording Conversions with Options

You can record conversions with additional options:

```elixir
# Record a conversion with an amount
ExAbby.record_success(socket, "button_color_test",
  amount: 100.0,
  success_type: :success2
)

# Record multiple conversions at once
ExAbby.record_successes(socket, ["landing_page_test", "button_color_test"])
```

Available options:
- `:amount` - Optional numeric value to track with the success (default: 0.0)
- `:success_type` - Type of success to record, either `:success1` or `:success2` (default: `:success1`)

---

## Linking Sessions to Users

When a user signs up or logs in, you can link their session-based experiment trials to their user account. This allows you to track how users who signed up through different experiment variations perform over time.

### Why Link Sessions to Users?

Session-based experiments are great for testing landing pages and signup flows. But once a user creates an account, you want to track their long-term behavior (purchases, engagement, retention) tied to the original experiment variation they saw.

### Usage

**In Controllers (Plug.Conn):**
```elixir
def create(conn, %{"user" => user_params}) do
  case Accounts.create_user(user_params) do
    {:ok, user} ->
      # Link all session experiments to the new user
      conn = ExAbby.link_session_to_user(conn, user)

      # Or link specific experiments only
      conn = ExAbby.link_session_to_user(conn, user, ["signup_flow_test", "landing_page_test"])

      conn
      |> put_flash(:info, "Account created!")
      |> redirect(to: "/dashboard")

    {:error, changeset} ->
      render(conn, "new.html", changeset: changeset)
  end
end
```

**In LiveView:**
```elixir
def handle_event("register", %{"user" => user_params}, socket) do
  case Accounts.create_user(user_params) do
    {:ok, user} ->
      # Link all session experiments to the new user
      socket = ExAbby.link_session_to_user(socket, user)

      # Or link specific experiments
      socket = ExAbby.link_session_to_user(socket, user, ["signup_flow_test"])

      {:noreply, push_navigate(socket, to: "/dashboard")}

    {:error, changeset} ->
      {:noreply, assign(socket, changeset: changeset)}
  end
end
```

### Flexible User Identification

You can pass the user in different ways:
```elixir
# Pass user struct (must have :id field)
ExAbby.link_session_to_user(conn, user)

# Pass user ID directly
ExAbby.link_session_to_user(conn, 12345)
```

### What Happens When You Link

1. The session trial's `user_id` field is updated to the provided user ID
2. Future success recordings for that user will be associated with the original variation
3. You can now analyze user-based metrics (lifetime value, retention) by experiment variation

---

## Archiving Experiments

Once an experiment has concluded and you've determined a winner (or decided to end it), you can archive it. Archived experiments:

- **Stop accepting new trials** - New users won't be assigned to the experiment
- **Preserve existing data** - All historical trials and conversions remain
- **Can declare a winner** - Optionally mark which variation won
- **Are hidden by default** - Admin UI shows Active experiments by default

### Archiving via Admin UI

1. Navigate to `/admin/ex_abby/:id` (experiment detail page)
2. Select an optional winner from the dropdown
3. Click "Archive Experiment"

To unarchive, click the "Unarchive" button on the archived experiment.

### Archiving via Seeds

You can archive experiments directly in your seeds file:

```elixir
experiments = [
  # Active experiment
  {
    "current_test",
    "Currently running experiment",
    [{"control", 0.5}, {"variant", 0.5}],
    [success1_label: "Signup", update_weights: false]
  },

  # Archived experiment with winner
  {
    "old_test",
    "Completed experiment",
    [{"control", 0.5}, {"variant", 0.5}],
    [archived: true, winner: "variant", update_weights: false]
  }
]

Enum.each(experiments, fn {name, description, variations, opts} ->
  ExAbby.upsert_experiment_and_update_weights(name, description, variations, opts)
end)
```

**Important:** If you archive an experiment via the Admin UI and then run seeds without `archived: true`, the experiment will **remain archived**. Seeds only modify the archived status when explicitly specified.

### Archiving Programmatically

```elixir
# Archive without winner
ExAbby.Experiments.archive_experiment(experiment_id)

# Archive with winner (by variation name)
ExAbby.Experiments.archive_experiment(experiment_id, "variant_a")

# Archive with winner (by variation ID)
ExAbby.Experiments.archive_experiment(experiment_id, 123)

# Unarchive
ExAbby.Experiments.unarchive_experiment(experiment_id)
```

### Filtering Experiments

```elixir
# List only active experiments (default in Admin UI)
ExAbby.Experiments.list_experiments(status: :active)

# List only archived experiments
ExAbby.Experiments.list_experiments(status: :archived)

# List all experiments
ExAbby.Experiments.list_experiments(status: :all)
```

---

## Troubleshooting

- **`No Ecto repo configured for :ex_abby`**  
  Add `config :ex_abby, repo: MyApp.Repo` in your host app’s `config.exs`.

- **No Experiment Found**  
  If you see a warning for no experiment found, make sure you have seeded the database wtih experiments nd variations.

---

**Enjoy A/B testing with ExAbby!** Feel free to customize it further for bandit algorithms, Bayesian stats, or other advanced features.