ruvector_elixir

# RuvectorElixir

[![Hex.pm](https://img.shields.io/hexpm/v/ruvector_elixir.svg)](https://hex.pm/packages/ruvector_elixir)
[![HexDocs](https://img.shields.io/badge/hex-docs-purple.svg)](https://hexdocs.pm/ruvector_elixir)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Official Elixir bindings for [**ruvector**](https://github.com/ruvnet/ruvector), an ultra-fast embedded vector database and GraphRAG metadata engine written in Rust.

`RuvectorElixir` allows you to store, index, and query high-dimensional vector embeddings with sub-millisecond approximate nearest neighbor (ANN) search directly from BEAM applications—without external services, daemon processes, or network overhead.

---

## Features

- **Embedded & Zero-Dependency**: Embedded vector storage with zero external database dependencies. Runs in-process inside your BEAM node.
- **High-Performance NIF**: Built with [Rustler](https://github.com/rusterlium/rustler). Heavy computation and disk I/O are safely executed on Erlang dirty schedulers (`DirtyCpu` and `DirtyIo`) to prevent scheduler starvation.
- **Multiple Distance Metrics**:
  - `:cosine` (Cosine Similarity)
  - `:euclidean` (Euclidean / L2 Distance)
  - `:dot_product` (Dot Product)
  - `:manhattan` (Manhattan / L1 Distance)
- **HNSW & Flat Indexing**: Graph-based Hierarchical Navigable Small World (HNSW) indexing for sub-millisecond retrieval on large datasets, plus exact flat indexing.
- **Rich JSON Metadata Filtering**: Filter vector search results using arbitrary JSON-serializable key-value metadata.
- **ACID Persistence**: Crash-safe on-disk embedded database files (`.rvf`).
- **Idiomatic Elixir**: First-class structs (`RuvectorElixir.VectorEntry`), comprehensive typespecs, and functional APIs.

---

## Installation

Add `ruvector_elixir` to your list of dependencies in `mix.exs`:

```elixir
def deps do
  [
    {:ruvector_elixir, "~> 0.1.0"}
  ]
end
```

### System Requirements

A working **Rust toolchain** (Rust 1.75+ or later) is required to compile the native extension:

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

---

## Quickstart

### 1. Opening a Database

Create or open a database with a specified vector dimension (e.g. 128 dimensions):

```elixir
# Default Cosine metric with exact flat indexing
{:ok, db} = RuvectorElixir.open("priv/data/embeddings.rvf", 128)

# With HNSW index and Euclidean distance
{:ok, db} = RuvectorElixir.open("priv/data/embeddings.rvf", 128, %{
  metric: :euclidean,
  hnsw: %{
    m: 16,
    ef_construction: 100,
    ef_search: 50,
    max_elements: 100_000
  }
})
```

### 2. Inserting Vectors

Vectors can be inserted individually or in bulk using `RuvectorElixir.VectorEntry` structs, maps, or raw lists of floats:

```elixir
alias RuvectorElixir.VectorEntry

# Insert with explicit ID and metadata using VectorEntry
entry = VectorEntry.new([0.1, 0.2, 0.3, ...], id: "doc_1", metadata: %{
  "title" => "Elixir Guide",
  "category" => "programming",
  "views" => 1200
})
{:ok, "doc_1"} = RuvectorElixir.insert(db, entry)

# Insert with auto-generated UUID
{:ok, id} = RuvectorElixir.insert(db, [0.1, 0.2, 0.3, ...])

# Bulk insert
entries = [
  %{id: "doc_2", vector: [...], metadata: %{"category" => "news"}},
  %{id: "doc_3", vector: [...], metadata: %{"category" => "programming"}}
]
{:ok, ["doc_2", "doc_3"]} = RuvectorElixir.insert_batch(db, entries)
```

### 3. Searching for Nearest Neighbors

Perform top-`k` similarity search using a query vector:

```elixir
query = [0.1, 0.2, 0.3, ...]

# Fast ID-only search
ids = RuvectorElixir.search(db, query, 5)
# => ["doc_1", "doc_3", "doc_2"]

# Search with metadata filtering
filtered_ids = RuvectorElixir.search(db, query, %{"category" => "programming"}, 5)
# => ["doc_1", "doc_3"]

# Detailed search (returns ID, distance score, vector, and metadata)
{:ok, results} = RuvectorElixir.search_detailed(db, query, %{"category" => "programming"}, 5)
Enum.each(results, fn r ->
  IO.puts("ID: #{r.id} (Score: #{r.score}) - Title: #{r.metadata["title"]}")
end)
```

### 4. Fetching, Deleting & Inspecting

```elixir
# Retrieve vector and metadata by ID
{:ok, entry} = RuvectorElixir.get(db, "doc_1")
IO.inspect(entry.vector)
IO.inspect(entry.metadata)

# Check count and keys
{:ok, count} = RuvectorElixir.len(db)
{:ok, all_ids} = RuvectorElixir.keys(db)
{:ok, false} = RuvectorElixir.empty?(db)

# Delete an entry
{:ok, true} = RuvectorElixir.delete(db, "doc_1")

# Database info
info = RuvectorElixir.info(db)
# => %{"dimensions" => 128, "distance_metric" => "euclidean", "node_count" => 2}
```

### 5. Distance Calculations

Direct vector-to-vector distance calculations without opening a database:

```elixir
v1 = [1.0, 0.0, 0.0]
v2 = [0.0, 1.0, 0.0]

{:ok, dist} = RuvectorElixir.distance(v1, v2, :cosine)
# => {:ok, 1.0}

{:ok, dist} = RuvectorElixir.calculate_distance(v1, v2, :euclidean)
# => {:ok, 1.4142135}
```

---

## Configuration Options

### Database Options

| Option | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `:metric` | atom / string | `:cosine` | Distance metric (`:cosine`, `:euclidean`, `:dot_product`, `:manhattan`). |
| `:hnsw` | boolean / map | `nil` | `true` for default HNSW, or map with tuning parameters (see below). |

### HNSW Configuration

| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `:m` | pos_integer | `16` | Maximum number of outgoing edges per node in the graph. |
| `:ef_construction` | pos_integer | `100` | Size of the candidate list evaluated during index construction. |
| `:ef_search` | pos_integer | `50` | Default candidate list size during search. |
| `:max_elements` | pos_integer | `100_000` | Initial pre-allocated capacity for vectors. |

---

## Architecture

`RuvectorElixir` communicates with `ruvector-core` via a native C-ABI bridge built with Rustler:

```
+-------------------------------------------------------------+
|                      Elixir Application                     |
|                   RuvectorElixir (BEAM)                     |
+------------------------------+------------------------------+
                               |
                   Rustler NIF Interface
                               |
               +---------------+---------------+
               |                               |
       [DirtyIo Threads]               [DirtyCpu Threads]
     - open_db / persistence         - k-NN search
     - insert / insert_batch         - distance calculations
     - get / delete / all_ids        - HNSW graph traversal
               |                               |
               +---------------+---------------+
                               |
                    ruvector-core (Rust)
                    - VectorDB & Sled Storage
                    - HNSW & Flat Indexing
                    - SIMD Distance Metrics
```

- **Thread Safety**: The Rust database engine features concurrent storage and internal index locking, managed directly inside a Rustler `ResourceArc`. Multiple Elixir processes (e.g. concurrent Tasks or GenServers) can concurrently read, insert, and search the database with zero locking overhead.
- **Zero BEAM Blocking**: All CPU-intensive searches run on Erlang `DirtyCpu` schedulers, while all disk I/O operations run on `DirtyIo` schedulers, keeping the normal BEAM schedulers free for Web / LiveView requests.
- **Idiomatic APIs & Bang Variants**: Supports both tuple results `{:ok, res} | {:error, reason}` and bang variants (`open!`, `insert!`, `search!`, `get!`, `delete!`, `len!`) that raise on failure. Options can be supplied as maps or keyword lists.

---

## Performance & Benchmarks

Because `ruvector` leverages SIMD instructions (AVX-512, AVX2, NEON) in Rust, vector operations achieve sub-millisecond query latencies:

- **Flat Search**: Exact search over 10,000 128-dimensional vectors in ~1.5ms.
- **HNSW Search**: Approximate search over 100,000+ vectors in < 0.2ms with >99% recall.
- **Batch Insertion**: Up to 100,000 vectors/sec with concurrent pipeline insertion.

---

## Repository & Development

This repository tracks the official upstream [ruvector](https://github.com/ruvnet/ruvector) repository as a Git submodule located at `reference/ruvector`.

### Cloning with Submodule

```bash
git clone --recurse-submodules https://github.com/ruvnet/ruvector.git
# Or if already cloned:
git submodule update --init --recursive
```

### Running Tests

Execute the complete test suite:

```bash
mix test
```

Generate ExDoc documentation:

```bash
mix docs
```

---

## Upstream Project

- **Core Repository**: [https://github.com/ruvnet/ruvector](https://github.com/ruvnet/ruvector)
- **Crates**: `ruvector-core`, `ruvector-graph`

---

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.