Packages

portfolio_index

0.4.0
0.4.0 0.3.1 0.3.0 0.2.0 0.1.1 0.1.0

Production adapters and pipelines for PortfolioCore. Vector stores, graph stores, embedders, Broadway pipelines, and advanced RAG strategies.

HexDocs

Current section

6 Versions

Jump to
Readme
6 Versions
21 Dependencies
3 Dependants
Files
Activity
Hide whitespace

Compare versions

21 files changed
+3024 additions
-712 deletions
  @@ -7,6 +7,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7 7
8 8 ## [Unreleased]
9 9
10 + ## [0.4.0] - 2026-01-08
11 +
12 + ### Added
13 +
14 + #### Local LLM Support
15 + - `PortfolioIndex.Adapters.LLM.Ollama` - Ollama LLM adapter using ollixir client
16 + - Local model orchestration with configurable base URL
17 + - OpenAI-style message formatting
18 + - Streaming support with delta normalization
19 + - Rate limiting and telemetry instrumentation
20 + - Configurable model info and supported models list
21 + - `PortfolioIndex.Adapters.LLM.VLLM` - vLLM adapter for OpenAI-compatible endpoints
22 + - Uses openai_ex for OpenAI-compatible API
23 + - Configurable base URL for local clusters
24 + - SSE stream parsing with finish reason normalization
25 + - Rate limiting and failure detection
26 +
27 + #### Local Embeddings Support
28 + - `PortfolioIndex.Adapters.Embedder.Ollama` - Ollama embeddings adapter using ollixir
29 + - Support for nomic-embed-text, mxbai-embed-large models
30 + - Batch embedding support via native API
31 + - Token count extraction from response or estimation fallback
32 + - Rate limiting and telemetry instrumentation
33 +
34 + #### Telemetry Enhancements
35 + - `PortfolioIndex.Telemetry.Context` - Lineage context helpers for telemetry metadata
36 + - Standard context keys: `trace_id`, `work_id`, `plan_id`, `step_id`
37 + - `merge/2` for combining context with telemetry metadata
38 + - Integrated into all LLM adapters (Anthropic, Codex, Gemini, OpenAI, Ollama, vLLM)
39 +
40 + #### Backend Integration
41 + - `PortfolioIndex.LLM.BackendBridge` - Bridge for CrucibleIR backend prompts
42 + - `prompt_to_messages/1` - Convert backend prompt structs to messages and opts
43 + - `completion_from_result/3` - Build completion maps from adapter results
44 + - Usage normalization across provider formats
45 +
46 + #### Examples
47 + - `examples/ollama_embedder.exs` - Ollama embedder demonstration
48 + - `examples/ollama_llm.exs` - Ollama LLM with streaming demonstration
49 + - `examples/ollama_setup.exs` - Automated Ollama model installation script
50 + - `examples/vllm_llm.exs` - vLLM adapter demonstration
51 + - `examples/support/ollama_helpers.exs` - Shared helpers for Ollama examples
52 +
53 + ### Changed
54 +
55 + - LLM adapters now pass opts to telemetry for lineage context propagation
56 + - Gemini embedder and LLM adapters support SDK injection via `:sdk` option
57 + - RateLimiter uses rescue instead of try for ETS table creation
58 + - Updated run_all.sh to include Ollama examples (vLLM skipped by default)
59 +
60 + ### Dependencies
61 +
62 + - Changed portfolio_core from path to hex: `~> 0.4.0`
63 + - Changed foundation from path to hex: `~> 0.2.0`
64 + - Changed gemini_ex from path to hex: `~> 0.8.8`
65 + - Changed claude_agent_sdk from path to hex: `~> 0.7.6`
66 + - Changed codex_sdk from path to hex: `~> 0.5.0`
67 + - Added ollixir `~> 0.1.0` for Ollama integration
68 +
10 69 ## [0.3.1] - 2025-12-30
11 70
12 71 ### Added
  @@ -367,7 +426,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
367 426
368 427 - Initial release of PortfolioIndex
369 428
370 - [Unreleased]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.3.1...HEAD
429 + [Unreleased]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.4.0...HEAD
430 + [0.4.0]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.3.1...v0.4.0
371 431 [0.3.1]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.3.0...v0.3.1
372 432 [0.3.0]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.2.0...v0.3.0
373 433 [0.2.0]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.1.1...v0.2.0
  @@ -22,7 +22,8 @@ Portfolio Index implements the port specifications defined in [Portfolio Core](h
22 22 - **Vector Store Adapters** - pgvector (PostgreSQL + fulltext hybrid)
23 23 - **Graph Store Adapters** - Neo4j via boltx + community operations
24 24 - **Embedding Providers** - Google Gemini
25 - - **LLM Providers** - Google Gemini, Anthropic Claude, OpenAI GPT/o1
25 + - **LLM Providers** - Google Gemini, Anthropic Claude, OpenAI (openai_ex), Codex (codex_sdk),
26 + Ollama, vLLM (OpenAI-compatible)
26 27 - **Broadway Pipelines** - Ingestion and embedding with backpressure
27 28 - **RAG Strategies** - Hybrid (RRF fusion), Self-RAG (self-critique), GraphRAG, Agentic
28 29
  @@ -75,7 +76,7 @@ Add `portfolio_index` to your list of dependencies in `mix.exs`:
75 76 ```elixir
76 77 def deps do
77 78 [
78 - {:portfolio_index, "~> 0.3.1"}
79 + {:portfolio_index, "~> 0.4.0"}
79 80 ]
80 81 end
81 82 ```
  @@ -199,6 +200,37 @@ alias PortfolioIndex.RAG.Strategies.SelfRAG
199 200 | `NEO4J_USER` | Neo4j username | `neo4j` |
200 201 | `NEO4J_PASSWORD` | Neo4j password | - |
201 202 | `GEMINI_API_KEY` | Google Gemini API key | - |
203 + | `OPENAI_API_KEY` | OpenAI API key (OpenAI + Codex) | - |
204 + | `OPENAI_ORGANIZATION` | OpenAI organization ID (optional) | - |
205 + | `ANTHROPIC_API_KEY` | Anthropic API key | - |
206 + | `CODEX_API_KEY` | Codex SDK API key (optional) | - |
207 + | `OLLAMA_HOST` | Ollama host URL | `http://localhost:11434` |
208 + | `OLLAMA_BASE_URL` | Ollama base URL (override) | `http://localhost:11434/api` |
209 + | `OLLAMA_API_KEY` | Ollama API key (optional) | - |
210 + | `VLLM_BASE_URL` | vLLM base URL | `http://localhost:8000/v1` |
211 + | `VLLM_API_KEY` | vLLM API key (optional) | - |
212 +
213 + ### Local Model Setup
214 +
215 + Ollama examples require a running Ollama server and these models:
216 +
217 + - `llama3.2` (LLM)
218 + - `nomic-embed-text` (embeddings)
219 +
220 + Install them with:
221 +
222 + ```bash
223 + ollama pull llama3.2
224 + ollama pull nomic-embed-text
225 + ```
226 +
227 + Or run:
228 +
229 + ```bash
230 + mix run examples/ollama_setup.exs
231 + ```
232 +
233 + The vLLM example expects a vLLM server running at `VLLM_BASE_URL`.
202 234
203 235 ### Config Files
204 236
  @@ -233,23 +265,32 @@ config :boltx, Boltx,
233 265 ### Embedders
234 266
235 267 - **Gemini** - Google Gemini text-embedding-004
268 + - **OpenAI** - text-embedding-3-small/large
269 + - **Ollama** - Local embeddings via ollixir
236 270
237 271 | Adapter | Provider | Model |
238 272 |---------|----------|-------|
239 273 | `Gemini` | Google | text-embedding-004 (768 dims) |
240 - | `OpenAI` | OpenAI | text-embedding-3-small/large (placeholder) |
274 + | `OpenAI` | OpenAI | text-embedding-3-small/large |
275 + | `Ollama` | Ollama | nomic-embed-text (default) |
241 276
242 277 ### LLMs
243 278
244 - - **Gemini** - gemini-2.5-flash with streaming
279 + - **Gemini** - gemini-flash-lite-latest with streaming
245 280 - **Anthropic** - Claude via claude_agent_sdk
246 - - **OpenAI** - GPT/o1 via codex_sdk
281 + - **OpenAI** - GPT-4o-mini (low-cost default) via openai_ex
282 + - **Codex** - OpenAI Codex SDK with agentic support
283 + - **Ollama** - Local models via ollixir
284 + - **vLLM** - OpenAI-compatible endpoints via openai_ex
247 285
248 286 | Adapter | Provider | Model |
249 287 |---------|----------|-------|
250 - | `Gemini` | Google | gemini-2.5-flash |
288 + | `Gemini` | Google | gemini-flash-lite-latest |
251 289 | `Anthropic` | Anthropic | Claude (SDK default) |
252 - | `OpenAI` | OpenAI | GPT/o1 (SDK default) |
290 + | `OpenAI` | OpenAI | gpt-4o-mini (default) |
291 + | `Codex` | OpenAI | Codex SDK default |
292 + | `Ollama` | Ollama | llama3.2 (default) |
293 + | `VLLM` | vLLM | llama3 (default) |
253 294
254 295 ### Chunker
255 296
  @@ -308,9 +349,9 @@ Recursive.chunk(text, :plain, %{
308 349
309 350 ```
310 351 ┌─────────────────────────────────────────────────────────────┐
311 - │ Portfolio Index │
352 + │ Portfolio Index │
312 353 ├─────────────────────────────────────────────────────────────┤
313 - │ Adapters │
354 + │ Adapters │
314 355 │ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
315 356 │ │ Vector Store │ │ Graph Store │ │ Embedder │ │
316 357 │ │ • Pgvector │ │ • Neo4j │ │ • Gemini │ │
  @@ -320,15 +361,19 @@ Recursive.chunk(text, :plain, %{
320 361 │ │ LLM │ │ Chunker │ │ Document Store│ │
321 362 │ │ • Gemini │ │ • Recursive │ │ • Postgres │ │
322 363 │ │ • Anthropic │ │ │ │ │ │
364 + │ │ • OpenAI │ │ │ │ │ │
365 + │ │ • Codex │ │ │ │ │ │
366 + │ │ • Ollama │ │ │ │ │ │
367 + │ │ • vLLM │ │ │ │ │ │
323 368 │ └───────────────┘ └───────────────┘ └───────────────┘ │
324 369 ├─────────────────────────────────────────────────────────────┤
325 - │ Pipelines (Broadway) │
370 + │ Pipelines (Broadway) │
326 371 │ ┌───────────────────────────┐ ┌───────────────────────────┐│
327 372 │ │ Ingestion │ │ Embedding ││
328 373 │ │ FileProducer → Chunker │ │ ETSProducer → VectorStore ││
329 374 │ └───────────────────────────┘ └───────────────────────────┘│
330 375 ├─────────────────────────────────────────────────────────────┤
331 - │ RAG Strategies │
376 + │ RAG Strategies │
332 377 │ ┌────────────────────┐ ┌────────────────────┐ │
333 378 │ │ Hybrid │ │ Self-RAG │ │
334 379 │ │ Vector + RRF fusion│ │ Critique + Refine │ │
  @@ -341,8 +386,8 @@ Recursive.chunk(text, :plain, %{
341 386 │
342 387 ▼
343 388 ┌─────────────────────────────────────────────────────────────┐
344 - │ Portfolio Core │
345 - │ (Port Specifications & Registry) │
389 + │ Portfolio Core │
390 + │ (Port Specifications & Registry) │
346 391 └─────────────────────────────────────────────────────────────┘
347 392 ```
348 393
  @@ -808,6 +853,20 @@ Attach handlers for monitoring:
808 853 - [`portfolio_core`](https://github.com/nshkrdotcom/portfolio_core) - Hexagonal architecture primitives
809 854 - [`portfolio_manager`](https://github.com/nshkrdotcom/portfolio_manager) - CLI and application layer
810 855
856 + ## Acknowledgments
857 +
858 + Significant portions of this library's architecture and features were derived from
859 + analysis of [Arcana](https://github.com/georgeguimaraes/arcana) by George Guimarães,
860 + licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
861 +
862 + Features inspired by Arcana include:
863 + - RAG pipeline architecture (query rewriting, expansion, decomposition)
864 + - Evaluation system design (IR metrics, test case generation)
865 + - Chunker token utilities and sizing options
866 + - Telemetry patterns and agent system design
867 +
868 + See `docs/20251230/arcana_gap_analysis/` for detailed analysis.
869 +
811 870 ## License
812 871
813 872 MIT License - see [LICENSE](LICENSE) for details.
  @@ -1,7 +1,7 @@
1 1 {<<"links">>,
2 2 [{<<"GitHub">>,<<"https://github.com/nshkrdotcom/portfolio_index">>}]}.
3 3 {<<"name">>,<<"portfolio_index">>}.
4 - {<<"version">>,<<"0.3.1">>}.
4 + {<<"version">>,<<"0.4.0">>}.
5 5 {<<"description">>,
6 6 <<"Production adapters and pipelines for PortfolioCore. Vector stores, graph stores, embedders, Broadway pipelines, and advanced RAG strategies.">>}.
7 7 {<<"elixir">>,<<"~> 1.15">>}.
  @@ -58,12 +58,17 @@
58 58 <<"lib/portfolio_index/adapters/embedder">>,
59 59 <<"lib/portfolio_index/adapters/embedder/function.ex">>,
60 60 <<"lib/portfolio_index/adapters/embedder/bumblebee.ex">>,
61 + <<"lib/portfolio_index/adapters/embedder/ollama.ex">>,
61 62 <<"lib/portfolio_index/adapters/embedder/gemini.ex">>,
62 63 <<"lib/portfolio_index/adapters/embedder/open_ai.ex">>,
64 + <<"lib/portfolio_index/adapters/rate_limiter.ex">>,
63 65 <<"lib/portfolio_index/adapters/llm">>,
66 + <<"lib/portfolio_index/adapters/llm/vllm.ex">>,
67 + <<"lib/portfolio_index/adapters/llm/openai.ex">>,
64 68 <<"lib/portfolio_index/adapters/llm/anthropic.ex">>,
69 + <<"lib/portfolio_index/adapters/llm/codex.ex">>,
70 + <<"lib/portfolio_index/adapters/llm/ollama.ex">>,
65 71 <<"lib/portfolio_index/adapters/llm/gemini.ex">>,
66 - <<"lib/portfolio_index/adapters/llm/open_ai.ex">>,
67 72 <<"lib/portfolio_index/adapters/vector_store">>,
68 73 <<"lib/portfolio_index/adapters/vector_store/pgvector">>,
69 74 <<"lib/portfolio_index/adapters/vector_store/pgvector/hybrid.ex">>,
  @@ -77,6 +82,7 @@
77 82 <<"lib/portfolio_index/telemetry/vector_store.ex">>,
78 83 <<"lib/portfolio_index/telemetry/logger.ex">>,
79 84 <<"lib/portfolio_index/telemetry/llm.ex">>,
85 + <<"lib/portfolio_index/telemetry/context.ex">>,
80 86 <<"lib/portfolio_index/embedder">>,
81 87 <<"lib/portfolio_index/embedder/dimension_detector.ex">>,
82 88 <<"lib/portfolio_index/embedder/registry.ex">>,
  @@ -107,6 +113,8 @@
107 113 <<"lib/portfolio_index/rag/strategies/hybrid.ex">>,
108 114 <<"lib/portfolio_index/rag/strategies/self_rag.ex">>,
109 115 <<"lib/portfolio_index/rag/strategies/graph_rag.ex">>,
116 + <<"lib/portfolio_index/llm">>,
117 + <<"lib/portfolio_index/llm/backend_bridge.ex">>,
110 118 <<"lib/portfolio_index/vector_store">>,
111 119 <<"lib/portfolio_index/vector_store/backend.ex">>,
112 120 <<"lib/portfolio_index/vector_store/collections.ex">>,
  @@ -120,7 +128,12 @@
120 128 [[{<<"name">>,<<"portfolio_core">>},
121 129 {<<"app">>,<<"portfolio_core">>},
122 130 {<<"optional">>,false},
123 - {<<"requirement">>,<<"~> 0.3.1">>},
131 + {<<"requirement">>,<<"~> 0.4.0">>},
132 + {<<"repository">>,<<"hexpm">>}],
133 + [{<<"name">>,<<"foundation">>},
134 + {<<"app">>,<<"foundation">>},
135 + {<<"optional">>,false},
136 + {<<"requirement">>,<<"~> 0.2.0">>},
124 137 {<<"repository">>,<<"hexpm">>}],
125 138 [{<<"name">>,<<"ecto_sql">>},
126 139 {<<"app">>,<<"ecto_sql">>},
  @@ -155,17 +168,27 @@
155 168 [{<<"name">>,<<"claude_agent_sdk">>},
156 169 {<<"app">>,<<"claude_agent_sdk">>},
157 170 {<<"optional">>,false},
158 - {<<"requirement">>,<<"~> 0.7.2">>},
171 + {<<"requirement">>,<<"~> 0.7.6">>},
159 172 {<<"repository">>,<<"hexpm">>}],
160 173 [{<<"name">>,<<"codex_sdk">>},
161 174 {<<"app">>,<<"codex_sdk">>},
162 175 {<<"optional">>,false},
163 - {<<"requirement">>,<<"~> 0.4.5">>},
176 + {<<"requirement">>,<<"~> 0.5.0">>},
177 + {<<"repository">>,<<"hexpm">>}],
178 + [{<<"name">>,<<"ollixir">>},
179 + {<<"app">>,<<"ollixir">>},
180 + {<<"optional">>,false},
181 + {<<"requirement">>,<<"~> 0.1.0">>},
182 + {<<"repository">>,<<"hexpm">>}],
183 + [{<<"name">>,<<"openai_ex">>},
184 + {<<"app">>,<<"openai_ex">>},
185 + {<<"optional">>,false},
186 + {<<"requirement">>,<<"~> 0.9.18">>},
164 187 {<<"repository">>,<<"hexpm">>}],
165 188 [{<<"name">>,<<"req">>},
166 189 {<<"app">>,<<"req">>},
167 190 {<<"optional">>,false},
168 - {<<"requirement">>,<<"~> 0.4">>},
191 + {<<"requirement">>,<<"~> 0.5">>},
169 192 {<<"repository">>,<<"hexpm">>}],
170 193 [{<<"name">>,<<"finch">>},
171 194 {<<"app">>,<<"finch">>},
  @@ -182,11 +205,6 @@
182 205 {<<"optional">>,false},
183 206 {<<"requirement">>,<<"~> 1.2">>},
184 207 {<<"repository">>,<<"hexpm">>}],
185 - [{<<"name">>,<<"hammer">>},
186 - {<<"app">>,<<"hammer">>},
187 - {<<"optional">>,false},
188 - {<<"requirement">>,<<"~> 6.1">>},
189 - {<<"repository">>,<<"hexpm">>}],
190 208 [{<<"name">>,<<"telemetry">>},
191 209 {<<"app">>,<<"telemetry">>},
192 210 {<<"optional">>,false},
  @@ -195,7 +213,7 @@
195 213 [{<<"name">>,<<"telemetry_metrics">>},
196 214 {<<"app">>,<<"telemetry_metrics">>},
197 215 {<<"optional">>,false},
198 - {<<"requirement">>,<<"~> 0.6">>},
216 + {<<"requirement">>,<<"~> 1.1">>},
199 217 {<<"repository">>,<<"hexpm">>}],
200 218 [{<<"name">>,<<"telemetry_poller">>},
201 219 {<<"app">>,<<"telemetry_poller">>},
  @@ -13,9 +13,13 @@ defmodule PortfolioIndex do
13 13 (hybrid fulltext via `PortfolioIndex.Adapters.VectorStore.Pgvector.Hybrid`)
14 14 - Graph Stores: `PortfolioIndex.Adapters.GraphStore.Neo4j`
15 15 (community ops via `PortfolioIndex.Adapters.GraphStore.Neo4j.Community`)
16 - - Embedders: `PortfolioIndex.Adapters.Embedder.Gemini`
16 + - Embedders: `PortfolioIndex.Adapters.Embedder.Gemini`,
17 + `PortfolioIndex.Adapters.Embedder.OpenAI`,
18 + `PortfolioIndex.Adapters.Embedder.Ollama`,
19 + `PortfolioIndex.Adapters.Embedder.Bumblebee`
17 20 - LLMs: `PortfolioIndex.Adapters.LLM.Gemini`, `PortfolioIndex.Adapters.LLM.Anthropic`,
18 - `PortfolioIndex.Adapters.LLM.OpenAI`
21 + `PortfolioIndex.Adapters.LLM.OpenAI`, `PortfolioIndex.Adapters.LLM.Codex`,
22 + `PortfolioIndex.Adapters.LLM.Ollama`, `PortfolioIndex.Adapters.LLM.VLLM`
19 23 - Chunkers: `PortfolioIndex.Adapters.Chunker.Recursive`
20 24 - Document Stores: `PortfolioIndex.Adapters.DocumentStore.Postgres`
  @@ -37,10 +37,15 @@ defmodule PortfolioIndex.Adapters.Embedder.Gemini do
37 37 require Logger
38 38 alias Gemini.Types.Response.ContentEmbedding
39 39 alias Gemini.Types.Response.EmbedContentResponse
40 + alias PortfolioIndex.Adapters.RateLimiter
40 41
41 42 @impl true
42 43 def embed(text, opts) do
44 + # Wait for rate limiter before making request
45 + RateLimiter.wait(:gemini, :embedding)
46 +
43 47 start_time = System.monotonic_time(:millisecond)
48 + sdk = resolve_sdk(opts)
44 49 {model_opt, effective_model} = resolve_embedding_model(opts)
45 50 dims = resolve_dimensions(opts, effective_model)
46 51
  @@ -48,11 +53,13 @@ defmodule PortfolioIndex.Adapters.Embedder.Gemini do
48 53 opts
49 54 |> Keyword.delete(:model)
50 55 |> Keyword.delete(:dimensions)
56 + |> Keyword.delete(:sdk)
51 57 |> Keyword.put(:output_dimensionality, dims)
52 58 |> maybe_put(:model, model_opt)
53 59
54 - case Gemini.embed_content(text, gemini_opts) do
60 + case sdk.embed_content(text, gemini_opts) do
55 61 {:ok, response} ->
62 + RateLimiter.record_success(:gemini, :embedding)
56 63 vector = extract_embedding(response)
57 64 normalized_vector = maybe_normalize(vector, effective_model, dims)
58 65
  @@ -78,6 +85,8 @@ defmodule PortfolioIndex.Adapters.Embedder.Gemini do
78 85 }}
79 86
80 87 {:error, reason} ->
88 + failure_type = detect_failure_type(reason)
89 + RateLimiter.record_failure(:gemini, :embedding, failure_type)
81 90 Logger.error("Embedding failed: #{inspect(reason)}")
82 91 {:error, reason}
83 92 end
  @@ -191,6 +200,14 @@ defmodule PortfolioIndex.Adapters.Embedder.Gemini do
191 200 defp maybe_put(opts, _key, nil), do: opts
192 201 defp maybe_put(opts, key, value), do: Keyword.put(opts, key, value)
193 202
203 + defp gemini_module do
204 + Application.get_env(:portfolio_index, :gemini_sdk, Gemini)
205 + end
206 +
207 + defp resolve_sdk(opts) do
208 + Keyword.get(opts, :sdk) || gemini_module()
209 + end
210 +
194 211 defp maybe_normalize(vector, model, dims) do
195 212 if Gemini.Config.needs_normalization?(model, dims) do
196 213 normalize(vector)
  @@ -224,4 +241,19 @@ defmodule PortfolioIndex.Adapters.Embedder.Gemini do
224 241 metadata
225 242 )
226 243 end
244 +
245 + defp detect_failure_type(reason) do
246 + reason_str = inspect(reason) |> String.downcase()
247 +
248 + cond do
249 + String.contains?(reason_str, "rate") or String.contains?(reason_str, "429") ->
250 + :rate_limited
251 +
252 + String.contains?(reason_str, "timeout") ->
253 + :timeout
254 +
255 + true ->
256 + :server_error
257 + end
258 + end
227 259 end
Loading more files…