Packages

portfolio_index

0.3.1
0.4.0 0.3.1 0.3.0 0.2.0 0.1.1 0.1.0

Production adapters and pipelines for PortfolioCore. Vector stores, graph stores, embedders, Broadway pipelines, and advanced RAG strategies.

HexDocs

Current section

6 Versions

Jump to
Readme
6 Versions
19 Dependencies
3 Dependants
Files
Activity
Hide whitespace

Compare versions

53 files changed
+10649 additions
-291 deletions
  @@ -7,6 +7,292 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7 7
8 8 ## [Unreleased]
9 9
10 + ## [0.3.1] - 2025-12-30
11 +
12 + ### Added
13 +
14 + #### Collection Selection & Self-Correction
15 + - `PortfolioIndex.Adapters.CollectionSelector.LLM` - LLM-based collection routing
16 + - Routes queries to relevant collections based on descriptions
17 + - Returns JSON with selected collections and reasoning
18 + - Custom prompt support via `:prompt` option
19 + - `PortfolioIndex.Adapters.CollectionSelector.RuleBased` - Rule-based collection routing
20 + - Keyword matching with configurable boost factors
21 + - Deterministic routing without LLM calls
22 + - Useful for testing and predictable behavior
23 + - `PortfolioIndex.RAG.SelfCorrectingSearch` - Search with sufficiency evaluation and query rewriting
24 + - Evaluates if results are sufficient for answering the question
25 + - Rewrites query and retries if insufficient
26 + - Configurable max iterations and custom prompts
27 + - Tracks correction history in context
28 + - `PortfolioIndex.RAG.SelfCorrectingAnswer` - Answer generation with grounding evaluation
29 + - Evaluates if answer is grounded in provided context
30 + - Identifies ungrounded claims and generates corrections
31 + - Configurable max corrections and grounding threshold
32 + - Tracks correction history for debugging
33 + - `PortfolioIndex.RAG.Reranker` - Pipeline-integrated reranking utilities
34 + - `rerank/2` - Rerank context results with threshold filtering
35 + - `rerank_chunks/3` - Direct chunk reranking
36 + - `deduplicate/2` - Remove duplicate chunks by id or content
37 + - Score tracking in context
38 +
39 + #### Enhanced Agentic Strategy
40 + - `PortfolioIndex.RAG.Strategies.Agentic.execute_pipeline/2` - Full pipeline execution with all enhancements
41 + - Query rewriting, expansion, decomposition
42 + - Collection selection and routing
43 + - Self-correcting search with sufficiency evaluation
44 + - Reranking with threshold filtering
45 + - Self-correcting answer with grounding evaluation
46 + - `PortfolioIndex.RAG.Strategies.Agentic.with_context/2` - Pipeline execution with Context struct
47 + - Enables functional composition with pipe operator
48 + - Configurable step skipping via `:skip` option
49 + - Returns full Context with all intermediate results
50 +
51 + #### Telemetry Standardization
52 + - `PortfolioIndex.Telemetry.Logger` - Human-readable telemetry logger with one-line attach
53 + - Text and JSON output formats
54 + - Component-level filtering (embedder, llm, rag, vector_store, evaluation)
55 + - Duration formatting (ms, s)
56 + - Context-aware metadata display
57 + - `PortfolioIndex.Telemetry.LLM` - LLM-specific telemetry with token tracking
58 + - `span/2` for wrapping LLM calls with detailed metadata
59 + - `estimate_tokens/1` for token count estimation
60 + - `extract_usage/1` for parsing provider-specific usage data
61 + - `PortfolioIndex.Telemetry.Embedder` - Embedder telemetry utilities
62 + - `span/2` for single embedding operations
63 + - `batch_span/2` for batch embedding operations
64 + - `PortfolioIndex.Telemetry.RAG` - RAG pipeline step telemetry
65 + - `step_span/3` for wrapping pipeline steps (rewrite, expand, decompose, etc.)
66 + - `search_span/3` for search-specific telemetry
67 + - `rerank_span/3` for rerank-specific telemetry
68 + - `correction_event/2` for self-correction tracking
69 + - `PortfolioIndex.Telemetry.VectorStore` - Vector store operation telemetry
70 + - `search_span/2` for search operations
71 + - `insert_span/2` for single insert operations
72 + - `batch_insert_span/2` for batch insert operations
73 +
74 + #### Vector Store Enhancements
75 + - `PortfolioIndex.Adapters.VectorStore.Memory` - In-memory HNSWLib vector store
76 + - GenServer-based in-memory storage for testing and development
77 + - Uses HNSWLib for approximate nearest neighbor (ANN) search
78 + - Soft deletion support (marks as deleted without index rebuild)
79 + - Optional file-based persistence via save/load
80 + - Thread-safe for concurrent reads and writes
81 + - Configurable dimensions, max_elements, ef_construction, and m parameters
82 + - `PortfolioIndex.VectorStore.Backend` - Backend resolution with per-call override
83 + - Runtime backend switching via `:backend` option
84 + - Backend aliases: `:pgvector`, `:memory`, `:qdrant`
85 + - Module and tuple configuration support: `{Memory, store: pid}`
86 + - Unified API for search, insert, insert_batch, delete, get
87 + - `PortfolioIndex.VectorStore.IndexManager` - Index auto-creation and management
88 + - `ensure_index/2` - Create index if not exists
89 + - `index_exists?/2` - Check index existence
90 + - `index_stats/2` - Get index statistics
91 + - `rebuild_index/2` - Rebuild index after bulk inserts
92 + - `drop_index/2` - Remove index
93 + - Backend-specific options for pgvector (HNSW, IVFFlat) and memory stores
94 + - `PortfolioIndex.VectorStore.Collections` - Collection-based organization
95 + - Logical grouping of vectors via metadata filtering
96 + - `search_collection/3` - Search within specific collection
97 + - `insert_to_collection/5` - Insert with collection tag
98 + - `list_collections/1` - List all collections
99 + - `collection_stats/2` - Get collection statistics
100 + - `clear_collection/2` - Delete all vectors in collection
101 + - `PortfolioIndex.VectorStore.SoftDelete` - Soft deletion support
102 + - `soft_delete/2` - Mark item as deleted without removal
103 + - `soft_delete_where/2` - Soft delete matching items
104 + - `restore/2` - Restore soft-deleted item
105 + - `purge_deleted/2` - Permanently delete old soft-deleted items
106 + - `count_deleted/1` - Count soft-deleted items
107 + - `PortfolioIndex.VectorStore.Search` - Enhanced search with hybrid support
108 + - `similarity_search/2` - Vector search with threshold and collection filtering
109 + - `hybrid_search/3` - Combine vector and keyword search with RRF
110 + - `filter_results/2` - Post-filter by metadata
111 + - `normalize_scores/2` - Normalize across distance metrics
112 + - `deduplicate/2` - Remove duplicate results by id or content_hash
113 +
114 + #### Embedder Enhancements
115 + - `PortfolioIndex.Adapters.Embedder.OpenAI` - OpenAI text-embedding API adapter
116 + - Full implementation replacing placeholder with actual API calls
117 + - Support for text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002
118 + - Batch embedding support via native API
119 + - Telemetry instrumentation for monitoring
120 + - `PortfolioIndex.Adapters.Embedder.Bumblebee` - Local Bumblebee/Nx.Serving embeddings
121 + - HuggingFace model loading with EXLA compilation
122 + - Support for BGE, MiniLM, and other sentence-transformers
123 + - Supervision tree integration via `child_spec/1`
124 + - No API calls required - fully local inference
125 + - `PortfolioIndex.Adapters.Embedder.Function` - Custom function wrapper adapter
126 + - Wrap any function as an embedder
127 + - Optional batch function support
128 + - Useful for testing and custom integrations
129 + - `PortfolioIndex.Embedder.Registry` - Model dimension registry
130 + - Pre-configured dimensions for OpenAI, Voyage, Bumblebee, Ollama models
131 + - Runtime registration for custom models
132 + - Provider lookup by model name
133 + - `PortfolioIndex.Embedder.Config` - Unified embedder configuration
134 + - Shorthand syntax (:openai, :bumblebee, etc.)
135 + - Module and tuple configuration support
136 + - Function wrapper auto-resolution
137 + - `current/0` and `current_dimensions/0` helpers
138 + - `PortfolioIndex.Embedder.DimensionDetector` - Automatic dimension detection
139 + - Multiple detection strategies (explicit, registry, probe)
140 + - Dimension validation for embeddings
141 + - Fallback probing for unknown models
142 +
143 + #### Retrieval Evaluation System
144 + - `PortfolioIndex.Adapters.RetrievalMetrics.Standard` - Standard IR metrics adapter
145 + - Implements `PortfolioCore.Ports.RetrievalMetrics` behaviour
146 + - `recall_at_k/3` - Recall at K (fraction of relevant items retrieved)
147 + - `precision_at_k/3` - Precision at K (fraction of retrieved items relevant)
148 + - `mrr/2` - Mean Reciprocal Rank (inverse of first relevant rank)
149 + - `hit_rate_at_k/3` - Hit Rate at K (1 if any relevant item in top K)
150 + - Aggregation with mean calculation across test cases
151 + - `PortfolioIndex.Schemas.TestCase` - Ecto schema for evaluation test cases
152 + - Links questions to ground truth chunks via many-to-many
153 + - Source: `:synthetic` (LLM-generated) or `:manual`
154 + - Collection and metadata support
155 + - `PortfolioIndex.Schemas.EvaluationRun` - Ecto schema for evaluation runs
156 + - Status tracking: `:running`, `:completed`, `:failed`
157 + - Aggregate metrics and per-case results storage
158 + - Timing and configuration tracking
159 + - `PortfolioIndex.Evaluation.Generator` - LLM-powered test case generation
160 + - `generate/2` - Sample chunks and generate synthetic questions
161 + - `generate_question/2` - Generate single question from chunk content
162 + - Chunk sampling with collection/source filtering
163 + - `PortfolioIndex.Evaluation` - Main evaluation orchestrator
164 + - `run/2` - Execute evaluation against test cases
165 + - `list_test_cases/2` - List with filtering options
166 + - `create_test_case/2` - Create manual test cases
167 + - `add_ground_truth/3` - Link chunks as ground truth
168 + - `list_runs/2` - Get historical evaluation runs
169 + - `compare_runs/2` - Compare metrics between runs
170 + - Database migration for evaluation tables
171 + - `portfolio_evaluation_test_cases` with source and collection indexes
172 + - `portfolio_evaluation_test_case_chunks` join table
173 + - `portfolio_evaluation_runs` with status index
174 +
175 + #### Production Maintenance Utilities
176 + - `PortfolioIndex.Maintenance` - Production maintenance utilities (reembed, diagnostics, retry)
177 + - `reembed/2` - Re-embed all chunks or filtered subset with progress tracking
178 + - `diagnostics/1` - Get system diagnostics including counts and storage usage
179 + - `retry_failed/2` - Reset failed documents to pending for reprocessing
180 + - `cleanup_deleted/2` - Permanently remove soft-deleted documents and chunks
181 + - `verify_embeddings/1` - Verify embedding consistency across chunks
182 + - `PortfolioIndex.Maintenance.Progress` - Progress reporting for maintenance operations
183 + - `cli_reporter/1` - Prints progress to stdout with percentage
184 + - `silent_reporter/0` - No-op reporter for silent operations
185 + - `telemetry_reporter/1` - Emits telemetry events for monitoring
186 + - `build_event/4` - Create progress events from components
187 +
188 + #### Mix Tasks
189 + - `mix portfolio.install` - Installation task for new projects
190 + - Generates database migrations for collections, documents, and chunks tables
191 + - Creates pgvector extension setup with HNSW index
192 + - Prints configuration instructions and next steps
193 + - Options: `--repo`, `--dimension`, `--no-migrations`
194 + - `mix portfolio.gen.embedding_migration` - Generate migration for dimension changes
195 + - Creates migration to alter vector column dimensions
196 + - Drops and recreates HNSW index for new dimensions
197 + - Options: `--dimension` (required), `--table`, `--column`
198 +
199 + #### Document Management Schemas
200 + - `PortfolioIndex.Schemas.Collection` - Ecto schema for document collections
201 + - Groups related documents for organized retrieval and routing
202 + - Unique name constraint with metadata support
203 + - Virtual `document_count` field for aggregation
204 + - `PortfolioIndex.Schemas.Document` - Ecto schema for ingested documents with status tracking
205 + - Status enum: `:pending`, `:processing`, `:completed`, `:failed`, `:deleted`
206 + - Content hash for deduplication via `compute_hash/1`
207 + - Collection relationship for document organization
208 + - Error message tracking for failed ingestions
209 + - `PortfolioIndex.Schemas.Chunk` - Ecto schema for document chunks with pgvector embeddings
210 + - Native `Pgvector.Ecto.Vector` type for similarity search
211 + - Document relationship with cascade delete
212 + - Character offset tracking (`start_char`, `end_char`)
213 + - Token count for LLM context budgeting
214 + - `PortfolioIndex.Schemas.Queries` - Query helpers for schema operations
215 + - `get_collection_by_name/2` - Fetch collection by name
216 + - `get_or_create_collection/3` - Upsert collection
217 + - `list_documents_by_status/3` - Filter documents by processing status
218 + - `get_document_with_chunks/2` - Load document with preloaded chunks
219 + - `similarity_search/3` - pgvector cosine similarity search
220 + - `count_chunks_without_embedding/1` - Count unprocessed chunks
221 + - `get_failed_documents/2` - Fetch documents for retry
222 + - `soft_delete_document/2` - Mark document as deleted
223 + - Database migration for document management tables
224 + - `portfolio_collections` with unique name index
225 + - `portfolio_documents` with status, source_id, and content_hash indexes
226 + - `portfolio_chunks` with HNSW vector index for fast similarity search
227 +
228 + #### Query Processing Pipeline
229 + - `PortfolioIndex.RAG.Pipeline.Context` struct for pipeline state tracking
230 + - Flows through RAG pipeline accumulating intermediate results
231 + - Tracks query transformations, routing, retrieval, and generation
232 + - Supports functional composition with pipe operator
233 + - Error propagation and halt semantics
234 + - `PortfolioIndex.Adapters.QueryRewriter.LLM` - LLM-based query cleaning
235 + - Removes greetings, filler phrases, conversational noise
236 + - Preserves technical terms and entity names
237 + - Custom prompt support
238 + - `PortfolioIndex.Adapters.QueryExpander.LLM` - LLM-based query expansion
239 + - Adds synonyms and related terms for better recall
240 + - Expands abbreviations (ML -> machine learning)
241 + - Tracks added terms for debugging
242 + - `PortfolioIndex.Adapters.QueryDecomposer.LLM` - LLM-based query decomposition
243 + - Breaks complex questions into 2-4 simpler sub-questions
244 + - Enables parallel retrieval for multi-faceted queries
245 + - JSON response parsing with fallback handling
246 + - `PortfolioIndex.RAG.QueryProcessor` - unified query processing module
247 + - `rewrite/2` - Apply query rewriting to pipeline context
248 + - `expand/2` - Apply query expansion to pipeline context
249 + - `decompose/2` - Apply query decomposition to pipeline context
250 + - `process/2` - Run all processing steps with skip options
251 + - `effective_query/1` - Get best query for retrieval
252 +
253 + #### Chunker Enhancements
254 + - **Separators module** - Centralized language-specific separators for 17+ formats
255 + - Language support: Elixir, Ruby, PHP, Python, JavaScript, TypeScript, Vue, HTML
256 + - Document formats: doc, docx, epub, latex, odt, pdf, rtf
257 + - Markdown with header-aware splitting
258 + - **Config module** - NimbleOptions-based configuration validation with compile-time schema
259 + - **Pluggable `get_chunk_size` function** - Token-based chunking support across all strategies
260 + - Character, byte, word, or custom tokenizer-based sizing
261 + - Defaults to `String.length/1` for backwards compatibility
262 +
263 + #### Token Utilities
264 + - **Tokens module** - Centralized token estimation utilities
265 + - `Tokens.estimate/2` - Estimate token count from text (~4 chars/token heuristic)
266 + - `Tokens.sizer/1` - Get sizing function for token-based chunking
267 + - `Tokens.to_chars/2` - Convert token count to character count
268 + - `Tokens.from_chars/2` - Convert character count to token count
269 + - `Tokens.default_ratio/0` - Returns the default chars-per-token ratio (4)
270 +
271 + #### Chunker Config Enhancement
272 + - **`size_unit` option** - Specify `:characters` or `:tokens` for chunk sizing
273 + - `:tokens` auto-configures `get_chunk_size` with token estimation
274 + - `:characters` (default) uses `String.length/1`
275 + - Implements `PortfolioCore.Ports.Chunker` port specification for `size_unit`
276 +
277 + #### Chunker Output Enhancement
278 + - **`token_count` in metadata** - All chunkers now include estimated token count
279 + - Useful for LLM context window budgeting
280 + - Approximately `char_count / 4`
281 + - Included in: Recursive, Character, Paragraph, Sentence, Semantic chunkers
282 +
283 + ### Changed
284 +
285 + - All chunker adapters now support `:get_chunk_size` option for custom size measurement
286 + - Recursive chunker uses new Separators module for format-specific splitting
287 + - Added nimble_options dependency for robust configuration validation
288 +
289 + ### Documentation
290 +
291 + - Added technical documentation in `docs/20251230/chunker-enhancements/`
292 + - `overview.md` - Feature overview and migration guide
293 + - `separators.md` - Language separator reference
294 + - `tokenization.md` - Token-based chunking guide
295 +
10 296 ## [0.3.0] - 2025-12-28
11 297
12 298 ### Added
  @@ -81,7 +367,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
81 367
82 368 - Initial release of PortfolioIndex
83 369
84 - [Unreleased]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.3.0...HEAD
370 + [Unreleased]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.3.1...HEAD
371 + [0.3.1]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.3.0...v0.3.1
85 372 [0.3.0]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.2.0...v0.3.0
86 373 [0.2.0]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.1.1...v0.2.0
87 374 [0.1.1]: https://github.com/nshkrdotcom/portfolio_index/compare/v0.1.0...v0.1.1
  @@ -75,7 +75,7 @@ Add `portfolio_index` to your list of dependencies in `mix.exs`:
75 75 ```elixir
76 76 def deps do
77 77 [
78 - {:portfolio_index, "~> 0.2.0"}
78 + {:portfolio_index, "~> 0.3.1"}
79 79 ]
80 80 end
81 81 ```
  @@ -255,7 +255,40 @@ config :boltx, Boltx,
255 255
256 256 | Adapter | Strategy | Features |
257 257 |---------|----------|----------|
258 - | `Recursive` | Recursive text splitting | Format-aware (markdown, code, plain) |
258 + | `Recursive` | Recursive text splitting | Format-aware for 17+ languages |
259 + | `Character` | Character-based | Boundary modes: word, sentence, none |
260 + | `Sentence` | Sentence-based | NLP tokenization, abbreviation handling |
261 + | `Paragraph` | Paragraph-based | Intelligent merge/split at boundaries |
262 + | `Semantic` | Embedding similarity | Groups by semantic coherence |
263 +
264 + #### Supported Formats
265 +
266 + | Category | Formats |
267 + |----------|---------|
268 + | Languages | Elixir, Ruby, PHP, Python, JavaScript, TypeScript, Vue |
269 + | Markup | Markdown, HTML, LaTeX |
270 + | Documents | doc, docx, epub, odt, pdf, rtf |
271 +
272 + #### Token-Based Chunking
273 +
274 + All chunkers support custom size measurement via `:get_chunk_size`:
275 +
276 + ```elixir
277 + # Character-based (default)
278 + Recursive.chunk(text, :elixir, %{chunk_size: 1000})
279 +
280 + # Token-based (for LLM context limits)
281 + Recursive.chunk(text, :elixir, %{
282 + chunk_size: 256,
283 + get_chunk_size: &MyTokenizer.count_tokens/1
284 + })
285 +
286 + # Byte-based (for storage limits)
287 + Recursive.chunk(text, :plain, %{
288 + chunk_size: 4096,
289 + get_chunk_size: &byte_size/1
290 + })
291 + ```
259 292
260 293 ## RAG Strategies
  @@ -1,16 +1,33 @@
1 1 {<<"links">>,
2 2 [{<<"GitHub">>,<<"https://github.com/nshkrdotcom/portfolio_index">>}]}.
3 3 {<<"name">>,<<"portfolio_index">>}.
4 - {<<"version">>,<<"0.3.0">>}.
4 + {<<"version">>,<<"0.3.1">>}.
5 5 {<<"description">>,
6 6 <<"Production adapters and pipelines for PortfolioCore. Vector stores, graph stores, embedders, Broadway pipelines, and advanced RAG strategies.">>}.
7 7 {<<"elixir">>,<<"~> 1.15">>}.
8 8 {<<"app">>,<<"portfolio_index">>}.
9 9 {<<"licenses">>,[<<"MIT">>]}.
10 10 {<<"files">>,
11 - [<<"lib">>,<<"lib/portfolio_index.ex">>,<<"lib/portfolio_index">>,
11 + [<<"lib">>,<<"lib/portfolio_index.ex">>,<<"lib/mix">>,<<"lib/mix/tasks">>,
12 + <<"lib/mix/tasks/portfolio.gen.embedding_migration.ex">>,
13 + <<"lib/mix/tasks/portfolio.install.ex">>,<<"lib/portfolio_index">>,
14 + <<"lib/portfolio_index/maintenance">>,
15 + <<"lib/portfolio_index/maintenance/progress.ex">>,
16 + <<"lib/portfolio_index/maintenance.ex">>,
12 17 <<"lib/portfolio_index/application.ex">>,
13 - <<"lib/portfolio_index/telemetry.ex">>,<<"lib/portfolio_index/adapters">>,
18 + <<"lib/portfolio_index/telemetry.ex">>,<<"lib/portfolio_index/schemas">>,
19 + <<"lib/portfolio_index/schemas/collection.ex">>,
20 + <<"lib/portfolio_index/schemas/queries.ex">>,
21 + <<"lib/portfolio_index/schemas/evaluation_run.ex">>,
22 + <<"lib/portfolio_index/schemas/document.ex">>,
23 + <<"lib/portfolio_index/schemas/chunk.ex">>,
24 + <<"lib/portfolio_index/schemas/test_case.ex">>,
25 + <<"lib/portfolio_index/adapters">>,
26 + <<"lib/portfolio_index/adapters/query_expander">>,
27 + <<"lib/portfolio_index/adapters/query_expander/llm.ex">>,
28 + <<"lib/portfolio_index/adapters/collection_selector">>,
29 + <<"lib/portfolio_index/adapters/collection_selector/rule_based.ex">>,
30 + <<"lib/portfolio_index/adapters/collection_selector/llm.ex">>,
14 31 <<"lib/portfolio_index/adapters/graph_store">>,
15 32 <<"lib/portfolio_index/adapters/graph_store/neo4j.ex">>,
16 33 <<"lib/portfolio_index/adapters/graph_store/neo4j">>,
  @@ -20,16 +37,27 @@
20 37 <<"lib/portfolio_index/adapters/graph_store/neo4j/community.ex">>,
21 38 <<"lib/portfolio_index/adapters/document_store">>,
22 39 <<"lib/portfolio_index/adapters/document_store/postgres.ex">>,
40 + <<"lib/portfolio_index/adapters/retrieval_metrics">>,
41 + <<"lib/portfolio_index/adapters/retrieval_metrics/standard.ex">>,
23 42 <<"lib/portfolio_index/adapters/reranker">>,
24 43 <<"lib/portfolio_index/adapters/reranker/passthrough.ex">>,
25 44 <<"lib/portfolio_index/adapters/reranker/llm.ex">>,
45 + <<"lib/portfolio_index/adapters/query_decomposer">>,
46 + <<"lib/portfolio_index/adapters/query_decomposer/llm.ex">>,
26 47 <<"lib/portfolio_index/adapters/chunker">>,
27 48 <<"lib/portfolio_index/adapters/chunker/sentence.ex">>,
28 49 <<"lib/portfolio_index/adapters/chunker/semantic.ex">>,
50 + <<"lib/portfolio_index/adapters/chunker/separators.ex">>,
29 51 <<"lib/portfolio_index/adapters/chunker/character.ex">>,
30 52 <<"lib/portfolio_index/adapters/chunker/recursive.ex">>,
53 + <<"lib/portfolio_index/adapters/chunker/config.ex">>,
54 + <<"lib/portfolio_index/adapters/chunker/tokens.ex">>,
31 55 <<"lib/portfolio_index/adapters/chunker/paragraph.ex">>,
56 + <<"lib/portfolio_index/adapters/query_rewriter">>,
57 + <<"lib/portfolio_index/adapters/query_rewriter/llm.ex">>,
32 58 <<"lib/portfolio_index/adapters/embedder">>,
59 + <<"lib/portfolio_index/adapters/embedder/function.ex">>,
60 + <<"lib/portfolio_index/adapters/embedder/bumblebee.ex">>,
33 61 <<"lib/portfolio_index/adapters/embedder/gemini.ex">>,
34 62 <<"lib/portfolio_index/adapters/embedder/open_ai.ex">>,
35 63 <<"lib/portfolio_index/adapters/llm">>,
  @@ -40,9 +68,20 @@
40 68 <<"lib/portfolio_index/adapters/vector_store/pgvector">>,
41 69 <<"lib/portfolio_index/adapters/vector_store/pgvector/hybrid.ex">>,
42 70 <<"lib/portfolio_index/adapters/vector_store/pgvector/fulltext.ex">>,
71 + <<"lib/portfolio_index/adapters/vector_store/memory.ex">>,
43 72 <<"lib/portfolio_index/adapters/vector_store/pgvector.ex">>,
44 73 <<"lib/portfolio_index/postgrex_types.ex">>,
45 - <<"lib/portfolio_index/graph_rag">>,
74 + <<"lib/portfolio_index/telemetry">>,
75 + <<"lib/portfolio_index/telemetry/rag.ex">>,
76 + <<"lib/portfolio_index/telemetry/embedder.ex">>,
77 + <<"lib/portfolio_index/telemetry/vector_store.ex">>,
78 + <<"lib/portfolio_index/telemetry/logger.ex">>,
79 + <<"lib/portfolio_index/telemetry/llm.ex">>,
80 + <<"lib/portfolio_index/embedder">>,
81 + <<"lib/portfolio_index/embedder/dimension_detector.ex">>,
82 + <<"lib/portfolio_index/embedder/registry.ex">>,
83 + <<"lib/portfolio_index/embedder/config.ex">>,
84 + <<"lib/portfolio_index/evaluation.ex">>,<<"lib/portfolio_index/graph_rag">>,
46 85 <<"lib/portfolio_index/graph_rag/entity_extractor.ex">>,
47 86 <<"lib/portfolio_index/graph_rag/community_detector.ex">>,
48 87 <<"lib/portfolio_index/graph_rag/community_summarizer.ex">>,
  @@ -52,13 +91,28 @@
52 91 <<"lib/portfolio_index/pipelines/producers">>,
53 92 <<"lib/portfolio_index/pipelines/producers/file_producer.ex">>,
54 93 <<"lib/portfolio_index/pipelines/producers/ets_producer.ex">>,
55 - <<"lib/portfolio_index/rag">>,<<"lib/portfolio_index/rag/strategy.ex">>,
94 + <<"lib/portfolio_index/evaluation">>,
95 + <<"lib/portfolio_index/evaluation/generator.ex">>,
96 + <<"lib/portfolio_index/rag">>,
97 + <<"lib/portfolio_index/rag/self_correcting_answer.ex">>,
98 + <<"lib/portfolio_index/rag/self_correcting_search.ex">>,
99 + <<"lib/portfolio_index/rag/strategy.ex">>,
100 + <<"lib/portfolio_index/rag/query_processor.ex">>,
56 101 <<"lib/portfolio_index/rag/adapter_resolver.ex">>,
102 + <<"lib/portfolio_index/rag/pipeline">>,
103 + <<"lib/portfolio_index/rag/pipeline/context.ex">>,
104 + <<"lib/portfolio_index/rag/reranker.ex">>,
57 105 <<"lib/portfolio_index/rag/strategies">>,
58 106 <<"lib/portfolio_index/rag/strategies/agentic.ex">>,
59 107 <<"lib/portfolio_index/rag/strategies/hybrid.ex">>,
60 108 <<"lib/portfolio_index/rag/strategies/self_rag.ex">>,
61 109 <<"lib/portfolio_index/rag/strategies/graph_rag.ex">>,
110 + <<"lib/portfolio_index/vector_store">>,
111 + <<"lib/portfolio_index/vector_store/backend.ex">>,
112 + <<"lib/portfolio_index/vector_store/collections.ex">>,
113 + <<"lib/portfolio_index/vector_store/index_manager.ex">>,
114 + <<"lib/portfolio_index/vector_store/soft_delete.ex">>,
115 + <<"lib/portfolio_index/vector_store/search.ex">>,
62 116 <<"lib/portfolio_index/repo.ex">>,<<"assets">>,
63 117 <<"assets/portfolio_index.svg">>,<<".formatter.exs">>,<<"mix.exs">>,
64 118 <<"README.md">>,<<"LICENSE">>,<<"CHANGELOG.md">>]}.
  @@ -66,7 +120,7 @@
66 120 [[{<<"name">>,<<"portfolio_core">>},
67 121 {<<"app">>,<<"portfolio_core">>},
68 122 {<<"optional">>,false},
69 - {<<"requirement">>,<<"~> 0.3.0">>},
123 + {<<"requirement">>,<<"~> 0.3.1">>},
70 124 {<<"repository">>,<<"hexpm">>}],
71 125 [{<<"name">>,<<"ecto_sql">>},
72 126 {<<"app">>,<<"ecto_sql">>},
  @@ -83,6 +137,11 @@
83 137 {<<"optional">>,false},
84 138 {<<"requirement">>,<<"~> 0.2">>},
85 139 {<<"repository">>,<<"hexpm">>}],
140 + [{<<"name">>,<<"hnswlib">>},
141 + {<<"app">>,<<"hnswlib">>},
142 + {<<"optional">>,false},
143 + {<<"requirement">>,<<"~> 0.1.6">>},
144 + {<<"repository">>,<<"hexpm">>}],
86 145 [{<<"name">>,<<"boltx">>},
87 146 {<<"app">>,<<"boltx">>},
88 147 {<<"optional">>,false},
  @@ -91,17 +150,17 @@
91 150 [{<<"name">>,<<"gemini_ex">>},
92 151 {<<"app">>,<<"gemini_ex">>},
93 152 {<<"optional">>,false},
94 - {<<"requirement">>,<<"~> 0.8.6">>},
153 + {<<"requirement">>,<<"~> 0.8.8">>},
95 154 {<<"repository">>,<<"hexpm">>}],
96 155 [{<<"name">>,<<"claude_agent_sdk">>},
97 156 {<<"app">>,<<"claude_agent_sdk">>},
98 157 {<<"optional">>,false},
99 - {<<"requirement">>,<<"~> 0.6.10">>},
158 + {<<"requirement">>,<<"~> 0.7.2">>},
100 159 {<<"repository">>,<<"hexpm">>}],
101 160 [{<<"name">>,<<"codex_sdk">>},
102 161 {<<"app">>,<<"codex_sdk">>},
103 162 {<<"optional">>,false},
104 - {<<"requirement">>,<<"~> 0.4.3">>},
163 + {<<"requirement">>,<<"~> 0.4.5">>},
105 164 {<<"repository">>,<<"hexpm">>}],
106 165 [{<<"name">>,<<"req">>},
107 166 {<<"app">>,<<"req">>},
  @@ -147,5 +206,10 @@
147 206 {<<"app">>,<<"jason">>},
148 207 {<<"optional">>,false},
149 208 {<<"requirement">>,<<"~> 1.4">>},
209 + {<<"repository">>,<<"hexpm">>}],
210 + [{<<"name">>,<<"nimble_options">>},
211 + {<<"app">>,<<"nimble_options">>},
212 + {<<"optional">>,false},
213 + {<<"requirement">>,<<"~> 1.1">>},
150 214 {<<"repository">>,<<"hexpm">>}]]}.
151 215 {<<"build_tools">>,[<<"mix">>]}.
  @@ -0,0 +1,136 @@
1 + defmodule Mix.Tasks.Portfolio.Gen.EmbeddingMigration do
2 + @moduledoc """
3 + Generates a migration for changing embedding dimensions.
4 +
5 + ## Usage
6 +
7 + mix portfolio.gen.embedding_migration --dimension 1536
8 +
9 + This is useful when switching embedding models with different dimensions.
10 +
11 + ## Options
12 +
13 + - `--dimension` - New embedding dimension (required)
14 + - `--table` - Table name (default: portfolio_chunks)
15 + - `--column` - Column name (default: embedding)
16 + - `--migrations-path` - Path for migrations (default: priv/repo/migrations)
17 + """
18 +
19 + use Mix.Task
20 +
21 + import Mix.Generator
22 +
23 + @shortdoc "Generate migration to change embedding dimensions"
24 +
25 + @impl Mix.Task
26 + def run(args) do
27 + {opts, _, _} =
28 + OptionParser.parse(args,
29 + strict: [
30 + dimension: :integer,
31 + table: :string,
32 + column: :string,
33 + migrations_path: :string
34 + ]
35 + )
36 +
37 + dimension = opts[:dimension]
38 + table = opts[:table] || "portfolio_chunks"
39 + column = opts[:column] || "embedding"
40 + migrations_path = opts[:migrations_path]
41 +
42 + unless dimension do
43 + Mix.raise(
44 + "--dimension option is required. Example: mix portfolio.gen.embedding_migration --dimension 1536"
45 + )
46 + end
47 +
48 + generate_migration(dimension, table, column, migrations_path)
49 + end
50 +
51 + defp generate_migration(dimension, table, column, migrations_path) do
52 + migrations_dir = migrations_path || default_migrations_path()
53 +
54 + # Ensure the directory exists
55 + File.mkdir_p!(migrations_dir)
56 +
57 + timestamp = Calendar.strftime(DateTime.utc_now(), "%Y%m%d%H%M%S")
58 + filename = "#{timestamp}_update_embedding_dimensions.exs"
59 + path = Path.join(migrations_dir, filename)
60 +
61 + content = migration_content(dimension, table, column)
62 +
63 + create_file(path, content)
64 +
65 + Mix.shell().info("""
66 +
67 + Generated migration: #{path}
68 +
69 + This migration will:
70 + 1. Drop the HNSW index on the #{column} column
71 + 2. Alter the #{column} column to #{dimension} dimensions
72 + 3. Recreate the HNSW index
73 +
74 + Run the migration with:
75 +
76 + mix ecto.migrate
77 +
78 + After migrating, re-embed all documents with:
79 +
80 + mix portfolio.reembed
81 +
82 + """)
83 + end
84 +
85 + defp default_migrations_path do
86 + Path.join(["priv", "repo", "migrations"])
87 + end
88 +
89 + defp migration_content(dimension, table, column) do
90 + """
91 + defmodule PortfolioIndex.Repo.Migrations.UpdateEmbeddingDimensions do
92 + use Ecto.Migration
93 +
94 + @table :#{table}
95 + @column :#{column}
96 + @new_dimension #{dimension}
97 + @old_dimension 384
98 +
99 + def up do
100 + # Drop the existing HNSW index
101 + execute "DROP INDEX IF EXISTS #{table}_#{column}_idx"
102 +
103 + # Alter the embedding column to new dimensions
104 + # This requires dropping and recreating the column due to pgvector constraints
105 + execute \"\"\"
106 + ALTER TABLE #{table}
107 + ALTER COLUMN #{column} TYPE vector(@new_dimension)
108 + \"\"\"
109 +
110 + # Recreate the HNSW index with the new dimensions
111 + execute \"\"\"
112 + CREATE INDEX #{table}_#{column}_idx ON #{table}
113 + USING hnsw (#{column} vector_cosine_ops)
114 + \"\"\"
115 + end
116 +
117 + def down do
118 + # Drop the new index
119 + execute "DROP INDEX IF EXISTS #{table}_#{column}_idx"
120 +
121 + # Revert to previous dimensions
122 + execute \"\"\"
123 + ALTER TABLE #{table}
124 + ALTER COLUMN #{column} TYPE vector(@old_dimension)
125 + \"\"\"
126 +
127 + # Recreate the original index
128 + execute \"\"\"
129 + CREATE INDEX #{table}_#{column}_idx ON #{table}
130 + USING hnsw (#{column} vector_cosine_ops)
131 + \"\"\"
132 + end
133 + end
134 + """
135 + end
136 + end
  @@ -0,0 +1,205 @@
1 + defmodule Mix.Tasks.Portfolio.Install do
2 + @moduledoc """
3 + Installs PortfolioIndex into a Phoenix application.
4 +
5 + ## Usage
6 +
7 + mix portfolio.install
8 +
9 + ## What it does
10 +
11 + 1. Creates required database migrations
12 + 2. Prints configuration instructions
13 + 3. Provides next steps for setup
14 +
15 + ## Options
16 +
17 + - `--repo` - Ecto repo module (default: inferred from app)
18 + - `--dimension` - Embedding vector dimension (default: 384)
19 + - `--no-migrations` - Skip migration generation
20 + """
21 +
22 + use Mix.Task
23 +
24 + import Mix.Generator
25 +
26 + @shortdoc "Install PortfolioIndex in your application"
27 +
28 + @migration_template """
29 + defmodule <%= @repo %>.Migrations.CreatePortfolioTables do
30 + use Ecto.Migration
31 +
32 + def up do
33 + execute "CREATE EXTENSION IF NOT EXISTS vector"
34 +
35 + create table(:portfolio_collections, primary_key: false) do
36 + add :id, :binary_id, primary_key: true
37 + add :name, :string, null: false
38 + add :description, :text
39 + add :metadata, :map, default: %{}
40 +
41 + timestamps(type: :utc_datetime)
42 + end
43 +
44 + create unique_index(:portfolio_collections, [:name])
45 +
46 + create table(:portfolio_documents, primary_key: false) do
47 + add :id, :binary_id, primary_key: true
48 + add :source_id, :string
49 + add :content_hash, :string
50 + add :title, :string
51 + add :source_path, :string
52 + add :metadata, :map, default: %{}
53 + add :status, :string, default: "pending"
54 + add :error_message, :text
55 + add :chunk_count, :integer, default: 0
56 + add :collection_id, references(:portfolio_collections, type: :binary_id, on_delete: :nilify_all)
57 +
58 + timestamps(type: :utc_datetime)
59 + end
60 +
61 + create index(:portfolio_documents, [:source_id])
62 + create index(:portfolio_documents, [:content_hash])
63 + create index(:portfolio_documents, [:status])
64 + create index(:portfolio_documents, [:collection_id])
65 +
66 + create table(:portfolio_chunks, primary_key: false) do
67 + add :id, :binary_id, primary_key: true
68 + add :content, :text, null: false
69 + add :embedding, :vector, size: <%= @dimension %>, null: true
70 + add :chunk_index, :integer, default: 0
71 + add :token_count, :integer
72 + add :start_char, :integer
73 + add :end_char, :integer
74 + add :metadata, :map, default: %{}
75 + add :document_id, references(:portfolio_documents, type: :binary_id, on_delete: :delete_all)
76 +
77 + timestamps(type: :utc_datetime)
78 + end
79 +
80 + create index(:portfolio_chunks, [:document_id])
81 +
82 + execute \"\"\"
83 + CREATE INDEX portfolio_chunks_embedding_idx ON portfolio_chunks
84 + USING hnsw (embedding vector_cosine_ops)
85 + \"\"\"
86 + end
87 +
88 + def down do
89 + drop table(:portfolio_chunks)
90 + drop table(:portfolio_documents)
91 + drop table(:portfolio_collections)
92 + # Note: We don't drop the vector extension as it may be used by other tables
93 + end
94 + end
95 + """
96 +
97 + @impl Mix.Task
98 + def run(args) do
99 + {opts, _, _} =
100 + OptionParser.parse(args,
101 + strict: [
102 + repo: :string,
103 + dimension: :integer,
104 + no_migrations: :boolean
105 + ]
106 + )
107 +
108 + repo = opts[:repo] || infer_repo()
109 + dimension = opts[:dimension] || 384
110 + skip_migrations = opts[:no_migrations] || false
111 +
112 + unless skip_migrations do
113 + generate_migration(repo, dimension)
114 + end
115 +
116 + print_instructions(repo, dimension)
117 + end
118 +
119 + defp generate_migration(repo, dimension) do
120 + repo_underscore =
121 + repo
122 + |> String.replace(".", "/")
123 + |> Macro.underscore()
124 + |> String.replace("/", "_")
125 +
126 + migrations_path = Path.join(["priv", repo_underscore, "migrations"])
127 + File.mkdir_p!(migrations_path)
128 +
129 + timestamp = Calendar.strftime(DateTime.utc_now(), "%Y%m%d%H%M%S")
130 + filename = "#{timestamp}_create_portfolio_tables.exs"
131 + path = Path.join(migrations_path, filename)
132 +
133 + content = EEx.eval_string(@migration_template, assigns: [repo: repo, dimension: dimension])
134 +
135 + create_file(path, content)
136 + end
137 +
138 + defp print_instructions(repo, dimension) do
139 + Mix.shell().info("""
140 +
141 + PortfolioIndex installation complete!
142 +
143 + Configuration for #{repo} with #{dimension} dimensions.
144 +
145 + == Next Steps ==
146 +
147 + 1. Run the migration:
148 +
149 + mix ecto.migrate
150 +
151 + 2. Configure pgvector types in your repo config:
152 +
153 + # config/config.exs
154 + config :my_app, #{repo},
155 + types: MyApp.PostgrexTypes
156 +
157 + 3. Create the types module (if not already present):
158 +
159 + # lib/my_app/postgrex_types.ex
160 + Postgrex.Types.define(
161 + MyApp.PostgrexTypes,
162 + [Pgvector.Extensions.Vector] ++ Ecto.Adapters.Postgres.extensions(),
163 + []
164 + )
165 +
166 + 4. Configure the embedder in your config:
167 +
168 + # config/config.exs
169 + config :portfolio_index,
170 + embedder: PortfolioIndex.Adapters.Embedder.Gemini,
171 + embedding: [
172 + default_dimensions: #{dimension}
173 + ]
174 +
175 + 5. Start using PortfolioIndex:
176 +
177 + # Ingest a document
178 + PortfolioIndex.Maintenance.diagnostics(#{repo})
179 +
180 + == Optional: Broadway Pipelines ==
181 +
182 + For async document processing, add the pipelines to your supervision tree:
183 +
184 + children = [
185 + #{repo},
186 + {PortfolioIndex.Pipelines.Ingestion, [repo: #{repo}]},
187 + {PortfolioIndex.Pipelines.Embedding, [repo: #{repo}]}
188 + ]
189 +
190 + """)
191 + end
192 +
193 + defp infer_repo do
194 + case Mix.Project.config()[:app] do
195 + nil ->
196 + "MyApp.Repo"
197 +
198 + app ->
199 + app
200 + |> to_string()
201 + |> Macro.camelize()
202 + |> Kernel.<>(".Repo")
203 + end
204 + end
205 + end
Loading more files…