crucible_datasets

# Changelog

All notable changes to this project will be documented in this file.

## [0.5.2] - 2025-12-25

### Added

- **NoRobots Loader:**
  - New `CrucibleDatasets.Loader.NoRobots` module for instruction-following dataset
  - Human-written instruction-response pairs (9,500 examples)
  - Support for `:train` and `:test` splits
  - Category metadata (Open QA, Generation, Brainstorm, Rewrite, Summarize, etc.)
  - Comprehensive test coverage (8 tests)

- **Telemetry Events:**
  - `[:crucible_datasets, :load, :start]` - Emitted when loading begins
  - `[:crucible_datasets, :load, :stop]` - Emitted when loading completes successfully
  - `[:crucible_datasets, :load, :exception]` - Emitted when loading fails
  - `[:crucible_datasets, :cache, :hit]` - Emitted on cache hit
  - `[:crucible_datasets, :cache, :miss]` - Emitted on cache miss
  - All events include dataset name and timing information

- **Cache Eviction:**
  - Implemented LRU-based cache eviction when cache exceeds 10GB limit
  - Evicts oldest datasets based on modification time
  - Frees space incrementally until under the limit

- **Development Dependencies:**
  - Added `credo ~> 1.7` for static code analysis
  - Run `mix credo --strict` for code quality checks

### Changed

- Version bump from 0.5.1 to 0.5.2
- Updated Registry with NoRobots metadata
- Updated Loader to dispatch NoRobots requests
- Enhanced Loader module documentation with telemetry event details

### Documentation

- Updated README with NoRobots dataset information
- Added telemetry events section to README
- Added `mix credo --strict` to static analysis commands
- 150 tests passing (8 new for NoRobots)

---

## [0.5.1] - 2025-12-23

### Added

- **inspect_ai Parity Features:**
  - **MemoryDataset:** New `CrucibleDatasets.MemoryDataset` module for lightweight in-memory dataset construction
    - `from_list/2` - Create datasets from lists of items
    - `from_samples/2` - Alias for clarity when using Sample structs
    - Auto-ID generation for items without explicit IDs
    - Customizable dataset name and version
  - **Dataset Extensions:** Enhanced `CrucibleDatasets.Dataset` with new methods
    - `filter/2` - Filter dataset items by predicate function
    - `sort/2` and `sort/3` - Sort items by key atom or function (ascending/descending)
    - `slice/2` and `slice/3` - Slice datasets by range or start/count
    - `shuffle_choices/2` - Shuffle multiple-choice options while preserving correct answer mapping
  - **FieldMapping:** New `CrucibleDatasets.FieldMapping` module for declarative field mapping
    - `new/1` - Create field mapping specifications
    - `apply/2` - Apply mapping to raw records
    - Support for input/expected/id/choices/metadata field mapping
    - Transform functions for field value processing
    - Flexible field lookup (atom and string keys)
  - **Generic Loader:** New `CrucibleDatasets.Loader.Generic` module
    - Load datasets from JSONL, JSON, and CSV files
    - Auto-format detection by file extension
    - Field mapping support for flexible schema handling
    - Options for limit, shuffle, auto-ID generation
    - Reproducible shuffling with seed support

### Changed

- Version bump from 0.5.0 to 0.5.1
- Enhanced field lookup to support both atom and string keys for better flexibility

### Documentation

- Comprehensive tests for all new modules (142 total tests, all passing)
- API documentation for MemoryDataset, FieldMapping, and Loader.Generic
- Updated README with version 0.5.1

---

## [0.5.0] - 2025-12-22

### Breaking Changes

This release reverts to the v0.3.0 codebase, removing the HuggingFace Hub integration that was added in v0.4.x. **Versions 0.4.0 and 0.4.1 are deprecated.**

### Removed

- **HuggingFace Hub Integration:**
  - Removed `hf_hub` dependency
  - Removed `explorer` dependency (Parquet support)
  - Removed `vix` dependency (image processing)
  - Removed `Source.HuggingFace` module
  - Removed `Format.Parquet` module
  - Removed `DatasetDict` and `IterableDataset` modules
  - Removed Features schema system
  - Removed streaming support
  - Removed vision/chat/code/preference/reasoning/rubric loaders
  - Removed `load_dataset/2` HuggingFace-style API

### Retained

- Core dataset management (GSM8K, HumanEval, MMLU loaders)
- CrucibleIR `DatasetRef` integration from v0.3.0
- Evaluation metrics (exact match, F1, BLEU, ROUGE)
- Dataset registry and metadata
- Result persistence and export (CSV, JSONL, Markdown, HTML)
- Sampling strategies (random, stratified, k-fold)
- Local JSONL file support
- Caching with version tracking

### Why This Change

The HuggingFace integration was experimental and added heavy dependencies (Explorer, Vix/libvips) that complicated installation. This library returns to its focused purpose: lightweight dataset management for AI research benchmarks within the Crucible framework.

---

## [0.4.1] - 2025-12-21 (DEPRECATED)

### Added

- **HuggingFace Parity API:** `load_dataset/2` with repo_id/config/split/streaming options
- **Data Discovery:** New `CrucibleDatasets.DataFiles` resolver using `HfHub.Api.list_repo_tree/2` and `dataset_splits/2`
- **Dataset Types:** DatasetDict and IterableDataset wired to the public API
- **Streaming:** JSONL streaming support; Parquet streaming supported with batch warning
- **Features + Images:** Features integrated into Dataset with Image decode via Vix/libvips
- **New Loaders:** Real MMLU, HumanEval, and vision datasets (caltech101, oxford_flowers102, oxford_iiit_pet, stanford_cars)
- **Examples/Docs:** New examples for `load_dataset`, DatasetDict, streaming, and vision; docs updated

### Changed

- **Registry + Loader Dispatcher:** Expanded to include all tinker datasets
- **Live Tests:** `mix test.live` uses `@tag :live`
- **Version:** 0.4.0 -> 0.4.1

## [0.4.0] - 2025-12-21 (DEPRECATED)

### Fixed

- **HuggingFace Source:** Fixed nil path handling in `filter_by_config/2` and `filter_by_split/2` - HuggingFace API returns files with `rfilename` key instead of `path`
- **Test Configuration:** Integration tests are now excluded by default to avoid slow network-dependent tests. Run with `mix test --include integration` to include them.

### Added

- **Source Abstraction Layer:**
  - New `Source` behaviour for data source abstraction
  - `Source.Local` - Local filesystem source with file listing and streaming
  - `Source.HuggingFace` - HuggingFace Hub source with download/stream support
  - Unified API: `list_files/2`, `download/3`, `stream/3`, `exists?/2`
  - Extensible design for future sources (S3, GCS, etc.)

- **Format Parser Layer:**
  - New `Format` behaviour for file format parsing
  - `Format.JSONL` - JSON Lines parser with streaming support
  - `Format.JSON` - JSON file parser
  - `Format.CSV` - CSV parser with header detection
  - `Format.Parquet` - Parquet parser via Explorer
  - Auto-detection of formats by file extension

- **Dataset Operations:**
  - `Dataset.map/2` - Transform each item
  - `Dataset.filter/2` - Filter items by predicate
  - `Dataset.shuffle/2` - Randomize order (with optional seed)
  - `Dataset.select/2` - Select specific columns
  - `Dataset.take/2`, `Dataset.skip/2` - Pagination
  - `Dataset.slice/3` - Slice with negative index support
  - `Dataset.batch/2` - Group into batches
  - `Dataset.concat/1,2` - Concatenate datasets
  - `Dataset.split/2` - Train/test splitting
  - `Dataset.shard/2` - Create shards for distributed processing
  - Column operations: `rename_column/3`, `add_column/3`, `remove_columns/2`
  - `Dataset.unique/2`, `Dataset.sort/2`, `Dataset.flatten/2`
  - Enumerable protocol for `for` comprehensions and Enum functions
  - Access behaviour for bracket notation (`dataset[0]`)

- **DatasetDict:**
  - Dictionary of splits (train/test/validation)
  - Python-like bracket access: `dd["train"]`
  - Operations across all splits: `map/2`, `filter/2`, `select/2`, `shuffle/2`
  - `flatten/1` - Combine all splits into single dataset
  - Enumerable protocol for iteration over splits

- **IterableDataset:**
  - Lazy, streaming dataset for memory-efficient processing
  - Lazy transformations: `map/2`, `filter/2`, `batch/2`
  - Buffered shuffle with seed support
  - Conversion: `from_stream/2`, `from_dataset/1`, `to_dataset/1`, `to_list/1`
  - Enumerable protocol for lazy consumption

- **Features Schema System:**
  - Type system for dataset columns
  - `Value` - Scalar types (int8-64, uint8-64, float16-64, string, bool, binary)
  - `ClassLabel` - Categorical with encode/decode
  - `Sequence` - Lists with fixed length support
  - `Image` - Image data with mode (RGB, L, RGBA)
  - `Audio` - Audio data with sample rate
  - Schema inference from dataset items
  - Value validation and casting

- **Enhanced Loaders:**
  - MMLU: HuggingFace integration
  - HumanEval: HuggingFace integration

### Changed

- Version bump from 0.3.0 to 0.4.0
- Loaders now use Source/Format abstractions internally
- All tests passing (282 tests, 0 failures)
- No dialyzer warnings

## [0.3.0] - 2025-11-26

### Added
- **CrucibleIR Integration:**
  - Added `crucible_ir` ~> 0.1.1 dependency for intermediate representation support
  - Added support for `CrucibleIR.DatasetRef` in `load/1` function
  - DatasetRef provides unified dataset references across Crucible framework components
  - Seamless integration: `CrucibleDatasets.load(%DatasetRef{name: :mmlu_stem, ...})`
- **Enhanced Documentation:**
  - Updated module documentation with DatasetRef usage examples
  - Added comprehensive test suite for DatasetRef functionality (220+ test cases)
  - Updated README with DatasetRef integration examples

### Changed
- Version bump from 0.2.0 to 0.3.0
- `CrucibleDatasets.Loader.load/2` now accepts `DatasetRef` struct in addition to atoms and strings
- Enhanced type specifications to include `DatasetRef.t()` in function signatures

## [0.2.0] - 2025-11-25

### Added
- **New Evaluation Metrics:**
  - BLEU score metric for machine translation and text generation evaluation
  - ROUGE score metrics (ROUGE-1, ROUGE-2, ROUGE-L) for summarization evaluation
  - Support for multiple ROUGE variants with precision, recall, and F1 scores
- **Dataset Registry:**
  - Centralized dataset registry with comprehensive metadata
  - Dataset discovery by domain, task type, difficulty, and tags
  - Search functionality for finding datasets by keyword
  - Dataset statistics and summary generation
- **Result Persistence:**
  - ResultStore module for persistent storage of evaluation results
  - Organized storage by date with searchable index
  - Query interface with filters (model, dataset, accuracy, date range)
  - Result management (save, load, delete, clear)
- **Export Functionality:**
  - CSV export for spreadsheet applications and data analysis
  - JSON Lines export for streaming processing
  - Markdown report generation with customizable formatting
  - HTML report generation with styling and theming options
  - Flexible export options (sorting, grouping, detail levels)
- **Enhanced API:**
  - Convenience delegates in main CrucibleDatasets module
  - `list_available()` - List all available datasets
  - `get_metadata/1` - Get dataset metadata
  - `save_result/2` - Save evaluation results
  - `load_result/1` - Load saved results
  - `query_results/1` - Query results with filters
  - `export_csv/3`, `export_jsonl/2`, `export_markdown/2`, `export_html/2`

### Changed
- Version bump from 0.1.0 to 0.2.0
- Enhanced Evaluator to support new metrics (`:bleu`, `:rouge`, `:rouge1`, `:rouge2`, `:rougel`)
- Expanded documentation with new features and examples
- Updated README with version 0.2.0

### Documentation
- Comprehensive design document at `docs/20251125/enhancement_design.md`
- Detailed architecture and implementation plans
- Enhanced examples for new features
- API documentation for all new modules

## [0.1.0] - 2025-10-07

### Added
- Initial release
- Centralized dataset management for AI evaluation research
- Unified interface for benchmark datasets (MMLU, HumanEval, GSM8K)
- Automatic caching with version tracking for fast access
- Comprehensive evaluation metrics (exact match, F1 score, custom metrics)
- Dataset sampling strategies (random, stratified, k-fold cross-validation)
- Support for custom datasets from local JSONL files
- Reproducibility features with deterministic sampling and version control

### Documentation
- Comprehensive README with examples
- API documentation for dataset loading and evaluation
- Usage examples for common research workflows
- Integration guide for research infrastructure libraries