crucible_bench

# Changelog

All notable changes to this project will be documented in this file.

## [0.3.2] - 2025-12-25

### Added
- **Stage Multi-Group Support** - Stage now supports multiple data layouts:
  - Two-group comparison: `control` and `treatment` keys for t-test, Mann-Whitney
  - Multi-group comparison: `groups` key for ANOVA, Kruskal-Wallis
  - Paired comparison: `before` and `after` keys for paired t-test, Wilcoxon
- **Stage Metrics Merging** - Results automatically merged into `context.metrics`:
  - Summary stats: `bench_n`, `bench_mean`, `bench_sd`, `bench_median`
  - Test p-values: `bench_ttest_p_value`, `bench_anova_p_value`, etc.
- **Stage Type Specifications** - Added `@spec` declarations for public functions
- **Stage Behaviour Compliance** - Conditional `@behaviour Crucible.Stage` when available
- **Updated SVG** - Professional bell curve design for `crucible_bench.svg`
- **Current State Documentation** - Added `docs/20251225/current_state.md` with complete module reference
- **Gap Analysis** - Added `docs/20251225/gaps.md` with improvement opportunities
- **Implementation Guide** - Added `docs/20251225/implementation_prompt.md` for Stage enhancements

### Fixed
- **Credo Strict Compliance** - All 18 issues resolved:
  - Number formatting (`20700` → `20_700`)
  - Alphabetically sorted aliases across 7 files
  - `Enum.map_join/3` instead of `Enum.map/2 |> Enum.join/2`
  - Reduced function arity using map parameters
  - Extracted helpers to reduce cyclomatic complexity and nesting depth
  - Replaced `length/1 == 0` with `Enum.empty?/1`

### Changed
- **Dependencies Updated**:
  - `crucible_ir` upgraded from `~> 0.1.1` to `~> 0.2.0`
  - `eval_ex` upgraded from `~> 0.1.2` to `~> 0.1.4`
  - Added `credo ~> 1.7` as dev/test dependency

### Documentation
- README updated with Advanced Stage Configuration section
- README updated with Metrics Merging section

### Testing
- New test suite for two-group comparisons
- New test suite for multi-group comparisons (ANOVA, Kruskal-Wallis)
- New test suite for paired comparisons (paired t-test, Wilcoxon)
- New test suite for metrics merging
- New test suite for behaviour compliance

## [0.3.1] - 2025-12-24

### Added
- **EvalLog Schema:** Inspect-ai compatible evaluation log structs
- **EvalEx Adapter:** Convert EvalEx.Result into CrucibleBench.EvalLog
- **Metric Extraction:** Helpers mirroring inspect-ai analysis extractors

### Documentation
- README updated with EvalLog adapter usage
- Inspect-ai parity requirements updated with current status

## [0.3.0] - 2025-11-26

### Added
- **CrucibleIR Integration** - Added dependency on `crucible_ir ~> 0.1.1` for shared IR structures
- **CrucibleBench.Stage** - New pipeline stage module for integration with crucible_framework
  - Implements stage behaviour for use in pipeline orchestration
  - Accepts `CrucibleIR.Reliability.Stats` configuration from experiment context
  - Extracts and analyzes data from pipeline context (`:outputs` or `:metrics` keys)
  - Returns comprehensive statistical analysis in `:bench` context key
  - Supports test selection, confidence intervals, and bootstrap analysis
  - Provides `describe/1` function for stage introspection
- **IR Config Support** - Main `CrucibleBench.compare/3` function now accepts `CrucibleIR.Reliability.Stats` struct
  - Automatically converts IR configuration to internal options format
  - Maintains backwards compatibility with keyword list options
  - Maps IR test types (`:ttest`, `:bootstrap`, etc.) to CrucibleBench implementations

### Changed
- Version bumped to 0.3.0 (MINOR version due to new functionality)
- Enhanced `CrucibleBench` module with multi-clause function definitions for config handling

### Documentation
- Complete API documentation for `CrucibleBench.Stage` module
- Updated README with Stage usage examples
- Enhanced main module documentation with IR config examples

### Testing
- New comprehensive test suite for `CrucibleBench.Stage` (18 tests)
- Tests cover context processing, error handling, config conversion, and integration
- Property-based validation of IR config acceptance

## [0.2.1] - 2025-11-25

### Added
- **Multiple Comparison Corrections** - Controls Type I error rates when conducting multiple tests
  - Bonferroni correction (most conservative, controls FWER)
  - Holm step-down method (less conservative than Bonferroni, still controls FWER)
  - Benjamini-Hochberg FDR correction (controls false discovery rate, more powerful)
  - New module: `CrucibleBench.Stats.MultipleComparisons`
  - Integration with hyperparameter sweep experiments (automatic p-value adjustment)
  - Detailed correction results with original and adjusted p-values

- **Formal Normality Tests** - Statistical tests for distribution assumptions
  - Shapiro-Wilk test (most powerful omnibus test, n = 3 to 5000)
  - Comprehensive normality assessment combining multiple approaches
  - Quick normality check using skewness/kurtosis thresholds
  - New module: `CrucibleBench.Stats.NormalityTests`

- **Variance Equality Tests** - Validates homogeneity of variance assumptions
  - Levene's test (robust to non-normality, uses median-based deviations)
  - F-test for two groups (parametric, sensitive to normality)
  - Quick variance check using variance ratios
  - New module: `CrucibleBench.Stats.VarianceTests`

### Changed
- Hyperparameter sweep experiments now apply multiple comparison corrections by default (Holm method)
- Export module updated to display adjusted p-values in pairwise comparison tables
- Experiment results include correction method information

### Documentation
- Comprehensive enhancement design document in `docs/20251125/enhancement_design.md`
- Complete API documentation for all new modules
- Examples demonstrating each new feature
- Best practices for multiple comparison handling

### Testing
- 3 new comprehensive test suites (multiple_comparisons_test, normality_tests_test, variance_tests_test)
- Property-based testing for monotonicity and boundary conditions
- Integration tests demonstrating end-to-end functionality

## [0.2.0] - 2025-11-24

### Added
- Complete statistical testing framework with parametric (Student's, Welch's, paired t-tests, one-way ANOVA) and non-parametric (Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis) coverage using accurate distribution functions
- Expanded effect size suite including Cohen's d, Hedges' g, Glass's delta, paired Cohen's d, eta-squared/omega-squared, and rank-biserial correlation with interpretation guidance
- Confidence intervals (analytical and bootstrap) and power analysis (a priori and post-hoc for t-tests and ANOVA) with actionable recommendations
- High-level analysis helpers for automatic test selection plus experiment DSL for A/B tests, ablation studies, and hyperparameter sweeps
- Publication-ready exports to Markdown, LaTeX, and HTML with standardized result metadata

## [0.1.0] - 2025-10-07

### Added
- Initial release
- Comprehensive statistical testing framework for AI/ML research
- Parametric tests (t-tests, ANOVA) and non-parametric tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis)
- Effect size measures (Cohen's d, Hedges' g, Glass's delta, eta-squared, omega-squared)
- Power analysis with a priori and post-hoc calculations
- Confidence intervals using bootstrap and analytical methods
- High-level experiment DSL for A/B tests, ablation studies, and hyperparameter sweeps
- Publication-ready export formats (Markdown, LaTeX, HTML)

### Documentation
- Comprehensive README with examples
- API documentation for all statistical tests
- Usage examples for common AI research scenarios
- Best practices guide for statistical rigor in AI research