crucible_bench

# Changelog

All notable changes to this project will be documented in this file.

## [0.2.1] - 2025-11-25

### Added
- **Multiple Comparison Corrections** - Controls Type I error rates when conducting multiple tests
  - Bonferroni correction (most conservative, controls FWER)
  - Holm step-down method (less conservative than Bonferroni, still controls FWER)
  - Benjamini-Hochberg FDR correction (controls false discovery rate, more powerful)
  - New module: `CrucibleBench.Stats.MultipleComparisons`
  - Integration with hyperparameter sweep experiments (automatic p-value adjustment)
  - Detailed correction results with original and adjusted p-values

- **Formal Normality Tests** - Statistical tests for distribution assumptions
  - Shapiro-Wilk test (most powerful omnibus test, n = 3 to 5000)
  - Comprehensive normality assessment combining multiple approaches
  - Quick normality check using skewness/kurtosis thresholds
  - New module: `CrucibleBench.Stats.NormalityTests`

- **Variance Equality Tests** - Validates homogeneity of variance assumptions
  - Levene's test (robust to non-normality, uses median-based deviations)
  - F-test for two groups (parametric, sensitive to normality)
  - Quick variance check using variance ratios
  - New module: `CrucibleBench.Stats.VarianceTests`

### Changed
- Hyperparameter sweep experiments now apply multiple comparison corrections by default (Holm method)
- Export module updated to display adjusted p-values in pairwise comparison tables
- Experiment results include correction method information

### Documentation
- Comprehensive enhancement design document in `docs/20251125/enhancement_design.md`
- Complete API documentation for all new modules
- Examples demonstrating each new feature
- Best practices for multiple comparison handling

### Testing
- 3 new comprehensive test suites (multiple_comparisons_test, normality_tests_test, variance_tests_test)
- Property-based testing for monotonicity and boundary conditions
- Integration tests demonstrating end-to-end functionality

## [0.2.0] - 2025-11-24

### Added
- Complete statistical testing framework with parametric (Student's, Welch's, paired t-tests, one-way ANOVA) and non-parametric (Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis) coverage using accurate distribution functions
- Expanded effect size suite including Cohen's d, Hedges' g, Glass's delta, paired Cohen's d, eta-squared/omega-squared, and rank-biserial correlation with interpretation guidance
- Confidence intervals (analytical and bootstrap) and power analysis (a priori and post-hoc for t-tests and ANOVA) with actionable recommendations
- High-level analysis helpers for automatic test selection plus experiment DSL for A/B tests, ablation studies, and hyperparameter sweeps
- Publication-ready exports to Markdown, LaTeX, and HTML with standardized result metadata

## [0.1.0] - 2025-10-07

### Added
- Initial release
- Comprehensive statistical testing framework for AI/ML research
- Parametric tests (t-tests, ANOVA) and non-parametric tests (Mann-Whitney, Wilcoxon, Kruskal-Wallis)
- Effect size measures (Cohen's d, Hedges' g, Glass's delta, eta-squared, omega-squared)
- Power analysis with a priori and post-hoc calculations
- Confidence intervals using bootstrap and analytical methods
- High-level experiment DSL for A/B tests, ablation studies, and hyperparameter sweeps
- Publication-ready export formats (Markdown, LaTeX, HTML)

### Documentation
- Comprehensive README with examples
- API documentation for all statistical tests
- Usage examples for common AI research scenarios
- Best practices guide for statistical rigor in AI research