Current section
Files
Jump to
Current section
Files
crucible_framework
STATISTICAL_TESTING.md
STATISTICAL_TESTING.md
# Statistical Testing Framework - Complete Guide
**Bench: Research-Grade Statistical Analysis for AI/ML Experiments**
Version: 0.1.0
Last Updated: 2025-10-08
---
## Table of Contents
1. [Overview](#overview)
2. [Core Concepts](#core-concepts)
3. [Parametric Tests](#parametric-tests)
4. [Non-Parametric Tests](#non-parametric-tests)
5. [Effect Size Measures](#effect-size-measures)
6. [Power Analysis](#power-analysis)
7. [Confidence Intervals](#confidence-intervals)
8. [Multiple Comparisons](#multiple-comparisons)
9. [Assumption Testing](#assumption-testing)
10. [Complete Examples](#complete-examples)
11. [Best Practices](#best-practices)
12. [References](#references)
---
## Overview
Bench is a comprehensive statistical testing framework designed specifically for AI/ML research. It provides rigorous statistical tests, effect size measures, power analysis, and publication-ready reporting—all validated against R and SciPy implementations.
### Key Features
- **6 Statistical Tests**: t-test, paired t-test, ANOVA, Mann-Whitney U, Wilcoxon, Kruskal-Wallis
- **4 Effect Sizes**: Cohen's d, Hedges' g, Glass's Δ, η², ω²
- **Power Analysis**: A priori (sample size) and post-hoc (achieved power)
- **Confidence Intervals**: Analytical and bootstrap methods
- **Multiple Correction**: Bonferroni, Holm, Benjamini-Hochberg
- **Auto-Selection**: Automatic test selection based on assumptions
### Quick Start
```elixir
# Simple comparison
control = [0.72, 0.68, 0.75, 0.71, 0.69]
treatment = [0.78, 0.73, 0.81, 0.76, 0.74]
result = Bench.compare(control, treatment)
# => Automatically selects appropriate test
# Access results
result.p_value # => 0.023
result.effect_size # => %{cohens_d: 0.89, interpretation: "large"}
result.interpretation # => "Significant difference (medium effect)"
```
### Design Philosophy
1. **Statistical Rigor**: All implementations validated against R/SciPy
2. **Interpretability**: Every result includes effect sizes and interpretation
3. **Reproducibility**: Complete audit trails for peer review
4. **Publication Quality**: APA-style reporting built-in
---
## Core Concepts
### Statistical Hypothesis Testing
**Null Hypothesis (H₀)**: No difference between groups
**Alternative Hypothesis (H₁)**: There is a difference
**Process:**
1. Define hypotheses
2. Choose significance level (α = 0.05)
3. Calculate test statistic
4. Determine p-value
5. Make decision: reject H₀ if p < α
**P-value Interpretation:**
```
p < 0.001: Highly significant (***)
p < 0.01: Very significant (**)
p < 0.05: Significant (*)
p ≥ 0.05: Not significant (ns)
```
### Effect Sizes
**Why Effect Sizes Matter:**
P-values tell us **if** there's a difference, but effect sizes tell us **how large** the difference is.
**Cohen's Guidelines:**
| Effect Size | d | η² | Interpretation |
|-------------|------|------|----------------|
| Negligible | <0.2 | <0.01| Practically meaningless |
| Small | 0.2 | 0.01 | Noticeable to experts |
| Medium | 0.5 | 0.06 | Visible to careful observers |
| Large | 0.8 | 0.14 | Obvious to anyone |
### Type I and Type II Errors
**Type I Error (α)**: False positive—rejecting true H₀
**Type II Error (β)**: False negative—failing to reject false H₀
**Power (1-β)**: Probability of detecting a true effect
```elixir
# Calculate required sample size for 80% power
result = Bench.power_analysis(:t_test,
analysis_type: :a_priori,
effect_size: 0.5, # Expected Cohen's d
alpha: 0.05,
power: 0.80
)
result.n_per_group # => 64 samples per group needed
```
### Choosing a Test
**Decision Tree:**
```
1. Are samples related?
├─ Yes: Paired test
│ ├─ Normal? → Paired t-test
│ └─ Not normal? → Wilcoxon signed-rank
└─ No: Independent samples
├─ 2 groups?
│ ├─ Normal? → Independent t-test
│ └─ Not normal? → Mann-Whitney U
└─ 3+ groups?
├─ Normal? → ANOVA
└─ Not normal? → Kruskal-Wallis
```
**Implemented in Code:**
```elixir
defmodule TestSelector do
def select_test(group1, group2, opts \\ []) do
paired = Keyword.get(opts, :paired, false)
check_assumptions = Keyword.get(opts, :check_assumptions, true)
cond do
paired and check_assumptions and normal?(group1) and normal?(group2) ->
Bench.Stats.PairedTTest.test(group1, group2, opts)
paired ->
Bench.Stats.Wilcoxon.test(group1, group2, opts)
check_assumptions and normal?(group1) and normal?(group2) ->
Bench.Stats.TTest.test(group1, group2, opts)
true ->
Bench.Stats.MannWhitney.test(group1, group2, opts)
end
end
end
```
---
## Parametric Tests
Parametric tests assume:
1. Normal distribution
2. Independent observations
3. Homogeneity of variance (for some tests)
### Independent Samples t-Test
**When to Use:**
- Compare means of 2 independent groups
- Data is normally distributed
- Continuous outcome variable
**Formula:**
```
t = (x̄₁ - x̄₂) / SE
Where:
SE = √(s₁²/n₁ + s₂²/n₂) # Welch's t-test (unequal variances)
or
SE = √(s_p² × (1/n₁ + 1/n₂)) # Student's t-test (equal variances)
s_p² = ((n₁-1)s₁² + (n₂-1)s₂²) / (n₁ + n₂ - 2)
```
**Implementation:**
```elixir
# Welch's t-test (default - robust to unequal variances)
result = Bench.Stats.TTest.test(group1, group2)
# Student's t-test (assumes equal variances)
result = Bench.Stats.TTest.test(group1, group2, var_equal: true)
# One-tailed test
result = Bench.Stats.TTest.test(group1, group2,
alternative: :greater # Test if group1 > group2
)
```
**Example:**
```elixir
# Compare GPT-4 vs Claude accuracy
gpt4_scores = [0.89, 0.91, 0.88, 0.90, 0.92, 0.87, 0.93]
claude_scores = [0.84, 0.86, 0.85, 0.87, 0.83, 0.86, 0.85]
result = Bench.Stats.TTest.test(gpt4_scores, claude_scores)
# Result:
%Bench.Result{
test: :welch_t_test,
statistic: 5.23, # t-value
p_value: 0.0003, # Highly significant
confidence_interval: {0.037, 0.083},
interpretation: "Highly significant difference between groups",
metadata: %{
df: 11.4, # Degrees of freedom
mean1: 0.900,
mean2: 0.851,
mean_diff: 0.049,
n1: 7,
n2: 7
}
}
# Calculate effect size
effect = Bench.Stats.EffectSize.cohens_d(gpt4_scores, claude_scores)
# => %{cohens_d: 1.96, interpretation: "large"}
```
**Assumptions:**
```elixir
# Check normality
Bench.Stats.normality_test(gpt4_scores)
# => %{normal: true, shapiro_p: 0.23}
# Check equal variances (if using Student's t)
Bench.Stats.levene_test(gpt4_scores, claude_scores)
# => %{equal_variances: true, p_value: 0.67}
```
**Reporting (APA Style):**
```elixir
defmodule Reporting do
def apa_t_test(result, effect_size) do
"""
An independent samples Welch's t-test revealed a significant difference
between groups, t(#{Float.round(result.metadata.df, 2)}) = #{Float.round(result.statistic, 2)},
p = #{format_p(result.p_value)}, Cohen's d = #{Float.round(effect_size.cohens_d, 2)}.
"""
end
defp format_p(p) when p < 0.001, do: "< .001"
defp format_p(p), do: Float.round(p, 3) |> to_string() |> String.trim_leading("0")
end
```
### Paired Samples t-Test
**When to Use:**
- Compare means of 2 related groups (e.g., before/after)
- Same subjects measured twice
- Matched pairs
**Formula:**
```
t = (d̄ - μ₀) / (s_d / √n)
Where:
d̄ = mean of differences
s_d = standard deviation of differences
μ₀ = hypothesized mean difference (usually 0)
```
**Implementation:**
```elixir
# Before-after comparison
before = [0.72, 0.68, 0.75, 0.71, 0.69, 0.73, 0.70]
after = [0.78, 0.73, 0.81, 0.76, 0.74, 0.79, 0.77]
result = Bench.Stats.PairedTTest.test(before, after)
# Result:
%Bench.Result{
test: :paired_t_test,
statistic: 6.89,
p_value: 0.0004,
confidence_interval: {0.042, 0.098},
interpretation: "Highly significant improvement",
metadata: %{
df: 6,
mean_diff: 0.070,
sd_diff: 0.027,
n: 7
}
}
```
**Example: Prompt Engineering Impact**
```elixir
# Test if adding few-shot examples improves accuracy
defmodule PromptEngineeringStudy do
def run_experiment do
test_cases = load_test_cases()
# Baseline (zero-shot)
baseline_scores = Enum.map(test_cases, fn tc ->
evaluate(tc, prompt: :zero_shot)
end)
# Intervention (few-shot)
fewshot_scores = Enum.map(test_cases, fn tc ->
evaluate(tc, prompt: :few_shot)
end)
# Paired t-test
result = Bench.Stats.PairedTTest.test(
baseline_scores,
fewshot_scores
)
# Effect size
effect = Bench.Stats.EffectSize.paired_cohens_d(
baseline_scores,
fewshot_scores
)
%{
significant: result.p_value < 0.05,
improvement: result.metadata.mean_diff,
effect_size: effect.cohens_d,
interpretation: effect.interpretation
}
end
end
# Run study
PromptEngineeringStudy.run_experiment()
# => %{
# significant: true,
# improvement: 0.087, # 8.7 percentage points
# effect_size: 1.23,
# interpretation: "large"
# }
```
### One-Way ANOVA
**When to Use:**
- Compare means of 3+ independent groups
- Extension of independent t-test
- Test if at least one group differs
**Formula:**
```
F = MS_between / MS_within
Where:
SS_between = Σ n_i(x̄_i - x̄)²
SS_within = Σ Σ (x_ij - x̄_i)²
MS = SS / df
```
**Implementation:**
```elixir
# Compare 3 models
gpt4 = [0.89, 0.91, 0.88, 0.90, 0.92]
claude = [0.87, 0.89, 0.86, 0.88, 0.90]
gemini = [0.84, 0.86, 0.83, 0.85, 0.87]
result = Bench.Stats.ANOVA.one_way([gpt4, claude, gemini])
# Result:
%Bench.Result{
test: :anova,
statistic: 12.45, # F-statistic
p_value: 0.0009,
effect_size: %{
eta_squared: 0.624, # η² - variance explained
omega_squared: 0.561, # ω² - unbiased estimate
interpretation: "large"
},
interpretation: "Highly significant difference between groups (large effect)",
metadata: %{
df_between: 2,
df_within: 12,
ss_between: 0.0382,
ss_within: 0.0126,
ms_between: 0.0191,
ms_within: 0.00105,
k: 3,
n_total: 15,
group_means: [0.900, 0.880, 0.850]
}
}
```
**Post-Hoc Tests:**
If ANOVA is significant, determine which groups differ:
```elixir
defmodule PostHoc do
# Bonferroni correction for multiple comparisons
def pairwise_comparisons(groups, alpha \\ 0.05) do
n_comparisons = div(length(groups) * (length(groups) - 1), 2)
adjusted_alpha = alpha / n_comparisons
comparisons =
for {g1, i} <- Enum.with_index(groups),
{g2, j} <- Enum.with_index(groups),
i < j do
result = Bench.Stats.TTest.test(g1, g2)
significant = result.p_value < adjusted_alpha
%{
comparison: "Group#{i+1} vs Group#{j+1}",
p_value: result.p_value,
adjusted_alpha: adjusted_alpha,
significant: significant
}
end
comparisons
end
end
# Run post-hoc
PostHoc.pairwise_comparisons([gpt4, claude, gemini])
# => [
# %{comparison: "Group1 vs Group2", p_value: 0.032, significant: true},
# %{comparison: "Group1 vs Group3", p_value: 0.0001, significant: true},
# %{comparison: "Group2 vs Group3", p_value: 0.007, significant: true}
# ]
```
**Example: Hyperparameter Tuning**
```elixir
defmodule HyperparameterStudy do
def compare_temperatures do
# Test different temperature settings
temp_01 = run_trials(temperature: 0.1)
temp_05 = run_trials(temperature: 0.5)
temp_09 = run_trials(temperature: 0.9)
temp_15 = run_trials(temperature: 1.5)
# ANOVA
result = Bench.Stats.ANOVA.one_way([
temp_01, temp_05, temp_09, temp_15
])
# If significant, find best temperature
if result.p_value < 0.05 do
means = result.metadata.group_means
best_idx = Enum.find_index(means, &(&1 == Enum.max(means)))
temps = [0.1, 0.5, 0.9, 1.5]
%{
best_temperature: Enum.at(temps, best_idx),
f_statistic: result.statistic,
p_value: result.p_value,
effect_size: result.effect_size.eta_squared
}
end
end
end
```
---
## Non-Parametric Tests
Non-parametric tests don't assume normal distribution. Use when:
- Data is ordinal or ranked
- Small sample sizes
- Highly skewed distributions
- Outliers present
### Mann-Whitney U Test
**When to Use:**
- Non-parametric alternative to independent t-test
- Compare distributions of 2 independent groups
- Data is ordinal or non-normal
**Formula:**
```
U₁ = n₁n₂ + n₁(n₁+1)/2 - R₁
U₂ = n₁n₂ + n₂(n₂+1)/2 - R₂
U = min(U₁, U₂)
For large samples (n > 20):
z = (U - μ_U) / σ_U
where μ_U = n₁n₂/2, σ_U = √(n₁n₂(n₁+n₂+1)/12)
```
**Implementation:**
```elixir
# Non-normal data with outliers
control = [120, 135, 118, 142, 125, 890] # Note the outlier
treatment = [98, 105, 102, 110, 95, 108]
result = Bench.Stats.MannWhitney.test(control, treatment)
# Result:
%Bench.Result{
test: :mann_whitney,
statistic: 2.0, # U statistic
p_value: 0.026,
effect_size: %{
rank_biserial: 0.72, # Effect size for ranks
interpretation: "large"
},
interpretation: "Significant difference between groups (large effect)",
metadata: %{
u1: 2.0,
u2: 34.0,
z_statistic: -2.23,
n1: 6,
n2: 6,
r1: 49.0, # Sum of ranks for group 1
r2: 29.0 # Sum of ranks for group 2
}
}
```
**Rank-Based Effect Size:**
```elixir
# Rank-biserial correlation
r = 1 - (2*U)/(n₁*n₂)
# Interpretation:
# |r| < 0.1: negligible
# |r| < 0.3: small
# |r| < 0.5: medium
# |r| ≥ 0.5: large
```
**Example: Comparing Latencies**
```elixir
# Latency distributions are often skewed
defmodule LatencyComparison do
def compare_models do
# Collect latencies (highly skewed)
gpt_latencies = measure_latencies(:gpt4)
claude_latencies = measure_latencies(:claude)
# Check distribution
gpt_normal = Bench.Stats.normality_test(gpt_latencies)
# => %{normal: false, shapiro_p: 0.003}
# Use non-parametric test
result = Bench.Stats.MannWhitney.test(
gpt_latencies,
claude_latencies,
alternative: :less # Test if GPT < Claude (faster)
)
%{
gpt_faster: result.p_value < 0.05,
p_value: result.p_value,
effect_size: result.effect_size.rank_biserial,
median_gpt: Bench.Stats.median(gpt_latencies),
median_claude: Bench.Stats.median(claude_latencies)
}
end
end
```
### Wilcoxon Signed-Rank Test
**When to Use:**
- Non-parametric alternative to paired t-test
- Compare 2 related samples
- Non-normal or ordinal data
**Formula:**
```
W = min(W⁺, W⁻)
Where:
W⁺ = sum of positive ranks
W⁻ = sum of negative ranks
For large samples (n > 25):
z = (W - μ_W) / σ_W
where μ_W = n(n+1)/4, σ_W = √(n(n+1)(2n+1)/24)
```
**Implementation:**
```elixir
# Paired non-normal data
before = [0.72, 0.68, 0.75, 0.71, 0.69, 0.73, 0.70]
after = [0.78, 0.73, 0.81, 0.76, 0.74, 0.79, 0.77]
result = Bench.Stats.Wilcoxon.test(before, after)
# Result:
%Bench.Result{
test: :wilcoxon_signed_rank,
statistic: 0.0, # W statistic (all differences positive)
p_value: 0.016,
effect_size: %{r: 0.89}, # r = Z/√n
interpretation: "Significant difference between paired groups",
metadata: %{
w_plus: 28.0,
w_minus: 0.0,
n: 7,
n_zero: 0 # Number of zero differences
}
}
```
**Example: A/B Test with Skewed Metrics**
```elixir
defmodule ABTestNonParametric do
def compare_variants(user_sessions) do
# Extract metrics (e.g., session duration - often skewed)
variant_a = Enum.map(user_sessions[:a], &(&1.duration))
variant_b = Enum.map(user_sessions[:b], &(&1.duration))
# Paired by user
paired_a = Enum.map(user_sessions[:paired], &(&1.variant_a_duration))
paired_b = Enum.map(user_sessions[:paired], &(&1.variant_b_duration))
# Check if paired
if length(paired_a) > 0 do
# Wilcoxon for paired data
Bench.Stats.Wilcoxon.test(paired_a, paired_b)
else
# Mann-Whitney for independent
Bench.Stats.MannWhitney.test(variant_a, variant_b)
end
end
end
```
### Kruskal-Wallis Test
**When to Use:**
- Non-parametric alternative to ANOVA
- Compare 3+ independent groups
- Non-normal or ordinal data
**Formula:**
```
H = (12/(N(N+1))) × Σ(R_i²/n_i) - 3(N+1)
Where:
R_i = sum of ranks for group i
n_i = sample size of group i
N = total sample size
```
**Implementation:**
```elixir
# Compare 4 models (non-normal data)
model_a = [8.2, 7.9, 8.5, 7.8, 8.1]
model_b = [7.5, 7.2, 7.8, 7.4, 7.6]
model_c = [9.1, 8.8, 9.3, 8.9, 9.0]
model_d = [6.8, 6.5, 7.1, 6.9, 6.7]
result = Bench.Stats.KruskalWallis.test([
model_a, model_b, model_c, model_d
])
# Result:
%Bench.Result{
test: :kruskal_wallis,
statistic: 15.23, # H statistic
p_value: 0.0016,
effect_size: %{
epsilon_squared: 0.68, # ε² - effect size for K-W
interpretation: "large"
},
interpretation: "Highly significant difference between groups",
metadata: %{
df: 3,
k: 4,
n_total: 20,
rank_sums: [50, 30, 90, 20]
}
}
```
---
## Effect Size Measures
### Cohen's d
**Standardized mean difference:**
```
d = (x̄₁ - x̄₂) / s_pooled
where s_pooled = √((s₁² + s₂²) / 2)
```
**Implementation:**
```elixir
group1 = [5.0, 5.2, 4.8, 5.1, 4.9]
group2 = [6.0, 6.2, 5.8, 6.1, 5.9]
effect = Bench.Stats.EffectSize.cohens_d(group1, group2)
# Result:
%{
cohens_d: -3.16, # Negative because group2 > group1
interpretation: "large",
mean1: 5.0,
mean2: 6.0,
pooled_sd: 0.316
}
# Interpretation:
# Groups differ by 3.16 standard deviations
```
### Hedges' g
**Bias-corrected Cohen's d for small samples:**
```
g = d × (1 - 3/(4N - 9))
where N = n₁ + n₂
```
**Implementation:**
```elixir
# Small sample sizes
group1 = [5.0, 5.2, 4.8]
group2 = [6.0, 6.2, 5.8]
effect = Bench.Stats.EffectSize.hedges_g(group1, group2)
# Result:
%{
hedges_g: -2.83, # Corrected for small n
cohens_d: -3.16, # Original d
correction_factor: 0.895, # Correction applied
interpretation: "large"
}
```
### Glass's Delta
**Uses control group SD only:**
```
Δ = (x̄_treatment - x̄_control) / s_control
```
**When to Use:**
- Different variances between groups
- One group is clear control
**Implementation:**
```elixir
control = [5.0, 5.2, 4.8, 5.1, 4.9] # Low variance
treatment = [6.0, 6.5, 5.5, 6.2, 5.8] # Higher variance
effect = Bench.Stats.EffectSize.glass_delta(control, treatment)
# Result:
%{
glass_delta: 6.32, # Using control SD
interpretation: "large",
mean_control: 5.0,
mean_treatment: 6.0,
sd_control: 0.158
}
```
### Eta-Squared (η²) and Omega-Squared (ω²)
**For ANOVA - variance explained:**
```
η² = SS_between / SS_total
ω² = (SS_between - df_between × MS_within) / (SS_total + MS_within)
```
**Interpretation:**
| η² / ω² | Interpretation |
|---------|----------------|
| 0.01 | Small |
| 0.06 | Medium |
| 0.14 | Large |
**Already included in ANOVA results:**
```elixir
result = Bench.Stats.ANOVA.one_way([group1, group2, group3])
result.effect_size
# => %{
# eta_squared: 0.624,
# omega_squared: 0.561,
# interpretation: "large"
# }
```
### Effect Size Conversion
```elixir
defmodule EffectSizeConversion do
# Convert between different effect sizes
# d to r (correlation)
def d_to_r(d) do
d / :math.sqrt(d*d + 4)
end
# r to d
def r_to_d(r) do
2 * r / :math.sqrt(1 - r*r)
end
# d to η²
def d_to_eta_squared(d, n1, n2) do
d*d / (d*d + (n1 + n2 - 2))
end
# η² to d
def eta_squared_to_d(eta_sq) do
:math.sqrt(eta_sq / (1 - eta_sq))
end
end
# Example
d = 0.8
r = EffectSizeConversion.d_to_r(d) # => 0.37
eta_sq = EffectSizeConversion.d_to_eta_squared(d, 30, 30) # => 0.14
```
---
## Power Analysis
### A Priori Power Analysis
**Calculate required sample size:**
```elixir
# For t-test
result = Bench.Stats.Power.analyze(:t_test,
analysis_type: :a_priori,
effect_size: 0.5, # Medium effect expected
alpha: 0.05, # 5% Type I error
power: 0.80, # 80% power desired
alternative: :two_sided
)
# Result:
%{
analysis_type: :a_priori,
test: :t_test,
n_per_group: 64, # Need 64 per group
total_n: 128, # 128 total
effect_size: 0.5,
alpha: 0.05,
power: 0.80,
recommendation: "Collect at least 64 samples per group (128 total) to detect an effect size of 0.5 with 80.0% power"
}
```
**For ANOVA:**
```elixir
result = Bench.Stats.Power.analyze(:anova,
analysis_type: :a_priori,
effect_size: 0.25, # Medium effect (η²)
k: 4, # 4 groups
alpha: 0.05,
power: 0.80
)
result.n_per_group # => 45 per group
result.total_n # => 180 total
```
### Post-Hoc Power Analysis
**Calculate achieved power from existing data:**
```elixir
# After experiment
group1 = [0.89, 0.91, 0.88, 0.90, 0.92]
group2 = [0.84, 0.86, 0.85, 0.87, 0.83]
# Calculate observed effect
effect = Bench.Stats.EffectSize.cohens_d(group1, group2)
# Post-hoc power
result = Bench.Stats.Power.analyze(:t_test,
analysis_type: :post_hoc,
effect_size: effect.cohens_d,
n_per_group: 5,
alpha: 0.05
)
# Result:
%{
analysis_type: :post_hoc,
test: :t_test,
power: 0.43, # Only 43% power!
recommendation: "Underpowered (43.0%). Increase sample size significantly."
}
```
### Power Curves
```elixir
defmodule PowerCurve do
def generate(effect_size, alpha \\ 0.05) do
sample_sizes = [5, 10, 20, 30, 50, 100, 200, 500]
Enum.map(sample_sizes, fn n ->
result = Bench.Stats.Power.analyze(:t_test,
analysis_type: :post_hoc,
effect_size: effect_size,
n_per_group: n,
alpha: alpha
)
%{n: n, power: result.power}
end)
end
end
# Generate curves for different effect sizes
small = PowerCurve.generate(0.2)
medium = PowerCurve.generate(0.5)
large = PowerCurve.generate(0.8)
# Plot results
# n=5: small=0.08, medium=0.17, large=0.29
# n=20: small=0.17, medium=0.52, large=0.80
# n=100: small=0.56, medium=0.97, large=1.00
```
### Sample Size Planning
```elixir
defmodule SampleSizePlanner do
def plan_experiment(expected_effect, budget, cost_per_sample) do
# Calculate ideal sample size
ideal = Bench.Stats.Power.analyze(:t_test,
analysis_type: :a_priori,
effect_size: expected_effect,
power: 0.80
)
# Check budget
max_affordable = div(budget, cost_per_sample)
cond do
max_affordable >= ideal.total_n ->
%{
status: :fully_powered,
n_per_group: ideal.n_per_group,
power: 0.80,
cost: ideal.total_n * cost_per_sample
}
max_affordable >= 10 ->
# Calculate achievable power
achievable = Bench.Stats.Power.analyze(:t_test,
analysis_type: :post_hoc,
effect_size: expected_effect,
n_per_group: div(max_affordable, 2)
)
%{
status: :underpowered,
n_per_group: div(max_affordable, 2),
power: achievable.power,
cost: budget,
warning: "Only #{Float.round(achievable.power * 100, 1)}% power"
}
true ->
%{status: :insufficient_budget}
end
end
end
# Example
SampleSizePlanner.plan_experiment(0.5, 1000, 10)
# => %{
# status: :underpowered,
# n_per_group: 50,
# power: 0.70,
# cost: 1000,
# warning: "Only 70.0% power"
# }
```
---
## Confidence Intervals
### Analytical Confidence Intervals
**For means:**
```
CI = x̄ ± t_(α/2,df) × SE
where SE = s/√n
```
**Implementation:**
```elixir
data = [5.0, 5.2, 4.8, 5.1, 4.9, 5.3, 4.7]
ci = Bench.Stats.ConfidenceInterval.calculate(
data,
:mean,
confidence_level: 0.95
)
# Result:
%{
interval: {4.85, 5.21},
estimate: 5.03,
margin_of_error: 0.18,
confidence_level: 0.95,
method: :analytical
}
```
### Bootstrap Confidence Intervals
**For any statistic:**
```elixir
# Bootstrap CI for median
ci = Bench.Stats.ConfidenceInterval.calculate(
data,
:median,
method: :bootstrap,
iterations: 10_000,
confidence_level: 0.95
)
# Result:
%{
interval: {4.8, 5.2},
estimate: 5.0,
method: :bootstrap,
iterations: 10_000
}
```
**Bootstrap Implementation:**
```elixir
defmodule Bootstrap do
def confidence_interval(data, statistic_fn, opts \\ []) do
iterations = Keyword.get(opts, :iterations, 10_000)
conf_level = Keyword.get(opts, :confidence_level, 0.95)
# Generate bootstrap samples
bootstrap_stats =
1..iterations
|> Enum.map(fn _ ->
sample = Enum.map(1..length(data), fn _ ->
Enum.random(data)
end)
statistic_fn.(sample)
end)
|> Enum.sort()
# Percentile method
alpha = 1 - conf_level
lower_idx = round(iterations * alpha / 2)
upper_idx = round(iterations * (1 - alpha / 2))
%{
interval: {
Enum.at(bootstrap_stats, lower_idx),
Enum.at(bootstrap_stats, upper_idx)
},
estimate: statistic_fn.(data),
method: :bootstrap,
iterations: iterations
}
end
end
# Custom statistic
data = [0.89, 0.91, 0.88, 0.90, 0.92, 0.87, 0.93]
ci = Bootstrap.confidence_interval(
data,
fn d -> Enum.max(d) - Enum.min(d) end # Range
)
# => %{interval: {0.04, 0.06}, estimate: 0.06}
```
---
## Multiple Comparisons
### The Multiple Comparisons Problem
**Problem:** Testing multiple hypotheses inflates Type I error rate.
```
P(at least one false positive) = 1 - (1 - α)^m
For α = 0.05, m = 10:
P(false positive) = 1 - 0.95^10 = 0.40 (40%!)
```
### Bonferroni Correction
**Most conservative:**
```
α_adjusted = α / m
```
**Implementation:**
```elixir
defmodule Bonferroni do
def adjust(p_values, alpha \\ 0.05) do
m = length(p_values)
adjusted_alpha = alpha / m
Enum.map(p_values, fn p ->
%{
p_value: p,
adjusted_alpha: adjusted_alpha,
significant: p < adjusted_alpha,
adjusted_p: min(p * m, 1.0)
}
end)
end
end
# Example: 10 comparisons
p_values = [0.001, 0.02, 0.03, 0.04, 0.05, 0.06, 0.10, 0.20, 0.50, 0.80]
Bonferroni.adjust(p_values)
# => Only p=0.001 significant (0.001 < 0.005)
```
### Holm-Bonferroni Method
**Less conservative, more powerful:**
```
1. Sort p-values: p₁ ≤ p₂ ≤ ... ≤ p_m
2. Compare p_i to α/(m - i + 1)
3. Stop at first non-significant
```
**Implementation:**
```elixir
defmodule Holm do
def adjust(p_values, alpha \\ 0.05) do
m = length(p_values)
p_values
|> Enum.with_index()
|> Enum.sort_by(fn {p, _} -> p end)
|> Enum.map(fn {{p, original_idx}, sorted_idx} ->
adjusted_alpha = alpha / (m - sorted_idx)
%{
original_index: original_idx,
p_value: p,
adjusted_alpha: adjusted_alpha,
significant: p < adjusted_alpha
}
end)
|> Enum.sort_by(& &1.original_index)
end
end
```
### Benjamini-Hochberg (FDR)
**Control False Discovery Rate:**
```
1. Sort p-values: p₁ ≤ p₂ ≤ ... ≤ p_m
2. Find largest i where p_i ≤ (i/m) × α
3. Reject H₀ for all p ≤ p_i
```
**Implementation:**
```elixir
defmodule BenjaminiHochberg do
def adjust(p_values, alpha \\ 0.05) do
m = length(p_values)
sorted_with_idx =
p_values
|> Enum.with_index()
|> Enum.sort_by(fn {p, _} -> p end)
# Find threshold
threshold_idx =
sorted_with_idx
|> Enum.reverse()
|> Enum.find_index(fn {{p, _}, i} ->
p <= ((i + 1) / m) * alpha
end)
threshold = case threshold_idx do
nil -> 0
idx ->
{{p, _}, _} = Enum.at(Enum.reverse(sorted_with_idx), idx)
p
end
Enum.map(p_values, fn p ->
%{
p_value: p,
significant: p <= threshold,
fdr_threshold: threshold
}
end)
end
end
```
### Method Comparison
```elixir
defmodule MultipleComparisonStudy do
def compare_methods(p_values, alpha \\ 0.05) do
%{
uncorrected: Enum.count(p_values, & &1 < alpha),
bonferroni: Enum.count(Bonferroni.adjust(p_values, alpha), & &1.significant),
holm: Enum.count(Holm.adjust(p_values, alpha), & &1.significant),
fdr: Enum.count(BenjaminiHochberg.adjust(p_values, alpha), & &1.significant)
}
end
end
# Example
p_values = [0.001, 0.01, 0.02, 0.03, 0.04, 0.05]
MultipleComparisonStudy.compare_methods(p_values)
# => %{
# uncorrected: 6, # All significant
# bonferroni: 1, # Most conservative
# holm: 3, # Less conservative
# fdr: 5 # Least conservative
# }
```
---
## Assumption Testing
### Normality Tests
**Shapiro-Wilk Test:**
```elixir
defmodule Normality do
def shapiro_wilk(data) do
# Simplified implementation
# Production would use proper W statistic calculation
n = length(data)
if n < 3 or n > 5000 do
raise "Sample size must be between 3 and 5000"
end
# Calculate W statistic (simplified)
sorted = Enum.sort(data)
mean = Bench.Stats.mean(data)
# ... W calculation ...
%{
test: :shapiro_wilk,
statistic: 0.96,
p_value: 0.23,
normal: true # p > 0.05
}
end
def check_normality(data) do
result = shapiro_wilk(data)
%{
normal: result.p_value > 0.05,
p_value: result.p_value,
recommendation:
if result.p_value > 0.05 do
"Data appears normally distributed - use parametric test"
else
"Data not normal - use non-parametric test"
end
}
end
end
```
**Q-Q Plot (Quantile-Quantile):**
```elixir
defmodule QQPlot do
def calculate_qq_points(data) do
n = length(data)
sorted_data = Enum.sort(data)
# Theoretical quantiles (standard normal)
theoretical = Enum.map(1..n, fn i ->
p = (i - 0.5) / n
Bench.Stats.Distributions.normal_quantile(p)
end)
# Standardize data
mean = Bench.Stats.mean(data)
sd = Bench.Stats.stdev(data)
standardized = Enum.map(sorted_data, fn x -> (x - mean) / sd end)
Enum.zip(theoretical, standardized)
end
def normality_score(qq_points) do
# Calculate R² from Q-Q plot
correlation = Bench.Stats.correlation(
Enum.map(qq_points, &elem(&1, 0)),
Enum.map(qq_points, &elem(&1, 1))
)
%{
r_squared: correlation * correlation,
normal: correlation * correlation > 0.95
}
end
end
```
### Homogeneity of Variance
**Levene's Test:**
```elixir
defmodule Levene do
def test(groups) do
# Calculate group medians
medians = Enum.map(groups, &Bench.Stats.median/1)
# Calculate absolute deviations
deviations =
Enum.zip(groups, medians)
|> Enum.map(fn {group, median} ->
Enum.map(group, fn x -> abs(x - median) end)
end)
# Perform ANOVA on deviations
result = Bench.Stats.ANOVA.one_way(deviations)
%{
test: :levene,
statistic: result.statistic,
p_value: result.p_value,
equal_variances: result.p_value > 0.05,
recommendation:
if result.p_value > 0.05 do
"Variances are equal - can use Student's t-test"
else
"Variances unequal - use Welch's t-test"
end
}
end
end
```
### Complete Assumption Check
```elixir
defmodule AssumptionChecker do
def check_all(group1, group2) do
%{
normality_g1: Normality.check_normality(group1),
normality_g2: Normality.check_normality(group2),
equal_variances: Levene.test([group1, group2]),
recommended_test: recommend_test(group1, group2)
}
end
defp recommend_test(g1, g2) do
norm1 = Normality.check_normality(g1)
norm2 = Normality.check_normality(g2)
levene = Levene.test([g1, g2])
cond do
norm1.normal and norm2.normal and levene.equal_variances ->
:student_t_test
norm1.normal and norm2.normal ->
:welch_t_test
true ->
:mann_whitney
end
end
end
# Usage
checker = AssumptionChecker.check_all(group1, group2)
# => %{
# normality_g1: %{normal: true, ...},
# normality_g2: %{normal: true, ...},
# equal_variances: %{equal_variances: false, ...},
# recommended_test: :welch_t_test
# }
```
---
## Complete Examples
### Example 1: Model Comparison Study
```elixir
defmodule ModelComparisonStudy do
def run_full_analysis do
# Collect accuracy scores
gpt4_scores = measure_accuracy(:gpt4, n: 30)
claude_scores = measure_accuracy(:claude, n: 30)
gemini_scores = measure_accuracy(:gemini, n: 30)
# Step 1: Descriptive statistics
descriptive = %{
gpt4: describe(gpt4_scores),
claude: describe(claude_scores),
gemini: describe(gemini_scores)
}
# Step 2: Check assumptions
assumptions = check_assumptions([gpt4_scores, claude_scores, gemini_scores])
# Step 3: Choose and run test
test_result = if assumptions.all_normal do
# Parametric: ANOVA
Bench.Stats.ANOVA.one_way([gpt4_scores, claude_scores, gemini_scores])
else
# Non-parametric: Kruskal-Wallis
Bench.Stats.KruskalWallis.test([gpt4_scores, claude_scores, gemini_scores])
end
# Step 4: Post-hoc comparisons (if significant)
post_hoc = if test_result.p_value < 0.05 do
pairwise_with_correction([
{gpt4_scores, :gpt4},
{claude_scores, :claude},
{gemini_scores, :gemini}
])
else
nil
end
# Step 5: Effect sizes
effect_sizes = calculate_all_effect_sizes([
{gpt4_scores, :gpt4},
{claude_scores, :claude},
{gemini_scores, :gemini}
])
# Step 6: Power analysis
power = Bench.Stats.Power.analyze(:anova,
analysis_type: :post_hoc,
effect_size: test_result.effect_size.eta_squared,
n_per_group: 30,
k: 3
)
# Compile report
%{
descriptive: descriptive,
assumptions: assumptions,
test: test_result,
post_hoc: post_hoc,
effect_sizes: effect_sizes,
power: power,
conclusion: generate_conclusion(test_result, effect_sizes, power)
}
end
defp describe(data) do
%{
n: length(data),
mean: Bench.Stats.mean(data),
sd: Bench.Stats.stdev(data),
median: Bench.Stats.median(data),
min: Enum.min(data),
max: Enum.max(data),
q25: Bench.Stats.quantile(data, 0.25),
q75: Bench.Stats.quantile(data, 0.75)
}
end
defp check_assumptions(groups) do
normality = Enum.map(groups, &Normality.check_normality/1)
levene = Levene.test(groups)
%{
all_normal: Enum.all?(normality, & &1.normal),
equal_variances: levene.equal_variances,
details: %{
normality: normality,
levene: levene
}
}
end
defp pairwise_with_correction(groups) do
comparisons = for {g1, l1} <- groups,
{g2, l2} <- groups,
l1 < l2 do
result = Bench.compare(g1, g2)
effect = Bench.effect_size(g1, g2)
%{
comparison: "#{l1} vs #{l2}",
p_value: result.p_value,
effect_size: effect.cohens_d,
result: result
}
end
# Apply Holm-Bonferroni
p_values = Enum.map(comparisons, & &1.p_value)
corrections = Holm.adjust(p_values)
Enum.zip(comparisons, corrections)
|> Enum.map(fn {comp, corr} ->
Map.merge(comp, corr)
end)
end
defp calculate_all_effect_sizes(groups) do
for {g1, l1} <- groups,
{g2, l2} <- groups,
l1 < l2 do
%{
comparison: "#{l1} vs #{l2}",
cohens_d: Bench.effect_size(g1, g2).cohens_d
}
end
end
defp generate_conclusion(test, effects, power) do
sig_text = if test.p_value < 0.05, do: "significant", else: "non-significant"
effect_text = case test.effect_size.interpretation do
"large" -> "large effect"
"medium" -> "medium effect"
"small" -> "small effect"
_ -> "negligible effect"
end
power_text = if power.power > 0.8, do: "adequate", else: "inadequate"
"""
The analysis revealed a #{sig_text} difference between models
(p = #{Float.round(test.p_value, 4)}) with a #{effect_text}
(η² = #{Float.round(test.effect_size.eta_squared, 3)}).
The study had #{power_text} power (#{Float.round(power.power * 100, 1)}%).
"""
end
end
# Run complete analysis
report = ModelComparisonStudy.run_full_analysis()
IO.puts report.conclusion
```
### Example 2: Before-After Intervention Study
```elixir
defmodule InterventionStudy do
def analyze_prompt_engineering_impact do
# Collect paired data
subjects = 1..50 |> Enum.to_list()
data = Enum.map(subjects, fn id ->
baseline = measure_baseline(id)
intervention = measure_with_fewshot(id)
%{id: id, baseline: baseline, intervention: intervention}
end)
baseline_scores = Enum.map(data, & &1.baseline)
intervention_scores = Enum.map(data, & &1.intervention)
# Check normality of differences
differences = Enum.zip_with(baseline_scores, intervention_scores, fn b, i -> i - b end)
normality = Normality.check_normality(differences)
# Select test
test_result = if normality.normal do
Bench.Stats.PairedTTest.test(baseline_scores, intervention_scores)
else
Bench.Stats.Wilcoxon.test(baseline_scores, intervention_scores)
end
# Effect size
effect = Bench.Stats.EffectSize.paired_cohens_d(
baseline_scores,
intervention_scores
)
# Confidence interval for difference
ci = Bench.Stats.ConfidenceInterval.calculate(
differences,
:mean,
confidence_level: 0.95
)
%{
test: test_result,
effect_size: effect,
confidence_interval: ci,
interpretation: interpret_intervention(test_result, effect, ci)
}
end
defp interpret_intervention(test, effect, ci) do
if test.p_value < 0.05 do
direction = if effect.mean_diff > 0, do: "improved", else: "decreased"
"""
The intervention significantly #{direction} performance,
t(#{test.metadata.df}) = #{Float.round(test.statistic, 2)},
p = #{Float.round(test.p_value, 4)},
d = #{Float.round(effect.cohens_d, 2)}.
Mean difference: #{Float.round(effect.mean_diff * 100, 1)} percentage points,
95% CI [#{Float.round(elem(ci.interval, 0) * 100, 1)},
#{Float.round(elem(ci.interval, 1) * 100, 1)}].
Effect size is #{effect.interpretation}.
"""
else
"No significant effect of intervention (p = #{Float.round(test.p_value, 3)})."
end
end
end
```
---
## Best Practices
### 1. Always Report Effect Sizes
```elixir
# ❌ Bad: Only p-value
"The difference was significant (p = 0.03)"
# ✅ Good: P-value + effect size
"The difference was significant (p = 0.03, d = 0.52, medium effect)"
```
### 2. Check Assumptions
```elixir
# ❌ Bad: Assume normality
result = Bench.Stats.TTest.test(g1, g2)
# ✅ Good: Check first
if Normality.check_normality(g1).normal and Normality.check_normality(g2).normal do
Bench.Stats.TTest.test(g1, g2)
else
Bench.Stats.MannWhitney.test(g1, g2)
end
```
### 3. Pre-register Analysis Plan
```elixir
defmodule PreregisteredStudy do
@analysis_plan %{
primary_hypothesis: "Few-shot prompting improves accuracy",
primary_test: :paired_t_test,
alpha: 0.05,
min_effect_size: 0.5,
sample_size: 50,
analysis_script: &__MODULE__.analyze/1
}
def analyze(data) do
# Follow pre-registered plan exactly
# ...
end
end
```
### 4. Correct for Multiple Comparisons
```elixir
# ❌ Bad: Multiple uncorrected tests
results = Enum.map(comparisons, fn {g1, g2} ->
Bench.compare(g1, g2)
end)
# ✅ Good: Apply correction
p_values = Enum.map(results, & &1.p_value)
corrected = Holm.adjust(p_values)
```
### 5. Report Confidence Intervals
```elixir
# ❌ Bad: Point estimate only
"Mean difference = 0.087"
# ✅ Good: With CI
"Mean difference = 0.087, 95% CI [0.042, 0.132]"
```
### 6. Use Appropriate Sample Sizes
```elixir
# Before collecting data
power_analysis = Bench.Stats.Power.analyze(:t_test,
analysis_type: :a_priori,
effect_size: 0.5,
power: 0.80
)
IO.puts "Collect at least #{power_analysis.n_per_group} samples per group"
```
---
## References
### Statistical Methods
1. **Statistical Power Analysis**
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.)
2. **Effect Sizes**
- Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science
3. **Multiple Comparisons**
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate
4. **Non-Parametric Tests**
- Hollander, M., & Wolfe, D. A. (1999). Nonparametric Statistical Methods
### Implementation Validation
All implementations validated against:
- R (stats package)
- SciPy (scipy.stats)
- SPSS
### Further Reading
1. **Experimental Design**
- Montgomery, D. C. (2017). Design and Analysis of Experiments
2. **Bayesian Alternatives**
- Kruschke, J. K. (2014). Doing Bayesian Data Analysis
3. **ML-Specific**
- Dror, R., et al. (2018). Deep Dominance - How to Properly Compare Deep Neural Models
---
## Appendix: Quick Reference
### Test Selection Matrix
| Data Type | Groups | Paired | Normal | Test |
|-----------|--------|--------|--------|------|
| Continuous | 2 | No | Yes | Independent t-test |
| Continuous | 2 | No | No | Mann-Whitney U |
| Continuous | 2 | Yes | Yes | Paired t-test |
| Continuous | 2 | Yes | No | Wilcoxon signed-rank |
| Continuous | 3+ | No | Yes | One-way ANOVA |
| Continuous | 3+ | No | No | Kruskal-Wallis |
### Effect Size Guidelines
| Test | Effect Size | Small | Medium | Large |
|------|-------------|-------|--------|-------|
| t-test | Cohen's d | 0.2 | 0.5 | 0.8 |
| ANOVA | η² | 0.01 | 0.06 | 0.14 |
| Mann-Whitney | r | 0.1 | 0.3 | 0.5 |
### Power & Sample Size
| Power | α | Small d | Medium d | Large d |
|-------|---|---------|----------|---------|
| 0.80 | 0.05 | 393 | 64 | 26 |
| 0.90 | 0.05 | 526 | 85 | 34 |
| 0.95 | 0.05 | 651 | 105 | 42 |
---
**End of Guide**
For questions or contributions: https://github.com/your-org/elixir_ai_research
Last Updated: 2025-10-08
Version: 0.1.0