Current section
Files
Jump to
Current section
Files
lib/ex_fairness.ex
defmodule ExFairness do
@moduledoc """
ExFairness - Fairness and bias detection library for Elixir AI/ML systems.
ExFairness provides comprehensive fairness metrics, bias detection algorithms,
and mitigation techniques to ensure equitable predictions across different
demographic groups.
## Features
- **Fairness Metrics**: Demographic parity, equalized odds, equal opportunity, and more
- **Bias Detection**: Statistical testing, disparate impact analysis, intersectional bias
- **Mitigation**: Reweighting, resampling, threshold optimization, adversarial debiasing
- **Reporting**: Comprehensive fairness reports with interpretations
## Quick Start
# Compute demographic parity
predictions = Nx.tensor([1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])
sensitive = Nx.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
result = ExFairness.demographic_parity(predictions, sensitive)
# => %{disparity: 0.0, passes: true, ...}
## Metrics
- `demographic_parity/3` - Demographic parity (statistical parity)
- `equalized_odds/4` - Equalized odds (equal TPR and FPR)
- `equal_opportunity/4` - Equal opportunity (equal TPR)
- `predictive_parity/4` - Predictive parity (equal PPV)
- More metrics coming soon...
"""
alias ExFairness.Metrics.Calibration
alias ExFairness.Metrics.DemographicParity
alias ExFairness.Metrics.EqualizedOdds
alias ExFairness.Metrics.EqualOpportunity
alias ExFairness.Metrics.PredictiveParity
alias ExFairness.Report
# Support for CrucibleIR integration (conditionally compiled)
# The evaluate/5 function is only available when crucible_ir is loaded
@doc """
Computes demographic parity disparity between groups.
Demographic parity requires that the probability of a positive prediction
is equal across groups defined by the sensitive attribute.
## Parameters
* `predictions` - Binary predictions tensor (0 or 1)
* `sensitive_attr` - Binary sensitive attribute tensor (0 or 1)
* `opts` - Options (see `ExFairness.Metrics.DemographicParity.compute/3`)
## Returns
A map containing fairness metrics. See `ExFairness.Metrics.DemographicParity.compute/3`
for details.
## Examples
iex> predictions = Nx.tensor([1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])
iex> sensitive = Nx.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
iex> result = ExFairness.demographic_parity(predictions, sensitive)
iex> result.passes
true
"""
@spec demographic_parity(Nx.Tensor.t(), Nx.Tensor.t(), keyword()) ::
DemographicParity.result()
def demographic_parity(predictions, sensitive_attr, opts \\ []) do
DemographicParity.compute(predictions, sensitive_attr, opts)
end
@doc """
Computes equalized odds disparity between groups.
Equalized odds requires that both TPR and FPR are equal across groups.
## Parameters
* `predictions` - Binary predictions tensor (0 or 1)
* `labels` - Binary labels tensor (0 or 1)
* `sensitive_attr` - Binary sensitive attribute tensor (0 or 1)
* `opts` - Options (see `ExFairness.Metrics.EqualizedOdds.compute/4`)
## Returns
A map containing fairness metrics. See `ExFairness.Metrics.EqualizedOdds.compute/4`
for details.
## Examples
iex> predictions = Nx.tensor([1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1])
iex> labels = Nx.tensor([1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1])
iex> sensitive = Nx.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
iex> result = ExFairness.equalized_odds(predictions, labels, sensitive)
iex> result.passes
true
"""
@spec equalized_odds(Nx.Tensor.t(), Nx.Tensor.t(), Nx.Tensor.t(), keyword()) ::
EqualizedOdds.result()
def equalized_odds(predictions, labels, sensitive_attr, opts \\ []) do
EqualizedOdds.compute(predictions, labels, sensitive_attr, opts)
end
@doc """
Computes equal opportunity disparity between groups.
Equal opportunity requires that TPR is equal across groups.
## Parameters
* `predictions` - Binary predictions tensor (0 or 1)
* `labels` - Binary labels tensor (0 or 1)
* `sensitive_attr` - Binary sensitive attribute tensor (0 or 1)
* `opts` - Options (see `ExFairness.Metrics.EqualOpportunity.compute/4`)
## Returns
A map containing fairness metrics. See `ExFairness.Metrics.EqualOpportunity.compute/4`
for details.
"""
@spec equal_opportunity(Nx.Tensor.t(), Nx.Tensor.t(), Nx.Tensor.t(), keyword()) ::
EqualOpportunity.result()
def equal_opportunity(predictions, labels, sensitive_attr, opts \\ []) do
EqualOpportunity.compute(predictions, labels, sensitive_attr, opts)
end
@doc """
Computes predictive parity disparity between groups.
Predictive parity requires that PPV/precision is equal across groups.
## Parameters
* `predictions` - Binary predictions tensor (0 or 1)
* `labels` - Binary labels tensor (0 or 1)
* `sensitive_attr` - Binary sensitive attribute tensor (0 or 1)
* `opts` - Options (see `ExFairness.Metrics.PredictiveParity.compute/4`)
## Returns
A map containing fairness metrics. See `ExFairness.Metrics.PredictiveParity.compute/4`
for details.
"""
@spec predictive_parity(Nx.Tensor.t(), Nx.Tensor.t(), Nx.Tensor.t(), keyword()) ::
PredictiveParity.result()
def predictive_parity(predictions, labels, sensitive_attr, opts \\ []) do
PredictiveParity.compute(predictions, labels, sensitive_attr, opts)
end
@doc """
Computes calibration fairness between groups using predicted probabilities.
Calibration checks whether predicted probabilities align with actual outcomes
equally across groups, reporting ECE/MCE per group and the disparity.
## Parameters
* `probabilities` - Predicted probabilities (0.0 to 1.0)
* `labels` - Binary labels tensor (0 or 1)
* `sensitive_attr` - Binary sensitive attribute tensor (0 or 1)
* `opts` - Options (see `ExFairness.Metrics.Calibration.compute/4`)
## Examples
iex> probs = Nx.tensor([0.1, 0.3, 0.6, 0.9, 0.2, 0.4, 0.7, 0.8, 0.5, 0.3,
...> 0.1, 0.3, 0.6, 0.9, 0.2, 0.4, 0.7, 0.8, 0.5, 0.3])
iex> labels = Nx.tensor([0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0])
iex> sensitive = Nx.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
iex> result = ExFairness.calibration(probs, labels, sensitive, n_bins: 5)
iex> result.passes
true
"""
@spec calibration(Nx.Tensor.t(), Nx.Tensor.t(), Nx.Tensor.t(), keyword()) ::
Calibration.result()
def calibration(probabilities, labels, sensitive_attr, opts \\ []) do
Calibration.compute(probabilities, labels, sensitive_attr, opts)
end
@doc """
Generates a comprehensive fairness report across multiple metrics.
## Parameters
* `predictions` - Binary predictions tensor (0 or 1)
* `labels` - Binary labels tensor (0 or 1)
* `sensitive_attr` - Binary sensitive attribute tensor (0 or 1)
* `opts` - Options (see `ExFairness.Report.generate/4`). To include calibration, pass `probabilities: probs`.
## Returns
A comprehensive fairness report. See `ExFairness.Report.generate/4` for details.
## Examples
iex> predictions = Nx.tensor([1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1])
iex> labels = Nx.tensor([1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1])
iex> sensitive = Nx.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
iex> report = ExFairness.fairness_report(predictions, labels, sensitive)
iex> report.total_count
4
"""
@spec fairness_report(Nx.Tensor.t(), Nx.Tensor.t(), Nx.Tensor.t(), keyword()) ::
Report.report()
def fairness_report(predictions, labels, sensitive_attr, opts \\ []) do
Report.generate(predictions, labels, sensitive_attr, opts)
end
@doc """
Evaluates fairness using a CrucibleIR.Reliability.Fairness configuration.
This function provides a bridge between the Crucible framework and ExFairness,
allowing fairness evaluation to be configured using CrucibleIR's configuration
structures.
Note: This function is available when the `crucible_ir` dependency is loaded.
## Parameters
* `predictions` - Binary predictions tensor (0 or 1)
* `labels` - Binary labels tensor (0 or 1)
* `sensitive_attr` - Binary sensitive attribute tensor (0 or 1)
* `config` - CrucibleIR.Reliability.Fairness configuration struct
* `probabilities` - (Optional) Prediction probabilities for calibration metrics
## Returns
A map containing:
* `:metrics` - Map of metric results for each configured metric
* `:overall_passes` - Boolean indicating if all metrics pass
* `:violations` - List of metrics that failed to pass
## Examples
iex> config = %CrucibleIR.Reliability.Fairness{
...> enabled: true,
...> metrics: [:demographic_parity],
...> threshold: 0.1
...> }
iex> predictions = Nx.tensor([1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0])
iex> labels = Nx.tensor([1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1])
iex> sensitive = Nx.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
iex> result = ExFairness.evaluate(predictions, labels, sensitive, config)
iex> result.overall_passes
true
"""
@spec evaluate(Nx.Tensor.t(), Nx.Tensor.t(), Nx.Tensor.t(), struct(), Nx.Tensor.t() | nil) ::
%{
metrics: map(),
overall_passes: boolean(),
violations: list(map())
}
def evaluate(predictions, labels, sensitive_attr, config, probabilities \\ nil)
def evaluate(
predictions,
labels,
sensitive_attr,
%{__struct__: CrucibleIR.Reliability.Fairness} = config,
probabilities
) do
if config.enabled do
evaluate_enabled(predictions, labels, sensitive_attr, config, probabilities)
else
%{metrics: %{}, overall_passes: true, violations: []}
end
end
defp evaluate_enabled(predictions, labels, sensitive_attr, config, probabilities) do
opts = build_metric_opts(config)
metrics_results =
compute_metrics_results(
config.metrics,
predictions,
labels,
sensitive_attr,
probabilities,
opts
)
violations = build_violations(metrics_results)
%{
metrics: metrics_results,
overall_passes: Enum.empty?(violations),
violations: violations
}
end
defp build_metric_opts(config) do
extra_opts = if config.options, do: Map.to_list(config.options), else: []
[threshold: config.threshold] ++ extra_opts
end
defp compute_metrics_results(
metrics,
predictions,
labels,
sensitive_attr,
probabilities,
opts
) do
Enum.reduce(metrics, %{}, fn metric, acc ->
Map.put(
acc,
metric,
compute_metric(metric, predictions, labels, sensitive_attr, probabilities, opts)
)
end)
end
defp compute_metric(:demographic_parity, predictions, _labels, sensitive_attr, _probs, opts) do
demographic_parity(predictions, sensitive_attr, opts)
end
defp compute_metric(:equalized_odds, predictions, labels, sensitive_attr, _probs, opts) do
equalized_odds(predictions, labels, sensitive_attr, opts)
end
defp compute_metric(:equal_opportunity, predictions, labels, sensitive_attr, _probs, opts) do
equal_opportunity(predictions, labels, sensitive_attr, opts)
end
defp compute_metric(:predictive_parity, predictions, labels, sensitive_attr, _probs, opts) do
predictive_parity(predictions, labels, sensitive_attr, opts)
end
defp compute_metric(:calibration, _preds, _labels, _sensitive_attr, nil, _opts) do
%{error: "Calibration requires probabilities", passes: false}
end
defp compute_metric(:calibration, _preds, labels, sensitive_attr, probabilities, opts) do
calibration(probabilities, labels, sensitive_attr, opts)
end
defp compute_metric(metric, _preds, _labels, _sensitive_attr, _probs, _opts) do
%{error: "Unknown metric: #{metric}", passes: false}
end
defp build_violations(metrics_results) do
metrics_results
|> Enum.filter(fn {_metric, result} -> not Map.get(result, :passes, false) end)
|> Enum.map(fn {metric, result} -> %{metric: metric, details: result} end)
end
end