Computes calibration statistics by binning predicted probabilities and comparing to observed frequencies. This helps diagnose whether the model's probabilistic predictions are reliable.
Arguments
- resp
Matrix of responsibilities (N x K), where resp[i,k] is the posterior probability that subject i belongs to cluster k. Rows should sum to 1.
- z_true
Integer vector of length N with true cluster labels (1 to K). If not available, can use hard cluster assignments from another method.
- n_bins
Integer. Number of bins for calibration curve. Default: 10.
Value
A list with class "calibration_result" containing:
- calibration_df
Data frame with columns: bin_midpoint, mean_predicted, mean_observed, n_samples, cluster
- ece
Expected Calibration Error (weighted average bin error)
- mce
Maximum Calibration Error (worst bin)
- n_bins
Number of bins used
- K
Number of clusters
- N
Number of subjects
Details
Calibration is computed for each cluster separately: 1. Bin the predicted probabilities into n_bins equal-width bins 2. For each bin, compute mean predicted probability and observed frequency 3. A well-calibrated model has mean_predicted close to mean_observed
The Expected Calibration Error (ECE) summarizes overall calibration: $$ECE = \sum_{b=1}^{B} \frac{|S_b|}{N} |acc(S_b) - conf(S_b)|$$ where S_b is the set of samples in bin b, acc is accuracy (observed frequency), and conf is confidence (mean predicted probability).
Examples
if (FALSE) { # \dontrun{
# Fit a mixture model
fit <- mixture_pca_em_fit(list_of_data, K = 3, r = 2)
# Compute calibration using true labels
calib <- compute_calibration(fit$resp, z_true)
# View ECE
print(calib$ece)
# Plot calibration curve
plot_calibration(calib)
} # }