Run Parameter Sweep with Model Selection — run_param_sweep

Orchestrates a comprehensive parameter sweep experiment that includes model selection (K, r) for each parameter combination. This function generates v2 dynamic synergy data, performs model selection using specified strategies, and evaluates performance metrics.

Usage

run_param_sweep_modelsel(
  test = FALSE,
  grid_type = NULL,
  mode = c("modelsel", "oracle_r", "oracle_Kr"),
  cores = 4,
  output_dir = NULL,
  param_grid = NULL,
  K_candidates = 1:5,
  r_candidates = 1:5,
  strategies = c("grid", "twostage"),
  methods = c("MFA", "MPCA"),
  criterion = c("ICL", "BIC"),
  icl_temperature = 2,
  parallel_strategy = c("grid", "sequential", "pbs", "future"),
  pbs_config = list(),
  pbs_submit = FALSE,
  pbs_wait = FALSE,
  future_backend = "multisession",
  future_resources = list(),
  max_iter = 50,
  nIterFA = 5,
  nIterPCA = 5,
  n_init = 1,
  tol = 0.001,
  use_kmeans_init = TRUE,
  n_threads = 1,
  save_models = FALSE,
  save_selection_summaries = TRUE,
  verbose = TRUE
)

Arguments

test

Logical. If TRUE, uses a smaller test grid for quick validation. Default is FALSE (full parameter grid).

grid_type

Character. Parameter grid type: "test", "paper_core", or "paper_stress". If NULL (default), uses the test parameter to determine the grid. When specified, overrides the test parameter.

mode

Character. Selection mode controlling the search behavior:

"modelsel": Normal model selection (K,r grid search)
"oracle_r": Use true r, only select K
"oracle_Kr": Use true K and r (no selection)

Default is "modelsel".

cores

Integer. Number of CPU cores for parallel processing. Default is 4. Set to 1 for sequential processing.

output_dir

Character. Directory to save results. Default is "inst/paper/results_sweep_modelsel" if it exists, otherwise "results_sweep_modelsel" in the current working directory.

param_grid

Optional. Custom parameter grid data frame. If NULL, uses the built-in full or test grid based on the test parameter.

K_candidates

Integer vector. Candidate K values for model selection. Default is 1:5.

r_candidates

Integer vector. Candidate r values for model selection. Default is 1:5.

strategies

Character vector. Model selection strategies to evaluate. Options: "grid", "twostage". Default is c("grid", "twostage").

methods

Character vector. Model types to evaluate. Options: "MFA", "MPCA". Default is c("MFA", "MPCA").

criterion

Character. Model selection criterion: "ICL" or "BIC". Default is "ICL".

icl_temperature

Numeric. Temperature parameter for ICL entropy scaling. Higher values favor more clusters. Default is 2.0.

parallel_strategy

Character. Parallelization strategy:

"grid": Local multicore parallelization using mclapply
"sequential": No parallelization (single core)
"pbs": PBS/OpenPBS array job submission for HPC clusters
"future": Use future framework (supports local and HPC backends)

Default is "grid".

pbs_config

List. PBS configuration options (only used when parallel_strategy = "pbs"). See submit_pbs_sweep for details.

pbs_submit

Logical. If TRUE and parallel_strategy = "pbs", submit the job to PBS. If FALSE, only generate scripts (dry run). Default is FALSE.

pbs_wait

Logical. If TRUE, wait for PBS job completion and aggregate results. Default is FALSE.

future_backend

Character. Backend for future parallelization (only used when parallel_strategy = "future"):

"multisession": Local parallel R sessions (default)
"multicore": Local forked processes (Unix only)
"batchtools_torque": PBS/TORQUE scheduler
"batchtools_slurm": SLURM scheduler

See setup_future_backend for all options.

future_resources

List. HPC resource specifications for future.batchtools backends (e.g., list(walltime = "04:00:00", memory = "4gb")).

max_iter

Maximum EM iterations. Default is 50.

nIterFA

Sub-iterations for FA update. Default is 5.

nIterPCA

Sub-iterations for PCA update. Default is 5.

n_init

Number of random initializations. Default is 1.

tol

Convergence tolerance. Default is 1e-3.

use_kmeans_init

Whether to use k-means initialization. Default is TRUE.

n_threads

Number of OpenMP threads. Default is 1.

save_models

Logical. If TRUE, save fitted models. Default is FALSE.

save_selection_summaries

Logical. If TRUE, save BIC summaries for each run. Default is TRUE.

verbose

Logical. If TRUE, print progress information. Default is TRUE.

Value

Invisibly returns a list containing:

raw_results: Data frame with raw results from all runs
aggregated_results: Data frame with results aggregated across seeds
win_rates: Data frame with win rates by method/strategy
output_dir: Directory where results were saved

Details

The parameter sweep evaluates model selection performance across:

Methods: MFA, MPCA
Strategies: grid (full BIC grid search), twostage (K first, then r)
Metrics: ARI (clustering accuracy), VAF (reconstruction quality), BIC, selection accuracy (K_correct, r_correct, Kr_correct)
Variable parameters: N, K, r, cluster_sep_spatial, cluster_sep_temporal, cluster_sep_stability

Output Schema

The raw_results data frame contains the following columns:

run_id, seed: Identifiers
N, K_true, r_true, M, T_each: True parameters
cluster_sep_spatial, cluster_sep_temporal, cluster_sep_stability: Separation parameters
method: "MFA" or "MPCA"
selection_strategy: "grid" or "twostage"
mode: Selection mode ("modelsel", "oracle_r", or "oracle_Kr")
r_mode: Always "global" (for future cluster-wise r extension)
K_hat, r_hat: Estimated parameters
K_correct, r_correct, Kr_correct: Selection accuracy
ARI, VAF, logLik, BIC: Performance metrics
subspace_similarity: Subspace recovery score (mean across clusters)
runtime_sec: Runtime in seconds
error_msg: Error message (empty string if successful)

Examples

if (FALSE) { # \dontrun{
# Run test mode with 2 cores
results <- run_param_sweep_modelsel(test = TRUE, cores = 2)

# Run full sweep with 8 cores
results <- run_param_sweep_modelsel(test = FALSE, cores = 8)

# Run with custom strategies
results <- run_param_sweep_modelsel(
  test = TRUE, cores = 4,
  strategies = c("twostage"),
  methods = c("MFA")
)
} # }