1. Inputs and Templates

A sweep/study commonly uses:

base case/solver/monitor/post templates,
study.yml defining parameter combinations and metrics,
optional cluster scheduler settings for array submission.

Starter templates are available under examples/*/*study*.yml and examples/master_template/.

2. Core Sweep Command

./bin/picurv sweep --study <study.yml> --cluster <cluster.yml>

Optional generation-only mode:

./bin/picurv sweep --study <study.yml> --cluster <cluster.yml> --no-submit

Delayed submit from existing staged study artifacts:

./bin/picurv submit --study-dir studies/<study_id>

There is no dedicated --dry-run flag on sweep; use --no-submit for non-submitting artifact generation.

With --pin-executables, each member's run directory receives its own simulator and postprocessor, and the array scripts launch the member's copy through $RUN_DIR. Without it every member launches the installation's bin/, checked against the identity read at staging. See 12. Binary Resolution and Rebuild Safety.

3. Study Contract Essentials

A study definition usually specifies:

base_configs:
- case, solver, monitor, post paths (all required)
study_type:
- one of grid_independence, timestep_independence, sensitivity
parameters:
- non-empty mapping of <target>.<yaml.path> -> non-empty list of values
- <target> must be one of case, solver, monitor, post
parameter_sets:
- optional alternative to parameters for coupled overrides that should move together
- non-empty list of explicit <target>.<yaml.path> -> scalar bundles
- provide exactly one of parameters or parameter_sets
metrics (optional):
- list of metric specs or metric names for aggregation
plotting (optional):
- output controls (enabled, output_format)
execution (optional):
- controls like max_concurrent_array_tasks for Slurm array throttling

Each combination yields a generated run with fully materialized config set.

Parameter keys can target nested case/solver/monitor/post values such as:

case.models.physics.particles.count
case.run_control.dt_physical

Not every study should use the default msd_final metric shorthand. Cases that write other scalar diagnostics, such as <run.analysis.metrics>/interpolation_error.csv, should define explicit CSV metrics instead. Search and migration characterization studies can aggregate <run.analysis.metrics>/search_metrics.csv columns such as search_failure_fraction, search_work_index, re_search_fraction, or normalized run-level signals derived from lost_cumulative.

CSV metric specs also support reduction: p95, per-row ratios via numerator_column plus denominator_column, and scalar normalization through normalize_by_parameter for observables such as run loss fraction.

4. Outputs and Aggregates

Expected study outputs include:

studies/<study_id>/cases/case_####/ per-combination run directories
studies/<study_id>/scheduler/case_index.tsv
studies/<study_id>/scheduler/solver_array.sbatch
studies/<study_id>/scheduler/post_array.sbatch
studies/<study_id>/scheduler/metrics_aggregate.sbatch
studies/<study_id>/scheduler/solver_<array_jobid>_<taskid>.out/.err after submission
studies/<study_id>/scheduler/post_<array_jobid>_<taskid>.out/.err after submission
studies/<study_id>/scheduler/submission.json (when jobs are submitted)
studies/<study_id>/output/analysis/metrics_table.csv
studies/<study_id>/output/analysis/plots/* (when plotting is enabled and matplotlib is available)
studies/<study_id>/study_manifest.json

This keeps raw run data and comparative study diagnostics in one reproducible structure.

Metrics aggregation runs automatically as a Slurm job chained after the post-processing array (afterany dependency). If the automatic metrics job fails (e.g. Python unavailable on compute nodes), use --reaggregate manually.

5. Operational Workflow

Recommended workflow:

run a tiny subset locally or with --no-submit,
verify parameter substitution and metric extraction,
launch full array, either directly with picurv sweep ... or later with picurv submit --study-dir ...,
inspect aggregate outputs (auto-collected by the metrics Slurm job, or via --reaggregate),
archive the exact study file with results for reproducibility.

picurv sweep is the scheduler-backed study path. For local parameter studies, repeat picurv run manually across a small set of edited case variants and compare the resulting run directories.

For fragile metrics, add smoke tests or fixture-based validation before large queue submissions.

Implementation details worth knowing:

case expansion uses the full cross-product over all parameters.* lists, or explicit paired bundles from parameter_sets when coupled overrides must move together.
within parameters, mapping a parent key to a list of dicts sweeps multiple sub-fields together as a unit — they are NOT cartesian-producted with each other, but ARE cross-producted against other parameters. Example: case.run_control: [{total_steps: 400, dt_physical: 0.005}, {total_steps: 800, dt_physical: 0.0025}] sweeps the whole run_control block as two groups.
generated case configs are revalidated through the same solver/post validators used by picurv run.
submission chain: solver array → post array (afterok) → metrics job (afterany).
<run.scheduler>/submission.json is the study-directory contract consumed by picurv submit --study-dir ....
generator/file grid external paths are rewritten to absolute paths during case materialization so they remain valid in studies/<study_id>/cases/....
generated solver_array.sbatch exports walltime metadata for the runtime walltime guard. After successful field processing, post_array.sbatch also finalizes each member's PVD collections when its post recipe enables io.paraview_series. Collections describe one member's run or restart ancestry; they do not concatenate independent sweep members into one time series.
post_array.sbatch uses the same nodes * ntasks_per_node allocation as the solver array; conflicting launcher -n/-np flags are rewritten to that task count.
each member's post runs under that member's post lock and processes only the steps whose output is missing or stale when its array task starts, as picurv run --post-process does (see 4. run: Single-Case Workflow).

6. Continuing a Partially-Completed Study

If any solver case is killed (e.g. by the walltime guard or Slurm time limit), the entire post array is cancelled (afterok dependency). Use --continue to resume the study:

./bin/picurv sweep --continue --study-dir studies/<study_id>

To override cluster resources (e.g. increase walltime):

./bin/picurv sweep --continue --study-dir studies/<study_id> \

--cluster cluster_more_time.yml

What --continue does:

Reads the original study.yml and case_index.tsv from the study directory.
Classifies each case as complete, partial, or empty by scanning checkpoints.
If all cases are complete, auto-aggregates metrics and exits (no jobs submitted).
For partial cases: updates case.yml (start_step, total_steps), sets particle restart_mode to load when a checkpoint exists, and delegates to resolve_restart_source for the full restart scenario matrix.
For empty cases (no checkpoint): re-runs from scratch with unmodified control files.
Submits a sparse solver array (incomplete cases only) → full post array → metrics aggregation. The post array covers every member, but a member whose output is up to date launches no postprocessor, so only the continued members' new steps are processed. With --no-submit the three scripts are staged instead; picurv submit --study-dir <study> submits them in that order.

Repeated continuation is safe: the target step count is always computed from the original study.yml, not from the (potentially modified) per-case case.yml.

6.1 When A Study Does Not Finish

One member's solve failed and the post array never ran. The post array waits on the whole solver array with afterok, so one failed member cancels it. Fix the cause, then picurv sweep --continue --study-dir <study> resubmits only the incomplete members.
**--continue refuses a member in cold storage.** Restore it with the picurv storage restore command the message prints, or pass --auto-fetch.
A metric column is empty for a member that completed. The metric's file_glob or regex did not match that member's output. Correct study.yml, then picurv sweep --reaggregate --study-dir <study> rebuilds the table without running anything.
No job ID after sweep or submit. Some sites' sbatch rejects a submission and still exits 0; 7. The Job Fails on a Cluster but Works Locally says how to see why.

7. Manual Metrics Re-Aggregation

If the automatic metrics Slurm job fails or you want to re-collect metrics after manual intervention:

./bin/picurv sweep --reaggregate --study-dir studies/<study_id>

This reads all case outputs, writes studies/<study_id>/output/analysis/metrics_table.csv, and generates plots (if enabled in study.yml).

7.1 Study Type Entries

Value	Maps to
`grid_independence`	`grid_independence`
`sensitivity`	`sensitivity`
`timestep_independence`	`timestep_independence`

grid_independence

Identity. study.yml -> study_type: grid_independence.

What it does. Expands a cross-product over grid parameters and aggregates the declared metrics against resolution, so the study answers whether the result has stopped changing with the mesh.

When to choose it. Before trusting any quantitative result from a new geometry. It is the first study to run and the one most often skipped; a converged solver on an unconverged grid is still the wrong answer.

Parameters it owns. parameters naming the grid keys to vary - typically case.grid.programmatic_settings.im/jm/km - and metrics naming what to compare.

Interactions. The sweep recognises the grid keys and orders the aggregated table by resolution. Combines naturally with grid.mode: grid_gen, which regenerates the mesh per case; with mode: file every resolution needs its own staged mesh.

Diagnostics. studies/<study_id>/output/analysis/metrics_table.csv carries one row per case. A metric that does not settle across the ladder is the finding, not a failure of the study.

Evidence. Production exercised - examples/flat_channel/grid_independence_study.yml, and on the cluster in cluster-session-grace-2026-10-01: a three-member ladder ran as a Slurm array throttled to one task at a time and aggregated one row per member.

Limitations. It varies what you list and nothing else: refining only one direction produces a table that looks converged while the unrefined direction still controls the error.

timestep_independence

Identity. study.yml -> study_type: timestep_independence.

What it does. Expands a cross-product over timestep parameters and aggregates metrics against dt, separating temporal discretisation error from spatial.

When to choose it. After grid independence, and before any claim about a time-dependent quantity. Run it separately from the grid ladder: varying both at once cannot attribute the change to either.

Parameters it owns. parameters naming the timestep keys, and metrics.

Interactions. Independent of the momentum solver chosen, but the useful dt range is not: an explicit path is stability-limited where the implicit paths are not, so the ladder that is informative for one may not run at all for the other.

Diagnostics. Same studies/<study_id>/output/analysis/metrics_table.csv shape as the grid ladder, ordered by dt.

Evidence. Production exercised - cluster-session-grace-2026-10-01: a two-member study ran as a Slurm array and aggregated one row per member. No shipped study selects it.

Limitations. Nothing here detects that the ladder crossed a stability boundary rather than a convergence one; a diverged case appears as a missing or absurd metric row.

sensitivity

Identity. study.yml -> study_type: sensitivity.

What it does. Expands either a cross-product or explicit parameter_sets over arbitrary configuration keys and aggregates the declared metrics, without assuming the varied parameter is a discretisation control.

When to choose it. For everything that is not a convergence ladder: Reynolds number, model coefficients, boundary treatments, solver tolerances. Use parameter_sets rather than parameters when the combinations are coupled and a full cross-product would include cases that make no sense.

Parameters it owns. parameters (cross-product) or parameter_sets (explicit coupled sets), and metrics.

Interactions. The only study type that does not order its table by a convergence quantity, so the aggregation is a plain comparison. execution.max_concurrent_array_tasks matters most here, because a sensitivity sweep is usually the widest.

Diagnostics. studies/<study_id>/output/analysis/metrics_table.csv with one row per case and the varied keys as columns.