1. Required Inputs

Typical cluster-enabled run uses:

case.yml
solver.yml
monitor.yml
post.yml
cluster.yml (scheduler contract)

Initialize templates from examples, then customize per cluster/account policy.

2. Core Command Patterns

Generate and submit:

./bin/picurv run \
  --solve --post-process \
  --case <case.yml> \
  --solver <solver.yml> \
  --monitor <monitor.yml> \
  --post <post.yml> \
  --cluster <cluster.yml>

Generate only (no submission):

./bin/picurv run ... --cluster <cluster.yml> --no-submit

The generated script compares the executable's --version with the identity read at staging, after module setup and before the launcher, so a job whose executable was rebuilt while it was queued stops instead of running the new build. Add --pin-executables to copy simulator and postprocessor into <run.config.bin> when you intend to keep rebuilding while the job waits. See 12. Binary Resolution and Rebuild Safety.

Submit existing staged artifacts later:

./bin/picurv submit --run-dir runs/<run_id>

Stop a submitted run by directory:

./bin/picurv cancel --run-dir runs/<run_id>

Request a solver final-output shutdown instead of an immediate hard cancel:

./bin/picurv cancel --run-dir runs/<run_id> --stage solve --graceful

--graceful sends SIGUSR1 to the solver process tree so PICurv can write the latest safe off-cadence step at the next runtime checkpoint. If the job is wedged or not reaching checkpoints, use plain picurv cancel --run-dir ... --stage solve.

Explicit scheduler cross-check (optional):

./bin/picurv run ... --cluster <cluster.yml> --scheduler slurm

Notes:

--scheduler is optional and only valid with --cluster.
if provided, it must match cluster.yml:scheduler.type.
in cluster mode, solver and field postprocessor rank counts are resolved from cluster.yml (nodes * ntasks_per_node).
cluster.yml does not currently name the run directory. picurv generates run_id as <case_basename>_<timestamp>, then derives Slurm job names like <run_id>_solve and <run_id>_post.
batch launcher precedence is: cluster.yml.execution first, then nearest .picurv-execution.yml (cluster_execution, then default_execution), then built-in srun.

3. Generated Scheduler Artifacts

In run directory, scheduler generation typically produces:

<run.scheduler>/solver.sbatch
<run.scheduler>/post.sbatch
<run.scheduler>/solver_<jobid>.out/.err after solver submission
<run.scheduler>/post_<jobid>.out/.err after post submission
runs/<run_id>/manifest.json
<run.scheduler>/submission.json (always in cluster mode when run artifacts are written; contains launch metadata and submission IDs when present)

These coexist with standard runtime control artifacts used by solver/postprocessor binaries. submission.json is also the run-directory contract consumed by picurv submit and picurv cancel.

4. Submission Flow

YAML validation and contract checks,
run directory + control artifact generation,
sbatch script rendering from scheduler settings,
optional solver submission (solver.sbatch) immediately, or later via picurv submit --run-dir ...,
optional post submission (post.sbatch), with afterok:<solver_jobid> dependency when solve+post are both requested in one command or when submit --stage all is used. submit --stage post-process on its own depends on the recorded solve job only while that job is still queued. Slurm forgets a finished job after its MinJobAge and then rejects any dependency on it, so a solve that has completed is not waited for: the post job plans its steps from the committed checkpoints when it starts. A solve that ended FAILED, CANCELLED, TIMEOUT or similar is refused unless --force is given. When sacct cannot say how the job ended, the post job is submitted without a dependency and a warning says so.

This allows consistent local dry-run and cluster production flow from the same inputs.

5. Operational Notes

Prefer --no-submit first when validating new scheduler settings.
Keep cluster defaults in reusable templates (examples/master_template/master_cluster.yml).
For personal cluster copies, prefer local operational files such as short_job.local.yml / long_job.local.yml instead of committing account/module-specific scheduler profiles.
If queue policies differ by partition/account, encode them in cluster.yml instead of editing generated scripts manually.
Generated Slurm solver jobs enable a runtime walltime guard by default:
- warmup: first 10 completed steps
- estimator: max(warmup_avg, ewma, latest_step)
- cutoff: max(60 s, 2.0 * estimator)
Override that policy in cluster.yml -> execution.walltime_guard only when the defaults do not fit your timestep variability or filesystem overhead.
Keep an early signal in cluster.yml -> execution.extra_sbatch.signal as fallback protection: signal: "USR1@300" for srun, or signal: "B:USR1@300" plus exec mpirun ... for direct mpirun batch launches.
Solver and field postprocessor stages use cluster.yml resources directly.
When the selected launcher is MPI-aware (srun, mpirun, mpiexec), PICurv strips conflicting size flags and forces the post launch to nodes * ntasks_per_node ranks.
Slurm stdout/stderr lives under <run.scheduler>/; solver-generated runtime logs still live under <run.runtime_logs>/.
picurv init now creates .picurv-execution.yml in each new case with inert defaults.
If your cluster needs the same MPI launcher tokens for login-node and batch runs, edit that file and let cluster.yml override only when batch jobs differ.
For one-off interactive multi-rank runs on cluster login nodes, PICURV_MPI_LAUNCHER still overrides everything.
--num-procs in cluster mode is a consistency guard, not an independent rank selector:
- allowed values are 1 (auto) or exactly nodes * ntasks_per_node.
- any other value is rejected as an inconsistent configuration combo.
use picurv run --dry-run --format json before submission when integrating with external launch wrappers.
for new profiles, the safest operational loop is --dry-run, then --no-submit, then submit --run-dir ....

6. What Has Been Verified On A Cluster

On TAMU Grace (Slurm, one node, 8 MPI tasks per job), cluster-session-grace-2026-10-01 recorded: picurv cancel stopping a running solver (CANCELLED); cancel --graceful making the solver write an off-cadence checkpoint and exit, and run --continue resuming from it; post jobs planning their steps when they start and skipping current output; and array studies honouring max_concurrent_array_tasks. turbulent-channel-nk-2026-09-29 recorded a 144-rank solve with a dependent post job, and cluster-cancel-pending-2026-09-29 the cancellation of pending jobs. The post-dependency rule of 4. Submission Flow and study continuation inside a workspace were fixed after that session and are verified by the local test suite only. Multi-node arrays and a walltime-guard stop on the cluster have not been exercised.