The Conductor Script: picurv

Table of Contents

• 1. General Usage
• 2. init: Create A New Case Directory
• 3. build: Build Project Executables
• 3b. sync-binaries / sync-config / pull-source / status-source
• 4. run: Single-Case Workflow
• 5. summarize: Read-Only Run Health Summary
• 5b. submit: Submit Existing Slurm Artifacts
• 5c. cancel: Stop A Slurm Run By Run Directory
• 6. sweep: Parameter Study via Slurm Arrays
• 7. validate: Config-Only Checks
• 8. Full Command and Option Matrix
• 8. <tt>validate --strict</tt>: Additional Checks
• 9. Dry-Run JSON Plan Schema
• 10. Structured Error Output Contract
• 11. Modular Profile Strategy
• 12. Binary Resolution and Rebuild Safety
• 13. Generated Runtime Artifacts
• 14. Next Steps
  • CFD Reader Guidance and Practical Use
    • What To Extract Before Changing A Case
    • Practical CFD Troubleshooting Pattern

picurv is the workflow orchestrator for PICurv. It validates YAML inputs, generates C runtime artifacts, and runs or schedules solver/postprocessing stages. It is also the primary user-facing contract layer: many defaults, aliases, and translation rules are enforced here before the C solver starts.

1. General Usage

picurv [COMMAND] [ARGS...]

After make all, bin/picurv is a symlink to scripts/picurv (the single source of truth). Source etc/picurv.sh to add bin/ to your PATH and expose scripts/ as a fallback so picurv works from any directory even if bin/picurv is temporarily absent before make conductor recreates the symlink. If bin/picurv does not exist yet, run ./scripts/picurv build or make conductor.

Primary commands:

• init
• build
• sync-binaries
• sync-config
• pull-source
• status-source
• run
• summarize
• submit
• cancel
• sweep
• validate

Core idea:

• case.yml, solver.yml, monitor.yml, and post.yml are modular profiles.
• You do not need to rewrite all four every time.
• In normal use, you mix and match a case definition with reusable solver, monitor, and post profiles as long as the combination is contract-compatible.
• Example: the same case.yml can be paired with a quieter monitor.yml, a different solver strategy, or a different post recipe without changing the physical setup file.

Help:

./bin/picurv --help
./bin/picurv init --help
./scripts/picurv build --help
./bin/picurv sync-binaries --help
./bin/picurv sync-config --help
./bin/picurv pull-source --help
./bin/picurv status-source --help
./bin/picurv run --help
./bin/picurv summarize --help
./bin/picurv submit --help
./bin/picurv cancel --help
./bin/picurv sweep --help
./bin/picurv validate --help

2. init: Create A New Case Directory

picurv init <template_name> [--dest <new_dir>] [--pin-binaries]

Behavior:

• copies examples/<template_name>/ into a new working directory,
• optionally renames the destination via --dest,
• writes .picurv-origin.json with the source repo path and template name,
• writes .picurv-execution.yml for optional site-specific launcher overrides,
• seeds that file from a repo-root .picurv-execution.yml when the source clone already has one, otherwise from inert defaults,
• does not copy binaries by default; runtime executables are resolved from the project bin/ directory via PATH.

Binary pinning (--pin-binaries):

• when --pin-binaries is passed, simulator and postprocessor are copied into the case directory,
• case-local copies take precedence over bin/ originals at runtime (resolve_runtime_executable checks the invocation directory first),
• use this when submitting Slurm jobs and you may rebuild the repo before the job runs,
• picurv itself is never copied — it is always used from PATH and is safe to update mid-run since it only launches the C binaries.

Equivalent manual step (after init): picurv sync-binaries --case-dir <case>.

Examples:

picurv init flat_channel --dest my_first_case
picurv init bent_channel --dest my_bent_case --pin-binaries

3. build: Build Project Executables

./scripts/picurv build [--source-root <repo>] [--case-dir <case>] [MAKE_ARGS...]

Behavior:

• calls the top-level Makefile via make,
• defaults to make all when you do not provide an explicit make target,
• resolves the source repo from .picurv-origin.json when run from an initialized case,
• passes any trailing arguments directly through to the Make/build layer,
• can rebuild or clean the source repo without leaving a copied case directory,
• writes the streamed build output to logs/build.log in the source repo,
• is the recommended command for normal users instead of invoking make manually.

Direct make all keeps the traditional stdout-only behavior. When you want the same source-repo build plus a warnings-only artifact, use:

make audit-build

This writes:

• logs/build.log
• logs/build.warnings.log

Examples:

./scripts/picurv build
./scripts/picurv build clean-project
./scripts/picurv build SYSTEM=cluster
./scripts/picurv build postprocessor
./my_case/picurv build clean-project
make audit-build

3b. sync-binaries / sync-config / pull-source / status-source

These commands are intended for case directories created by init.

sync-binaries copies simulator and postprocessor from the source repo bin/ into a case directory for version-pinning (optional — normally binaries are resolved via PATH):

./my_case/picurv sync-binaries

sync-config refreshes files from examples/<template_name>/ into the case directory. By default it preserves user-modified files and only copies missing files:

./my_case/picurv sync-config
./my_case/picurv sync-config --overwrite
./my_case/picurv sync-config --prune

--prune is conservative: it removes only files previously recorded as template-managed that no longer exist in the source template. User-created case files are not pruned. sync-config does not copy execution.example.yml into a case; instead it creates .picurv-execution.yml only when the case does not already have one.

pull-source refreshes every local branch with a configured upstream, then restores the branch you started on, so you can update code without leaving the case directory:

./my_case/picurv pull-source
./my_case/picurv pull-source --current-branch-only
./my_case/picurv pull-source --no-rebase
./my_case/picurv pull-source --remote origin --branch main

status-source inspects source commit drift, copied binary drift, and template-file drift before you decide what to sync:

./my_case/picurv status-source
./my_case/picurv status-source --format json

For older cases that do not yet have .picurv-origin.json, pass --source-root /path/to/PICurv. For sync-config, also pass --template-name <example_name> if the template cannot be inferred.

status-source --format json payload highlights:

• source_repo_root, case_dir
• last_known_source_git_commit, current_source_git_commit, source_commit_changed
• binaries:
  • case_bin_current, case_bin_different, case_bin_missing
• config:
  • case_current_files, case_modified_files, case_missing_files
  • template_removed_since_last_sync (when template-managed tracking is available)

4. run: Single-Case Workflow

./bin/picurv run [STAGES] [INPUTS] [OPTIONS]

Stages:

• --solve
• --post-process

Inputs for --solve:

• --case <case.yml>
• --solver <solver.yml>
• --monitor <monitor.yml>

Inputs for --post-process:

• --post <post.yml>
• either same invocation with --solve, or --run-dir <existing_run_dir>

MPI/local options:

• -n, --num-procs
  • applies to solver stage launch sizing.
  • post-processing is forced to one rank/task. PICurv strips conflicting MPI size flags from post launches and rewrites them to rank 1.
  • local multi-rank runs resolve launcher overrides in this order: PICURV_MPI_LAUNCHER, MPI_LAUNCHER, nearest .picurv-execution.yml, nearest legacy .picurv-local.yml, then default mpiexec.

Cluster/Slurm options:

• --cluster <cluster.yml>
• --scheduler slurm (optional explicit selector)
• --no-submit (generate scripts/manifests only)

Preflight options:

• --dry-run (resolve and print launch/artifact plan only)
• --format json (machine-readable output for --dry-run)

Local example:

./bin/picurv run --solve --post-process -n 8 \
  --case my_case/case.yml \
  --solver my_case/solver.yml \
  --monitor my_case/monitor.yml \
  --post my_case/post.yml

In this command, solver runs with 8 ranks; post-processing is still forced to 1 rank.

Slurm example (generate + submit):

./bin/picurv run --solve --post-process \
  --case my_case/case.yml \
  --solver my_case/solver.yml \
  --monitor my_case/monitor.yml \
  --post my_case/post.yml \
  --cluster my_case/cluster.yml

Slurm example (generate only):

./bin/picurv run --solve --post-process \
  --case my_case/case.yml \
  --solver my_case/solver.yml \
  --monitor my_case/monitor.yml \
  --post my_case/post.yml \
  --cluster my_case/cluster.yml \
  --no-submit

Follow-up submission from existing artifacts:

./bin/picurv submit --run-dir runs/<run_id>

Post-only continuation examples:

• Catch up a live run without editing post.yml.start_step. Keep the full desired analysis window in post.yml, then let --continue move the launch cursor inside that window:

./bin/picurv run --post-process --continue \
  --run-dir runs/search_robustness_20260322-073415 \
  --post search_robustness_analysis.yml

• If the solver has only written source data through step 420, PICurv launches only the fully available prefix in the requested stride. A later --continue run picks up the newer steps after the solver produces them.
• If the same recipe already post-processed the requested window, PICurv skips the launch and reports that the run is already caught up.
• If you change the recipe itself, for example by adding Qcrit or changing the statistics output prefix, PICurv treats that as a new recipe lineage and starts again from the configured start_step.
• PICurv allows only one post writer per run directory. If a second post job targets the same runs/<run_id>, it is refused immediately instead of racing on viz/ or statistics/.

Graceful shutdown note:

• generated Slurm solver jobs enable a runtime walltime guard by default; after 10 completed warmup steps, PICurv estimates timestep cost and requests the same graceful final-write path before remaining walltime gets too tight.
• if cluster.yml -> execution.extra_sbatch.signal requests an early warning signal, PICurv also traps SIGUSR1, SIGTERM, and SIGINT, then writes one last snapshot at the next safe checkpoint even when the normal recording interval has not been reached.
• tune the automatic estimator in cluster.yml -> execution.walltime_guard; keep execution.extra_sbatch.signal as fallback protection for preemption/manual termination or jobs that may not reach the warmup window.
• use signal: "USR1@300" for srun-launched jobs, or signal: "B:USR1@300" plus exec mpirun ... for direct mpirun batch launches.

Runtime stream logs:

• C-managed logs remain under runs/<run_id>/logs/.
• wrapper stdout/stderr stream logs now live under runs/<run_id>/scheduler/ for both local and Slurm launches.
• this avoids collisions with solver startup, which recreates the C log directory.

5. summarize: Read-Only Run Health Summary

./bin/picurv summarize --run-dir <run_dir> [--latest | --step <n>] [--format json]

Behavior:

• reads copied run configs plus existing runtime artifacts,
• builds a best-effort step summary without changing solver output,
• works for active runs and completed runs,
• reports unavailable sections when a source log is missing or disabled.

Examples:

./bin/picurv summarize --run-dir runs/my_case_20260310-120000 --latest
./bin/picurv summarize --run-dir runs/my_case_20260310-120000 --step 500
./bin/picurv summarize --run-dir runs/my_case_20260310-120000 --latest --format json

Typical sources:

• logs/Continuity_Metrics.log
• logs/Particle_Metrics.log (per-step Lost plus run-local Lost Total when available)
• logs/Momentum_Solver_Convergence_History_Block_*.log
• logs/Poisson_Solver_Convergence_History_Block_*.log
• logs/Profiling_Timestep_Summary.csv when enabled
• scheduler/*_solver.log or scheduler/solver_*.out for sampled particle snapshot previews

When particle snapshots are available, summarize reports sampled diagnostics such as:

• speed min/mean/max/std and stagnant-count
• sampled position bounds and centroid
• sampled rank counts and duplicate sampled cells
• sampled weight min/max by component
• sanity checks for duplicate PIDs and non-finite/zero/negative weights
• top sampled speeds and sampled deltas versus the previous snapshot when matching PIDs exist

Dry-run example (no file writes):

./bin/picurv run --solve --post-process \
  --case my_case/case.yml \
  --solver my_case/solver.yml \
  --monitor my_case/monitor.yml \
  --post my_case/post.yml \
  --dry-run --format json

Common run use cases:

• first full local run: --solve --post-process
• solver-only run: --solve
• post-only rerun on an existing run directory: --post-process --run-dir ...
• cluster script generation without submit: --cluster ... --no-submit
• delayed submission of an already-staged run: submit --run-dir ...
• planning and CI-style checks: --dry-run

5b. submit: Submit Existing Slurm Artifacts

./bin/picurv submit [--run-dir <run_dir> | --study-dir <study_dir>] \
  [--stage {all,solve,post-process}] [--force] [--dry-run]

Behavior:

• consumes existing --no-submit artifacts without regenerating configs or scripts,
• reads scheduler/submission.json to locate the staged Slurm scripts,
• submits solve, post-process, or both,
• wires the post stage dependency automatically when all is selected,
• refuses re-submission unless --force is explicitly provided.

Examples:

./bin/picurv submit --run-dir runs/my_case_20260310-120000
./bin/picurv submit --run-dir runs/my_case_20260310-120000 --stage solve
./bin/picurv submit --study-dir studies/my_study_20260310-120000 --dry-run

Notes:

• works only for Slurm-staged artifacts,
• --dry-run prints the exact sbatch plan,
• --force is the opt-in path for deliberate resubmission.

5c. cancel: Stop A Slurm Run By Run Directory

./bin/picurv cancel --run-dir <run_dir> [--stage {all,solve,post-process}] [--dry-run]

Behavior:

• reads scheduler/submission.json from an existing run directory,
• resolves the recorded Slurm job IDs for solve and/or post-process,
• runs scancel for the selected stage set,
• avoids manual job-id lookup when the run directory is already known.

Examples:

./bin/picurv cancel --run-dir runs/my_case_20260310-120000
./bin/picurv cancel --run-dir runs/my_case_20260310-120000 --stage solve
./bin/picurv cancel --run-dir runs/my_case_20260310-120000 --dry-run

Notes:

• works only for Slurm-submitted runs that have scheduler/submission.json,
• does not apply to local runs,
• --dry-run is useful when you want to confirm the recorded stage/job mapping first.

6. sweep: Parameter Study via Slurm Arrays

# Launch new study
./bin/picurv sweep \
  --study my_study/study.yml \
  --cluster my_study/cluster.yml [--no-submit]
 
# Continue a partially-completed study
./bin/picurv sweep --continue --study-dir studies/<study_id> \
  [--cluster cluster_more_time.yml]
 
# Re-aggregate metrics manually
./bin/picurv sweep --reaggregate --study-dir studies/<study_id>

Behavior (new study):

• expands parameter matrix from study.yml
• materializes case directories under studies/<study_id>/cases/
• generates solver_array.sbatch, post_array.sbatch, and metrics_aggregate.sbatch
• renders post_array.sbatch with a single-task allocation and a forced one-rank launcher command
• submits solver → post (afterok) → metrics (afterany) chain (unless --no-submit)

Behavior (--continue):

• detects per-case completion status (complete / partial / empty)
• if all cases complete, auto-aggregates metrics and exits
• otherwise prepares continuation for incomplete cases (checkpoint restart via resolve_restart_source)
• submits sparse solver array (incomplete cases only) → full post array → metrics aggregation

Behavior (--reaggregate):

• re-runs metrics collection and plot generation on existing study outputs

7. validate: Config-Only Checks

./bin/picurv validate \
  --case my_case/case.yml \
  --solver my_case/solver.yml \
  --monitor my_case/monitor.yml \
  --post my_case/post.yml

validate does not launch solver/post and does not create run/study artifacts.

What validate is for:

• check a new profile combination before running,
• confirm a modified template still satisfies the current schema,
• catch mode-dependent contract errors before the C runtime,
• inspect warnings where picurv preserves a C-side default intentionally.

8. Full Command and Option Matrix

This section is intentionally exhaustive and mirrors the current argparse contract in scripts/picurv. Use it as the authoritative option reference when writing docs, examples, wrappers, or CI jobs.

run:

• stages:
  • --solve (requires --case, --solver, --monitor)
  • --post-process (requires --post; requires --run-dir when --solve is not selected)
• solver inputs:
  • --case <path>
  • --solver <path>
  • --monitor <path>
• post inputs:
  • --post <path>
  • --run-dir <path>
• launch controls:
  • -n, --num-procs <int> (solver stage only; post is forced to 1 rank/task)
  • --cluster <cluster.yml> (enables Slurm mode)
  • --scheduler <name> (must be used with --cluster; must match cluster.yml:scheduler.type)
  • --no-submit (render scripts/manifests without sbatch)
  • --dry-run (no file writes; plan only)
  • --format {text,json} (dry-run output format)

validate:

• role selectors:
  • --case <path>
  • --solver <path>
  • --monitor <path>
  • --post <path>
  • --cluster <path>
  • --study <path>
• stricter policy:
  • --strict (adds additional checks for selected roles; documented below)

submit:

• required:
  • exactly one of --run-dir <path> or --study-dir <path>
• optional:
  • --stage {all,solve,post-process}
  • --force
  • --dry-run

cancel:

• required:
  • --run-dir <path>
• optional:
  • --stage {all,solve,post-process}
  • --dry-run

sweep:

• new study mode (default):
  • --study <study.yml> (required)
  • --cluster <cluster.yml> (required)
  • --no-submit (optional)
• continuation mode:
  • --continue (required)
  • --study-dir <path> (required)
  • --cluster <cluster.yml> (optional; overrides original cluster resources)
• reaggregation mode:
  • --reaggregate (required)
  • --study-dir <path> (required)

init:

• positional:
  • <template_name>
• optional:
  • --dest <dir>
  • --source-root <repo>
  • --pin-binaries (copy simulator/postprocessor into the case for version-pinning)

build:

• optional:
  • --source-root <repo>
  • --case-dir <case_dir>
• passthrough:
  • trailing MAKE_ARGS... are passed directly to make

sync-binaries:

• --case-dir <case_dir>
• --source-root <repo>

sync-config:

• --case-dir <case_dir>
• --source-root <repo>
• --template-name <template>
• --overwrite
• --prune

pull-source:

• --case-dir <case_dir>
• --source-root <repo>
• --remote <git_remote>
• --branch <git_branch>
• --no-rebase

status-source:

• --case-dir <case_dir>
• --source-root <repo>
• --template-name <template>
• --format {text,json}

8. <tt>validate --strict</tt>: Additional Checks

--strict does not change baseline schema validation, but it adds file-system consistency checks for selected roles:

• with --post:
  • if source_data.directory is not <solver_output_dir>, the resolved directory must exist.
• with --study:
  • base configs listed in study.base_configs are loaded and revalidated as real case/solver/monitor/post bundles.
  • this catches study files that are syntactically valid but point to invalid base configurations.

Use --strict in CI/pre-submit checks when validating reusable profile libraries or study manifests.

9. Dry-Run JSON Plan Schema

picurv run --dry-run --format json emits a deterministic plan payload with these top-level keys:

• mode ("dry-run")
• created_at (ISO timestamp)
• launch_mode (local or slurm)
• warnings (list)
• inputs (resolved absolute paths)
• stages (stage-specific launch plans)
• artifacts (predicted files/directories, deduplicated)
• run_id_preview / run_dir_preview (when known)
• solver_num_procs_effective
• post_num_procs_effective
• num_procs_effective (currently mirrors solver count)

Stage entries under stages.solve and stages.post-process include:

• mode (local or slurm)
• num_procs_effective
• launch_command (tokenized command list)
• launch_command_string (shell-ready display string)

Additional stage fields:

• script (Slurm script path in cluster mode)
• source_data_directory (post stage source directory resolution)
• restart_source_directory (solve stage, when restart source is resolved)

Dry-run guarantees:

• no run directory creation,
• no control/post recipe writes,
• no scheduler script writes,
• no job submission.

10. Structured Error Output Contract

Validation and CLI usage errors are emitted as one-line, machine-parseable records:

ERROR <CODE> | key=<yaml_or_cli_key> | file=<path_or_dash> | message=<summary> | hint=<actionable_hint>

Current normalized error code set:

• CLI_USAGE_INVALID
• CFG_MISSING_SECTION
• CFG_MISSING_KEY
• CFG_INVALID_TYPE
• CFG_INVALID_VALUE
• CFG_FILE_NOT_FOUND
• CFG_GRID_PARSE
• CFG_INCONSISTENT_COMBO

This contract is exercised by Python tests and should remain stable for wrappers and CI parsers.

11. Modular Profile Strategy

PICurv is intended to be used with reusable profile libraries.

Typical pattern:

1. keep case.yml focused on physics, grid, BCs, and run duration,
2. keep solver.yml focused on numerical strategy,
3. keep monitor.yml focused on logging and I/O cadence,
4. keep post.yml focused on analysis outputs,
5. recombine them as needed for different studies.

Examples:

• same case.yml + a lighter monitor.yml for fast debug runs,
• same case.yml + a stricter solver.yml for convergence checks,
• same case.yml + multiple post.yml recipes for different analysis outputs,
• same solver.yml reused across many cases when the discretization strategy is stable.

This is why picurv treats these roles as separate inputs instead of one monolithic file.

For prebuilt reusable profiles, also see the local guides under:

• config/solvers/
• config/monitors/
• config/postprocessors/
• config/schedulers/
• examples/master_template/

12. Binary Resolution and Rebuild Safety

picurv resolves simulator and postprocessor at launch time using this precedence:

1. Invocation directory — if the binary exists as a sibling of the invoked picurv script (e.g. case-local copies from --pin-binaries or sync-binaries), it is used first.
2. Project bin/ directory — the default location after make all.

bin/picurv is a symlink to scripts/picurv. This means:

• there is exactly one source file (scripts/picurv), so code changes never drift,
• make conductor recreates the symlink (idempotent),
• etc/picurv.sh adds bin/ to PATH so picurv works from any directory.

Rebuilding while jobs are running:

• Updating picurv (the Python script) mid-run is always safe — it is only used to launch jobs, not during solver execution.
• Rebuilding simulator/postprocessor (make all) overwrites the binaries in bin/. If a Slurm job references bin/simulator by absolute path and has not yet started, the running binary may be replaced before execution begins.
• To protect against this, use --pin-binaries at init time or sync-binaries before submission. Case-local copies are isolated from repo rebuilds.

Recommended workflow for concurrent development and production:

picurv init flat_channel --dest production_case --pin-binaries
picurv run --solve --cluster cluster.yml ...   # uses case-local binaries
# safe to rebuild in the repo now — production_case has its own copies
make all

13. Generated Runtime Artifacts

Single run (run):

• runs/<run_id>/config/*.control, bcs*.run, post.run, plus optional whitelist.run / profile.run sidecars when enabled
• runs/<run_id>/logs/* (runtime logs and metrics written by solver/postprocessor)
• runs/<run_id>/scheduler/solver.sbatch, post.sbatch (cluster mode)
• runs/<run_id>/scheduler/solver_<jobid>.out/.err, post_<jobid>.out/.err (cluster mode, after submission)
• runs/<run_id>/scheduler/submission.json (cluster mode)
• runs/<run_id>/manifest.json

Sweep (sweep):

• studies/<study_id>/cases/<case_i>/...
• studies/<study_id>/scheduler/case_index.tsv
• studies/<study_id>/scheduler/solver_array.sbatch
• studies/<study_id>/scheduler/post_array.sbatch
• studies/<study_id>/scheduler/solver_<array_jobid>_<taskid>.out/.err, post_<array_jobid>_<taskid>.out/.err
• studies/<study_id>/scheduler/submission.json
• studies/<study_id>/results/metrics_table.csv
• studies/<study_id>/results/plots/*.png (if plotting enabled and matplotlib available)
• studies/<study_id>/study_manifest.json

14. Next Steps

• Config contract: Configuration Contract (YAML -> Generated Artifacts -> Runtime)
• User workflows: User How-To Guides
• Extensibility: Workflow Extensibility Guide
• First-run path: Tutorial: Your First Simulation (Flat Channel)
• Grid generator details: Grid Generator Guide: scripts/grid.gen
• Modular examples and recipes: Workflow Recipes and Config Cookbook
• Troubleshooting map: Common Fatal Errors and Fixes
• Smoke/CI verification details: Testing and Validation Guide

CFD Reader Guidance and Practical Use

This page describes The Conductor Script: picurv within the PICurv workflow. For CFD users, the most reliable reading strategy is to map the page content to a concrete run decision: what is configured, what runtime stage it influences, and which diagnostics should confirm expected behavior.

Treat this page as both a conceptual reference and a runbook. If you are debugging, pair the method/procedure described here with monitor output, generated runtime artifacts under runs/<run_id>/config, and the associated solver/post logs so numerical intent and implementation behavior stay aligned.

What To Extract Before Changing A Case

• Identify which YAML role or runtime stage this page governs.
• List the primary control knobs (tolerances, cadence, paths, selectors, or mode flags).
• Record expected success indicators (convergence trend, artifact presence, or stable derived metrics).
• Record failure signals that require rollback or parameter isolation.

Practical CFD Troubleshooting Pattern

1. Reproduce the issue on a tiny case or narrow timestep window.
2. Change one control at a time and keep all other roles/configs fixed.
3. Validate generated artifacts and logs after each change before scaling up.
4. If behavior remains inconsistent, compare against a known-good baseline example and re-check grid/BC consistency.