|
PICurv 0.1.0
A Parallel Particle-In-Cell Solver for Curvilinear LES
|
|
Loading...
Searching...
No Matches
|
PICurv 0.1.0
A Parallel Particle-In-Cell Solver for Curvilinear LES
|
|
This page explains how a PICurv run moves from a new solve to restart, post-processing reuse, and cluster job generation. It is the operational view of the run directory lifecycle.
For verified remote protection, cold offload, and later restoration of these artifacts, see Storage Management Guide.
Every arrow is a command you can issue. --continue resumes the same run in place; --restart-from seeds a new run from an existing checkpoint. Those are different operations with different provenance, and choosing between them is the decision this page exists to support.
picurv init <template> --dest <workspace> creates one self-contained campaign workspace. Editable files, imported inputs, reusable generated assets, standalone runs, and studies all have fixed homes beneath it:
The directories are created at initialization even when empty. Files inside them are created only when used. This gives every workspace the same navigable shape without generating unused data.
case.yml -> title is the human run label. A fresh run is named <sanitized-title>_<timestamp>; the title, manifest, and asset hashes carry identity, so users do not need to rename run directories to remember them. The timestamp has one-second resolution, so two runs of the same case started within the same second would name the same directory; the second one takes a -2 suffix rather than being refused, because creating a run never writes into an existing one. A study uses its own title the same way and keeps numbered members below cases/case_####. When an umbrella example contains repeated filenames, initialization preserves the extra configurations below config/variants/<original-path> instead of overwriting them.
A run publishes what it is in its own <run.manifest>: artifact_type, run_id, study_id and case_id for a study member, the paths map naming every component directory, and the lineage record described in §5.5. A study publishes the same way in study_manifest.json.
Everything downstream reads those records rather than re-deriving them. picurv storage takes a run's identity and its component boundaries from the manifest, so a run directory that was renamed, copied, or restored under a different name still reports the identity it was created with, and still classifies its own output correctly. A committed checkpoint's step likewise comes from the -checkpoint_step its checkpoint.meta records, not from the step_############ directory it sits in, so a bundle copied without being re-stamped is not mistaken for the step its new name claims.
A directory carrying no manifest — one staged by an older release — still works: the fixed topology above is assumed, and the fallback is reported as identity_source: directory-name rather than presented as a recorded answer.
A run owns exactly five directories directly below <run.root>: <run.config>, <run.inputs>, <run.solver_output>, <run.runtime_logs>, and <run.scheduler>. Staging a run, resuming one, and post-processing one all refuse a run root that has grown a sixth.
The refusal is not tidiness. A directory beside <run.solver_output> is routed by nothing, so the solver never writes there deliberately; it is classified unclassified by storage, so it is archived in every copy of the run and pruned by no policy; and it is absent from the paths map in <run.manifest>, so no reader can say what it holds. Catching it when the run is staged is the last point at which the answer is still "move it".
Scientific output belongs under <run.checkpoints>, <run.analysis>, or <run.visualization>. An unexpected file at the run root is reported and allowed: a stray note costs nothing, and refusing to resume a long campaign over one would be a worse failure than the one being prevented.
Generated control files, post recipes, scheduler scripts, and a study's case index hold absolute paths into the run or study. <run.manifest> records the run's root (root_path), and study_manifest.json records the study's (paths.study_dir). When picurv run --post-process, run --solve --continue, submit, or sweep --continue/--reaggregate opens a run or study whose recorded root differs from where it now is, it first rewrites those paths to the new location, the same rewrite a relocated picurv storage restore applies, and reports how many files it changed. A --dry-run reports the move without rewriting anything. A path is rewritten only where it ends at a path boundary, so moving runs/run1 leaves runs/run10 untouched. Checkpoint bundles and workspace asset objects are never rewritten, because their contents are checked against recorded digests.
For a run staged before the manifest recorded its root, the old location is read from a control-file path that names one of the run's own files. Paths a run records for other runs, such as a branch's parent_path, are not rewritten.
Run commands from the workspace and refer to canonical configuration paths:
Generator descriptions such as grid .cfg files and expressions.cfg belong in <workspace>/config/grids/, <workspace>/config/initial_conditions/, or <workspace>/config/inlet_profiles/. User-supplied data belong in the matching <workspace>/inputs/ directory. Register an external file explicitly so its checksum and ownership mode are catalogued:
reference writes a small .reference.yml; it never silently copies, archives, or prunes the external target. Configuration paths resolve from the workspace root. Missing configured inputs fail loudly before submission.
Recommended preflight is validate, then run --dry-run, then run --no-submit. The dry run reports the proposed run identity, canonical paths, asset actions, and blocking provider dependencies without creating the directory.
picurv precompute --case <workspace>/config/case.yml resolves a dependency graph for grid, initial-condition, and inlet-profile providers. File and Python providers build in an isolated temporary run layout. PICurv publishes their complete output only after every selected provider succeeds:
Precompute exists so a grid, field, or profile can be looked at and corrected before a solve is committed, so every published object carries inspection material beside its payload. A preview is skipped, and validation.json says so, when the first block exceeds two million nodes: past that the ASCII geometry costs more than the look is worth. Inspection material describes the payload and is deliberately excluded from the object's identity, so changing a preview format does not re-identify every asset.
An asset set is keyed by the case file's name, but reuse is keyed by provider identity: a case file with no matching set adopts any published object carrying the same provider_spec_sha256 and records it in its own set. Payload files record paths relative to themselves, never the temporary build directory, so an unchanged provider rebuilds to the same asset id.
Generated destinations are PICurv's. grid.generator.output_file, stats_file, and vts_file are rejected rather than honoured, as are the generated initial-condition keys output_file, summary_json, and spectrum_csv and a generated or field_slice inlet profile's source.output_file: a configuration file that names its own output path creates a competing directory outside the store, which is what the fixed topology exists to prevent.
The payload keeps its run-relative shape rather than a flat set of canonical filenames, because an inlet-profile asset holds one file per block and face and a flat root cannot express that. Materialization exposes each payload path into the run at the same relative location.
The object identity includes normalized provider settings, the case values each build reads, the identities of the assets it depends on, checksums of referenced files, and the PICurv build. Changing an equation, grid config, imported field, inlet parameters, or generator code therefore selects a new object. Unchanged inputs reuse the existing object. The provider settings include each input's workspace path, so the same bytes imported under two names build two objects. An external reference enters through its target's current checksum: a target changed since registration builds a new object rather than reusing a stale one, and the checksum recorded at registration is kept for provenance, not compared. --only grid,initial-condition selects a dependency closure; publication remains all-or-nothing for that invocation.
A provider executed only in C is reported as runtime-c. Precompute does not imitate it in Python. If a Python initial-condition generator needs a grid configured as programmatic_c, precompute and run planning fail and tell the user to choose file or grid_gen. Otherwise the simulator announces the runtime provider and generates it during startup.
Fresh runs reuse valid assets automatically. --require-precomputed refuses a missing or stale object instead of building it, while --fetch-missing searches configured storage archives before rebuilding. Each run receives physical run-local files by reflink, hardlink, or copy and records the exact object/checksum mapping in inputs/assets.lock.yml.
Every run has this fixed shape:
No peer diagnostics/, results/, or arbitrary monitor-selected output root is created. C runtime metrics use <run.analysis.metrics>; post statistics and spectra use their analysis homes; renderable VTK output uses <run.visualization>.
The initial YAML snapshot is immutable evidence. An in-place continuation stores the new YAML and generated controls under <run.config.history>/<revision>/ and updates <run.config.active>; it does not erase the original. A continuation keeps the pinned executables recorded in <run.config.active>; an explicit re-pin writes new copies under that revision instead of replacing <run.config.bin>. manifest.json records the workspace identity, active build, canonical paths, stages, locked assets, per-executable build identity of the executables the run launches (binaries), and each stage's lifecycle state (components: e.g. not_requested, planned, complete, offloaded).
A run reads its assets from run-local files. Each payload file is exposed into the run by reflink when the filesystem allows it, otherwise by hardlink, otherwise by copy, and inputs/assets.lock.yml records the object and checksum behind every file. On a Lustre scratch filesystem, which refuses reflinks, a run in the same filesystem as its workspace therefore holds hardlinks to the shared objects: never edit a run's input files in place.
A run that starts from saved state (start_step > 0), or whose Eulerian field comes from the load or an analytical source, never reads the configured initial condition. Planning marks that asset unused and neither builds nor materializes it; the grid and inlet profiles are still resolved.
A moved run or study is adopted where it now stands (1.3 Moving Or Copying A Run Or Study). picurv storage restore --run-dir restores the run's own files but not the shared objects in assets/objects/; those come back through --fetch-missing when a run is staged, or by restoring a workspace archive's assets component. Objects that no run references are reclaimed only by an explicit picurv storage prune --assets --unused-locally (4.5 Reclaiming a Shared Asset).
Required precomputed asset(s) are missing or stale comes from --require-precomputed; run the picurv precompute ... --only ... command the message names.Asset generation completed without satisfying: <kind> means precompute ran but that kind still has no object; picurv precompute --case <case> --only <kind> shows the provider's own output.--mode reflink that fails with cp's "Operation not supported" is on a filesystem without reflinks, such as Lustre or ext4. Use copy, or hardlink within one filesystem; nothing falls back silently.--mode hardlink fails across filesystems, for example from a home directory into scratch. Use copy.reference import whose target has disappeared fails when it is consumed. Make the target available at the same path, or import a copy. A target whose bytes changed is used as it now is and builds a new object, without a warning.storage prune --assets --unused-locally reclaims the unused object.PICurv now separates case physics from site execution policy.
Local multi-rank precedence:
PICURV_MPI_LAUNCHERMPI_LAUNCHER.picurv-execution.yml.picurv-local.ymlmpiexecCluster batch precedence:
cluster.yml -> execution.picurv-execution.yml -> cluster_execution.picurv-execution.yml -> default_executionsrunThis gives three clean cases:
picurv init creates .picurv-execution.yml in each new case with inert defaults,.picurv-execution.yml when needed,cluster.yml needs a batch-specific override.Restart and continuation use the normal solve workflow. There is no separate restart command. The restart source is specified entirely through CLI flags rather than YAML keys.
Three scenarios are supported:
Use --restart-from to create a new run that continues solving from another run's checkpoint data.
Relevant YAML settings:
case.yml: set start_step to the checkpoint step (e.g. 500) and total_steps to the desired additional count.solver.yml: set eulerian_field_source: "solve" so the solver advances the Eulerian fields from the restart state.Operational meaning:
inputs/restart/checkpoints/ directory,runs/<new_run_id>/ directory.Use --restart-from when the Eulerian flow is already computed and you only need to track particles through it.
Relevant YAML settings:
solver.yml: set eulerian_field_source: "load" so the solver reads pre-computed Eulerian fields instead of advancing them.Operational meaning:
restart_dir directly at the source run's output (no file copy),runs/<new_run_id>/ directory.Use --continue --run-dir to resume a run that was interrupted or stopped early, writing into the same run directory.
Operational meaning:
case.yml -> run_control.start_step must be the saved checkpoint step and must be greater than zero; use a normal run without --continue for a fresh start at step zero,start_step may not be earlier than the run's last committed checkpoint. The solver never overwrites a committed checkpoint, so a continuation from an earlier one would keep the later states of the old trajectory and mix two histories in one run. To redo a stretch from an earlier checkpoint, branch a new run from that step with --restart-from,<run.checkpoints> and materializes it into <run.inputs>/restart, the one restart home every mode uses, so the generated control carries a single canonical path. Same-filesystem reflink or hardlink materialization makes this metadata-cheap rather than a second copy,runs/my_run/ directory.When start_step > 0, the initial Eulerian state is always loaded from the restart source regardless of the eulerian_field_source setting. The eulerian_field_source value only controls what happens on subsequent steps: "solve" advances the fields, "load" reads pre-computed fields.
Before launching any restart or continuation:
start_step equal to the saved checkpoint step, not the next desired step,--continue is for the same physical case. It permits changes to run-control time settings, the MPI domain decomposition (grid.da_processors_x/y/z and the rank count), solver parameters/type, monitoring, postprocessing, and resource settings. Other case.yml changes are rejected; use --restart-from to create a new branch. --restart-from may change case physics, but C still requires the same grid geometry/layout. A newly enabled optional subsystem such as particles uses its normal initialization path unless its restart mode explicitly requests saved state; subsequent checkpoints then include that subsystem.
A run created with --restart-from writes a lineage record into its <run.manifest>:
Without it a branch is indistinguishable from a fresh run. The bundle copied into the run's restart input carries the parent's geometry digest and software identity, but names no run, so the trajectory a result belongs to could not be reconstructed afterwards.
The parent is named from its own manifest, so a parent that was later renamed is still identified correctly; parent_identity_source says whether that lookup succeeded or fell back to the directory name. requested_source records what was asked for, which is how a --restart-from latest is distinguished from the explicit path it resolved to.
A fresh run records {"relationship": "root"} rather than omitting the key, so a reader never has to distinguish "started from nothing" from "written before lineage
existed". An in-place --continue keeps whatever lineage the run was created with: what a run branched from does not change because it was resumed.
When solver outputs already exist, reuse the run directory directly:
Use this when:
PICurv auto-identifies the active case, monitor, and control artifacts from <run.config.active> and its referenced revision. Every normalized recipe gets a stable ID. Its controls live under <run.post_recipes>/<recipe-id>/, its field visualization under <run.visualization>/<recipe-id>/, and its statistics/spectra below the matching analysis directories. Two changed recipes therefore coexist instead of overwriting one another.
Operational patterns for post-only reuse:
post.yml as the full analysis window you want. Each post run processes only the steps whose output is missing or stale and records each step it produces in config/post-recipes/<recipe-id>/state.json, so you never edit start_step during batch catch-up.post.yml requests 0..1000 every 10, but solver source files currently exist only through step 420, PICurv processes 0..420 on the first pass. A later run processes 430 onward after those source files appear.Field_00070.vts exists but the required MSD CSV still stops at 60, step 70 is incomplete and the next run processes it again. A job that fails part-way records the steps it finished.--recompute regenerates every requested step with a committed checkpoint, rewrites the VTK files for those steps, and rewrites repeated statistics rows so each step still appears once in the final CSV.run_dir at a different post.yml recipe, such as adding Qcrit_nodal or changing the statistics prefix, it is a separate recipe and its whole window is processed.config/post-recipes/<recipe-id>/post.steps lists the steps it processes and post.steps.json holds what recording them needs; both are removed when the job ends.post.yml -> io.paraview_series.enabled: true and scope: lineage, PICurv also writes, after success, a flattened .pvd in the branch's recipe visualization directory. It references compatible ancestor VTK files through each recorded fork and uses every checkpoint's recorded physical time. Repeating the post command while the solver is active grows the collection with the steps committed since.In cluster mode, picurv writes scheduler artifacts into the new run directory:
<run.scheduler>/solver.sbatch<run.scheduler>/solver_<jobid>.out / <run.scheduler>/solver_<jobid>.err<run.scheduler>/post.sbatch<run.scheduler>/post_<jobid>.out / <run.scheduler>/post_<jobid>.err<run.scheduler>/submission.jsonRecommended operational pattern:
--dry-run to confirm launch commands and artifact paths--no-submit to inspect generated local commands or batch scriptspicurv submit --run-dir runs/<run_id> only after the staged artifacts look correctpicurv cancel --run-dir runs/<run_id> for Slurm runs when you need to stop a submitted stage without separate job-id bookkeepingThis is especially useful when changing:
Operational examples:
Generated Slurm solver jobs also export runtime walltime metadata into solver.sbatch, so the solver can estimate completed-step cost and request a graceful final write before remaining walltime gets too tight. If the cluster profile also requests an early signal, PICurv traps SIGUSR1, SIGTERM, and SIGINT, then uses the same safe-checkpoint final-write path. Use signal: "USR1@300" for srun, or signal: "B:USR1@300" plus exec mpirun ... for direct mpirun batch launches.
For manual cancellation, plain picurv cancel is a hard Slurm cancel. Add --graceful when you want the solver process tree to receive SIGUSR1, stop at the next safe checkpoint, and write the latest safe off-cadence output first. Fall back to plain cancel if the job is wedged or not reaching checkpoints.
Before changing anything about a run you intend to carry forward, decide which of three operations you want:
--continue) — same run, same identity, more timesteps.--restart-from) — a new run seeded from an existing checkpoint, with its own identity and provenance.The tables below keep three things apart that are easy to conflate:
enforced (validation accepts or rejects it), tested (a test exercises it), characterized (measured, not gated), unknown (not established).unknown means nobody has checked. It is not a synonym for "no". Where a cell says unknown, the tool may well permit the change and produce something meaningless without complaining.Continuation guards physical case identity: validate_continue_case_identity compares a hash of case.yml with run_control, the particle restart_mode, and grid.da_processors_x/y/z excluded. Everything else in case.yml is part of that identity.
| Change | Accepted? | Advised? | Basis |
|---|---|---|---|
run_control (more timesteps) | yes | yes | enforced — excluded from the identity hash |
Particle restart_mode | yes | yes | enforced — excluded from the identity hash |
| Monitor cadence / logging | yes | yes | enforced — monitor.yml is outside the identity |
| Post-processing recipe | yes | yes | enforced — post.yml is outside the identity |
| Solver tolerances | yes | discouraged | enforced (accepted); effect not characterized |
| Momentum solver selection | yes | discouraged | enforced (accepted); effect unknown |
| Timestep size | yes | discouraged | enforced (accepted); effect not characterized |
| Boundary conditions | no | — | enforced — part of the case identity |
| Turbulence model | no | — | enforced — part of the case identity |
| Physical properties | no | — | enforced — part of the case identity |
| Grid dimensions or geometry | no | — | enforced — part of the case identity |
MPI rank count or da_processors layout | yes | yes | enforced — the layout is excluded from the identity hash (test_continue_accepts_a_changed_domain_decomposition); a 25 x 25 x 97 channel continued from 2 ranks (1 x 1 x 2) to 2 ranks (2 x 1 x 1) and then to 4 (2 x 1 x 2) on 2026-10-09, with divergence at the 1e-8 level of the first segment |
solver.yml, monitor.yml, or post.yml is accepted during continuation by design; the guard is on the physical case only. The "discouraged" rows are a scientific judgement, not a tooling restriction: the resulting trajectory concatenates two differently-converged segments, which is rarely what you want in a published result.A new run carries its own identity, so the case-identity guard does not apply.
| Change | Accepted? | Advised? | Basis |
|---|---|---|---|
| Anything in solver / monitor / post | yes | yes | enforced |
| Boundary conditions | yes | yes, with care | enforced (accepted); physical continuity unknown |
| Turbulence model | yes | yes, with care | enforced (accepted); effect unknown |
| MPI rank count | yes | yes | tested — the field-statistics suite covers a changed rank count across restart |
| Physical properties | yes | treat as a new study | enforced (accepted); the seeded field is no longer consistent with the new physics |
| Grid dimensions or geometry | no | — | enforced — the checkpoint cannot be interpreted on a different grid |
Even with no changes, a restarted run does not reproduce an uninterrupted one bit for bit. The offset is a structural floor from boundary re-application at the restart step, it decays rather than amplifies, and it is invariant to solver tolerance — tightening tolerances does not remove it. Measurements are in Maintenance Backlog and Low-Priority Fixes.
Acceptance criteria for restart equivalence must therefore be set from that measured floor, not from machine epsilon.
constant_flux re-reads its target from the bcs file on every run, so editing it between segments takes effect deliberately. initial_flux instead restores its latched target from the checkpoint, so a resumed run holds the original target rather than re-measuring a drifted one.
| Value | Maps to |
|---|---|
carry | carry |
reset | reset |
Both modes behave as documented on a known field (field-statistics-tgv3d-2026-09-21): carry continues a window exactly and reset starts it afresh. The derived statistics they continue have not yet been compared against a turbulent reference profile.
Identity. picurv run --solve --restart-from ... --statistics-state reset.
What it does. Starts field-statistics accumulators empty in the new branched run; the checkpoint still seeds physical fields.
When to choose it. Use reset for a changed case, a new measurement window, or whenever the new run should stand alone statistically. It is safer than carry when solver or monitoring definitions changed.
Neither value is a default. Branching a run with field_statistics.enabled: true and no --statistics-state is refused, because guessing is wrong in both directions: resetting silently reports a shorter average than the user expects, and carrying averages two trajectories together. With field statistics disabled there is nothing to decide and the flag stays optional.
Parameters it owns. None.
Interactions. Applies to a new run created with --restart-from; in-place --continue already continues the same run state. It does not change checkpoint selection or Eulerian restart authority.
Diagnostics. The generated control omits -field_statistics_continue true, and the new run's active configuration records statistics_state: reset.
Evidence. Unit verified — tests/test_workspace_lifecycle.py exercises the CLI choice and generated run control surface. Analytically verified - field-statistics-tgv3d-2026-09-21: a TGV3D run restarted at step 10 with reset anchored each window at the first state the restarted run completed and accumulated the remaining samples exactly.
Limitations. It cannot merge old and new statistics later; preserve or postprocess the old window separately if both are needed. The interval between the restart point and the first state the new run completes is not represented in the reset window.
Identity. picurv run --solve --restart-from ... --statistics-state carry.
What it does. Requests compatible checkpointed field-statistics accumulator state for the new run before sampling continues.
When to choose it. Use carry only when the grid, field-statistics windows, weighting, fields, and covariance definitions are unchanged and the new run is a scientific continuation of the same measurement.
Parameters it owns. None.
Interactions. Emits -field_statistics_continue true. Checkpoint compatibility validation still owns grid layout and required state; changing the statistics recipe can make the requested state unusable.
Diagnostics. The generated control contains the continue flag. A missing or incompatible checkpoint statistics payload fails during restart setup rather than silently resetting. The restart bundle is staged whenever carry is requested, whatever the Eulerian source: an analytical-source run used to stage none, because it needs no fields, and then failed at start-up looking for the accumulators.
Evidence. Unit verified — tests/test_workspace_lifecycle.py exercises the CLI choice and generated run control surface. Analytically verified - field-statistics-tgv3d-2026-09-21: a TGV3D run restarted at step 10 with carry reproduced the continuous run's windows to 1.1e-16 at step 20.
Limitations. Measured on one rank. make smoke-mpi checks that a window's payloads are identical between one rank and several within a single run, but carrying a window across a restart that changes the rank count, or the optional field set, has not been measured.
<run.config>/ as the ground truth for what the binaries actually consumed..picurv-execution.yml..picurv-execution.yml; keep scheduler policy in cluster.yml.execution.walltime_guard in the cluster profile rather than editing generated scripts.
1.9.8