PICurv 0.1.0
A Parallel Particle-In-Cell Solver for Curvilinear LES
 
Loading...
Searching...
No Matches
User How-To Guides

Table of Contents

This page provides operational recipes for common PICurv tasks. Each recipe includes what to change, why it matters, and a quick verification action.

1. Setup and Physics

1.1 Change Reynolds Number

What to change (in case.yml):

properties:
scaling:
length_ref: 0.1
velocity_ref: 1.5
fluid:
density: 1000.0
viscosity: 0.001

Why:

\[ Re = \frac{\rho U L}{\mu} \]

These values set the non-dimensional operating point consumed by solver controls.

Quick check:

  • rerun picurv validate ...,
  • inspect generated .control file for updated values.

1.2 Run in 2D

models:
physics:
dimensionality: "2D"
grid:
mode: programmatic_c
programmatic_settings:
im: 3

Why:

  • 2D holds the i velocity component fixed (-TwoD 1), so the flow plane is j-k and the thin direction must be i; thinning j or k instead freezes an in-plane component,
  • the grid stays three-dimensional, so a small im keeps the structured-grid machinery working without resolving a direction the flow does not use.

Quick check:

  • confirm the generated control file contains -TwoD 1 and the expected i resolution,
  • start from an initial condition whose i component is zero, since 2D never changes it.

Status: 2D is experimental; see 2D.

1.3 Increase Grid Resolution

  • programmatic_c: increase im/jm/km arrays,
  • file: use finer .picgrid,
  • grid_gen: increase generator resolution args.

Verification:

  • compare runtime memory/cost and key output metrics across resolutions.

2. Boundary Conditions

2.1 Set a Constant-Velocity Inlet and Walls

boundary_conditions:
- face: "-Zeta"
type: "INLET"
handler: "constant_velocity"
params: {vx: 0.0, vy: 0.0, vz: 1.5}
- face: "-Eta"
type: "WALL"
handler: "noslip"
# define all remaining faces explicitly

Why:

  • handler/type compatibility is validated,
  • all faces must be covered for each block.

Verification:

  • use validate first and check BC generation files under <run.config>/.

2.2 Enable Periodicity in One Direction

boundary_conditions:
blocks:
- id: 0
faces:
- {face: "-Xi", type: PERIODIC, handler: geometric}
- {face: "+Xi", type: PERIODIC, handler: geometric}

Periodicity is derived exclusively from paired PERIODIC boundary conditions; there are no separate YAML periodic flags. Geometric periodicity requires the opposite grid surfaces to match pointwise under one constant Cartesian translation and at least four physical nodes along the periodic axis.

Verification:

  • confirm the startup banner reports the expected BC-derived periodic axis and validated translation.

3. Running and Monitoring

3.1 Run in Parallel and Control DMDA Layout

./bin/picurv run -n 16 --solve ...

Optional partition hints:

grid:
da_processors_x: 4
da_processors_y: 2
da_processors_z: 2

Note: da_processors_* are scalar globals, not per-block vectors.

3.2 Run on Slurm (Generate and Submit)

./bin/picurv run --solve --post-process \
--case my_case/case.yml \
--solver my_case/solver.yml \
--monitor my_case/monitor.yml \
--post my_case/post.yml \
--cluster my_case/cluster.yml

Stage artifacts without starting execution:

./bin/picurv run --solve --post-process \
--case my_case/case.yml \
--solver my_case/solver.yml \
--monitor my_case/monitor.yml \
--post my_case/post.yml \
--no-submit

Add --cluster my_case/cluster.yml to stage Slurm scripts instead of local command metadata.

Submit an already staged run later:

./bin/picurv submit --run-dir runs/<run_id>

Cancel a submitted run by directory:

./bin/picurv cancel --run-dir runs/<run_id> --stage solve

Request a solver final-output shutdown before exiting:

./bin/picurv cancel --run-dir runs/<run_id> --stage solve --graceful

Use plain cancel if the solver is wedged or not reaching runtime checkpoints.

Generated Slurm solver jobs already enable an automatic runtime walltime guard. Override it only when you need a different warmup/headroom policy:

execution:
walltime_guard:
enabled: true
warmup_steps: 10
multiplier: 2.0
min_seconds: 60
estimator_alpha: 0.35

Ask Slurm for an early warning signal as fallback protection when you want PICurv to flush one last snapshot before walltime or preemption:

execution:
extra_sbatch:
signal: "USR1@300"

If the batch script launches mpirun directly, use signal: "B:USR1@300" and prefer exec mpirun ....

Verification:

  • inspect <run.scheduler>/*.sbatch and submission.json in run directory.
  • confirm the generated solver script exports PICURV_JOB_START_EPOCH and PICURV_WALLTIME_LIMIT_SECONDS.
  • confirm the generated cluster profile contains the intended signal fallback policy before submission.

3.3 Restart from a Saved Step

run_control:
start_step: 500
total_steps: 1000

Pass --restart-from on the CLI to point at the previous run:

./bin/picurv run --solve --post-process \
--restart-from ../runs/flat_channel_20260303-120000 \
--case restart_case/case.yml \
--solver restart_case/solver.yml \
--monitor restart_case/monitor.yml \
--post restart_case/post.yml

Meaning:

  • If a run has completed through step 500, set start_step: 500.
  • The next run loads the saved state at step 500.
  • The first new step advanced is step 501.
  • total_steps is the number of additional steps to run.
  • In this example, the restarted run advances from step 501 through step 1500.
  • When --restart-from is given, picurv automatically resolves the previous run's restart directory from that run's <run.config>/monitor.yml and injects the correct -restart_dir into the new control file.

Keep solver.yml -> operation_mode.eulerian_field_source: "solve" (the default). With start_step > 0 the solver reads the restart state at start_step and then advances it. load is a different mode: it replays a stored field at every step and never solves, so a continuation under load needs a checkpoint at every step it would advance through.

Particle restart choices (case.yml):

Full restart of the existing particle swarm:

models:
physics:
particles:
restart_mode: "load"

Restart the flow field but reseed particles from scratch:

models:
physics:
particles:
restart_mode: "init"

Common combinations:

  • Full restart: start_step > 0, eulerian_field_source: solve, restart_mode: load
  • Flow restart + fresh particles: start_step > 0, eulerian_field_source: solve, restart_mode: init
  • Frozen-field particle replay: eulerian_field_source: load, with a checkpoint at every replayed step
  • Analytical mode is different: eulerian_field_source: analytical regenerates the analytical field at the requested (t, step) instead of loading restart files.

How to think about this workflow:

  • restart uses the normal run --solve path; there is no separate restart command,
  • the new run directory is a fresh run artifact,
  • the saved field state is loaded from existing restart/output files referenced by the current restart path contract,
  • --restart-from on the CLI is the preferred way to point picurv at the old run automatically,
  • use --continue as a shorthand when resuming from the most recent run of the same case,
  • the old run directory is not mutated in place by picurv.

Before launching a restart, verify:

  • the previous run actually wrote solver outputs for the target start_step,
  • the --restart-from path points to the intended previous run directory,
  • the selected previous run has its checkpoint component locally available (restore it with picurv storage restore --run-dir <previous_run> --checkpoint <start_step> when cold),
  • restart source files for the requested step exist,
  • start_step matches an actual saved timestep, not just a desired number.

Common restart mistakes:

  • Setting start_step: 501 after a run that ended at 500. Use start_step: 500.
  • Setting solver.yml -> operation_mode.eulerian_field_source: load for a continuation. That replays stored fields instead of solving, and fails at the first step with no checkpoint.
  • Forgetting to choose particles.restart_mode: load or init explicitly.
  • Trying to restart from a step that was never written to disk.

Verification:

  • confirm the selected checkpoint bundle and step in run logs,
  • confirm the banner/load path shows the expected restart step,
  • if particles are enabled, confirm the log shows the intended particle restart mode.

See also:

3.4 Enable Targeted Debug Logging

logging:
verbosity: "DEBUG"
enabled_functions:
- Projection
- UpdatePressure

Use this for local diagnosis of instability or boundary anomalies. Prefer narrow function lists to keep logs manageable.

4. Post-Processing Recipes

4.1 Postprocess an Existing Run

./bin/picurv run --post-process \
--run-dir runs/flat_channel_20240401-153000 \
--post my_study/standard_analysis.yml

Use when solver outputs already exist and you are iterating only on analysis pipeline.

To catch up the same recipe in batches while the solver is still running, repeat the same command and keep the full desired window in post.yml. Each run processes only the steps whose output is missing or stale.

Behavior notes:

  • if the same recipe already produced steps 0..60 and post.yml still asks for 0..100, PICurv processes only 70..100.
  • if source files currently exist only through 420, PICurv processes the committed steps and exits successfully; a later run picks up the newer steps.
  • if you change the recipe itself, it is a new recipe with its own output directory, and its whole window is processed.
  • to regenerate output that is still up to date, for example after rebuilding the postprocessor, add --recompute.

For the broader run-directory lifecycle around restart, post-only reuse, and generated scheduler artifacts, see Run Artifact Lifecycle Contract.

For a physical-time movie across a restart campaign, add io.paraview_series: {enabled: true, scope: lineage} to the existing post recipe. Postprocess the parent before the child, using the same recipe and a window that includes the desired parent history. The child processes only its owned steps and writes <run.visualization>/<recipe_id>/<output_filename_prefix>.pvd, which references both runs. Open that PVD in ParaView. A later catch-up invocation refreshes it; reopen it to load new frames. Use scope: run to restrict the view to one run. Different solver timesteps use checkpoint physical time; changing the post stride or fields creates a different recipe ID, which is not automatically combined with the earlier one. See Configuration Reference: Postprocessor YAML for reset and retention rules.

4.2 Add Q-Criterion to Eulerian Pipeline

eulerian_pipeline:
- task: nodal_average
input_field: Ucat
output_field: Ucat_nodal
- task: q_criterion
- task: nodal_average
input_field: Qcrit
output_field: Qcrit_nodal
io:
eulerian_fields:
- Ucat_nodal
- Qcrit_nodal

Qcrit itself is cell-centred and cannot be written directly; the nodal average places it on the grid nodes the .vts file uses.

Verification:

  • open VTK output and confirm the Qcrit_nodal field is present.

4.3 Enable Statistics Output (MSD)

statistics_pipeline:
output_prefix: "Stats"
tasks:
- task: msd

Verification:

  • check <run.analysis>/statistics/<recipe_id>/Stats_msd.csv.

4.4 Measure Turbulent Energy Spectra

spectra:
output_prefix: "Spectrum"
tasks:
- task: shell_spectrum
field: Ucat
symbol: continuum
subtract_mean: none # or window:<name> where the flow is inhomogeneous
./bin/picurv run --post-process --only spectra \
--run-dir runs/dit_20240401-153000 \
--post my_study/post.yml
./bin/picurv summarize --run-dir runs/dit_20240401-153000 --plot-spectrum

--only spectra skips the field post-processor, so re-measuring after changing the binning or the fluctuation definition costs seconds rather than a full rebuild of the .vts output.

Verification:

  • <run.analysis>/spectra/<recipe_id>/Spectrum_shell_spectrum_Ucat_block0000_continuum.csv holds step,time,k,energy, one row per shell per processed step.
  • the _history.csv beside it holds one row per step; parseval_residual there must stay at round-off, since summed shell energy must equal the resolved kinetic energy.

shell_spectrum requires a triply periodic, single-block, uniform Cartesian box, and picurv validate --case ... --post ... refuses a case that is not one before any field is read.

For anything else, which spectrum applies depends on how many directions are statistically homogeneous:

  • one or two — a periodic channel or straight duct. line_spectrum and plane_spectrum tasks transform one selected physical line or plane on a single-block Cartesian grid. Set axes to the uniform periodic directions and fixed_indices to the remaining physical cell indices; see 8. spectra. Parallel-sample averaging remains planned in 13. Offline Line And Plane Spectra.
  • none — a bend, a wake, an immersed geometry. No spatial spectrum exists; what is wanted is a frequency spectrum at a probe, also planned; see 14. Temporal Spectra From Bounded Probe Histories.

5. Case Initialization and Binary Management

5.1 Initialize a New Case

picurv init flat_channel --dest my_case

This copies template files and writes metadata. Runtime binaries (simulator, postprocessor) are resolved from the project bin/ directory via PATH — no copies are placed in the case.

5.2 Pin Executables for Reproducibility

picurv run --solve --cluster cluster.yml --pin-executables --case case.yml --solver solver.yml --monitor monitor.yml

--pin-executables copies simulator and postprocessor into the run's <run.config.bin> and launches those copies at every stage of the run, so rebuilding the repository after staging does not change what the job runs. Set reproducibility.pin_executables: true in the workspace to pin every run staged there. See 12. Binary Resolution and Rebuild Safety for continuation and sweep behavior.

picurv init --pin-binaries predates this. Its case-local copies are used only when the picurv in that case directory is invoked, which init does not set up.

5.3 Rebuild Safety

  • picurv (the Python conductor) can be updated at any time — it only launches jobs, it does not run during solver execution.
  • simulator and postprocessor in bin/ are overwritten by make all. If a queued Slurm job references them by absolute path, the running binary may change.
  • Pinned runs are unaffected by a rebuild. An unpinned Slurm job that has not started stops at its job-start identity check instead of running the new build; restage it, or stage with --pin-executables if you expect to rebuild while it is queued.

6. Sweep Studies

./bin/picurv sweep \
--study my_study/study.yml \
--cluster my_study/cluster.yml

What you get:

  • expanded case matrix,
  • scheduler array scripts,
  • aggregated metrics table (auto-collected after jobs complete),
  • optional plots.

If a case is killed (e.g. walltime), continue the study:

./bin/picurv sweep --continue --study-dir studies/<study_id>

Re-aggregate metrics manually:

./bin/picurv sweep --reaggregate --study-dir studies/<study_id>

See Sweep and Study Guide for full contract details.

7. Next Steps

Generated by doxygen 1.9.8