Common Fatal Errors and Fixes

Table of Contents

• 1. Structured Validation Errors (ERROR <CODE>)
• 2. High-Frequency Fatal Messages (Runtime Paths)
• 3. Restart Mistakes To Check First
• 4. Recommended Debug Workflow

Use this page to map error output to concrete fix steps.

1. Structured Validation Errors (ERROR <CODE>)

Validation and CLI-combination failures use:

ERROR <CODE> | key=<yaml.path or -> | file=<path[:line] or -> | message=<...> | hint=<...>

Code reference:

Code	Meaning	Typical Fix
CLI_USAGE_INVALID	Invalid argument combination	Run picurv <command> --help and follow required flag groups.
CFG_MISSING_SECTION	Required YAML section missing	Add section from examples/master_template/*.yml.
CFG_MISSING_KEY	Required YAML key missing	Add required key at the reported path.
CFG_INVALID_TYPE	Wrong YAML value type	Change scalar/list/mapping type per contract docs.
CFG_INVALID_VALUE	Unsupported value/range	Use allowed enums/ranges from reference pages.
CFG_FILE_NOT_FOUND	Referenced file/path is missing	Fix path or create file.
CFG_GRID_PARSE	Grid payload/format error	Fix block count, dimensions, or coordinate rows.
CFG_INCONSISTENT_COMBO	Conflicting options/keys	Align related flags/keys (periodic pairs, scheduler, process counts).

Cross-file combinations that no single file can rule out are reported the same way:

Message fragment	Likely cause	Fix
is not a multiple of io.data_output_frequency	The post recipe asks for steps at a finer stride than the solver commits checkpoints at	For multiple checkpoints, use a positive multiple of the monitor cadence. For one checkpoint, set start_step = end_step and step_interval: 1.
is not defined in the monitor configuration	A post recipe names a field-statistics window the monitor never defines	Correct the window name, or add the window to the monitor configuration.
produce no field for window	The requested derived outputs need accumulator state the window does not keep	Add second to a field's moments for stresses, RMS, or TKE, or add a covariance for a flux.

Geometric-periodic runtime validation can also stop before timestepping:

Message fragment	Likely cause	Fix
Periodic geometry requires at least four physical nodes	Periodic direction is too short for the supported stencils	Increase the node count along that axis.
Periodic geometry translation ... is zero	Opposite surfaces occupy the same coordinates	Supply distinct translated opposite surfaces or remove periodic BCs.
Periodic geometry translation varies / surfaces do not match	Opposite surfaces are not pointwise copies under one constant translation	Regenerate or correct the grid so every paired point differs by the same translation.

2. High-Frequency Fatal Messages (Runtime Paths)

Message fragment	Likely cause	Fix command(s)
[FATAL] –post-process requires –run-dir when not used with –solve.	Post-only run missing input run directory	picurv run –post-process –run-dir runs/<run_id> –post post.yml
[FATAL] Could not automatically identify required config files in <run>/config	Missing/renamed case.yml, monitor.yml, or *.control	Re-run solve stage or restore expected files under runs/<run_id>/config/.
[FATAL] Source data directory for post-processing not found or empty	Post source data path points to missing/empty solver output	Verify monitor output directory and existing solver output files.
[FATAL] Unsupported scheduler '<x>'. Only Slurm is supported.	cluster.yml has unsupported scheduler type	Set cluster.yml -> scheduler.type: slurm.
[FATAL] In cluster mode, –num-procs must be 1 (auto) or exactly nodes*ntasks_per_node (...)	MPI count mismatch against cluster resources	Set -n 1 or set -n to nodes * ntasks_per_node.
Local x-width of domain x 2 is smaller than stencil width s 3 (or the y/z variants)	poisson_solver.multigrid.levels is too deep for the MPI rank layout. The coarsest level halves down to fewer nodes per rank than the stencil width, which is 3 whenever any axis is periodic. Aborts during DM creation, before the first timestep.	Lower multigrid.levels and delete the now-unused level_solvers.level_N entry, or reduce the rank count. Constraint is floor((N / 2^(levels-1) + 2) / ranks_per_axis) >= stencil_width for N cells; worked maxima are tabulated in Pressure-Poisson, GMRES, and Multigrid. When grid.da_processors_x/y/z is set, picurv run (including --dry-run) refuses such a layout before launch.
undefined symbol: petscstack when simulator or postprocessor starts	The executable was compiled against a debug PETSc, and the library the loader found is an optimized one (or another PETSc altogether). On module clusters this is a build made with petsc-debug loaded, launched where petsc-prod is loaded - including inside jobs whose profile restores petsc-prod.	picurv version shows the PETSc each executable was built against and the libpetsc this shell would load. Load the module the executables were built with, or rebuild under the module the jobs use (module restore petsc-prod, then picurv build or picurv versions install).
‘libgfortran.so.5: version 'GFORTRAN_10’ not foundorlibstdc++.so.6: version 'GLIBCXX_3.4.29' not found\ilinebr </td> <td class="markdownTableBodyNone"> PETSc was built with a newer GCC toolchain than the runtime libraries found by the dynamic loader. On module-based clusters this usually means the solver ranks are launching without the matching compiler runtime library path, a stale.picurv-python-envprepended/lib64, or the binary was built under a different module stack. \ilinebr </td> <td class="markdownTableBodyNone"> Reload the same compiler/MPI/PETSc modules used for the build, rebuild from clean objects, then verifyldd bin/simulatorresolveslibgfortranandlibstdc++to the module tree instead of/lib64. Iflddis correct butpicurv runstill fails, inspect.picurv-python-envand remove generic system dirs such as/lib64; then checkmpirun -n 1 envand exportLD_LIBRARY_PATH` to MPI ranks explicitly if needed.

3. Restart Mistakes To Check First

Mistake	Likely result	Fix
Set start_step: 501 after a run that ended at 500	Off-by-one mismatch against the saved restart state	Set start_step: 500; the first new step will be 501.
Set eulerian_field_source: load to continue a run	The run replays stored fields instead of solving, and aborts at the first step past the source run's last checkpoint	Keep eulerian_field_source: solve; start_step > 0 already loads the restart state. Use load only to replay a stored field.
Omit or mis-set particles.restart_mode	Unexpected particle reseed/load behavior or warning-driven behavior	Set restart_mode: load or restart_mode: init explicitly.
--continue with a start_step before the run's last checkpoint	Refused: an in-place continuation cannot rewind, since later committed checkpoints would stay from the old trajectory	Set start_step to the last checkpoint to extend the run, or branch from the earlier step with --restart-from.
Choose a start_step that was never written	Runtime file-not-found or missing-data restart failure	Verify the requested restart step exists in the saved <run.solver_output>/restart files.

4. Recommended Debug Workflow

1. Run config-only checks first:

./bin/picurv validate --case case.yml --solver solver.yml --monitor monitor.yml --post post.yml

1. Preview launch plan:

./bin/picurv run --solve --post-process --case case.yml --solver solver.yml --monitor monitor.yml --post post.yml --dry-run

1. If cluster path is used, validate cluster.yml separately:

./bin/picurv validate --cluster cluster.yml

Related pages:

• Tutorial: Your First Simulation (Flat Channel)
• Configuration Contract (YAML -> Generated Artifacts -> Runtime)
• The Conductor Script: picurv
• Testing and Validation Guide