PICurv 0.1.0
A Parallel Particle-In-Cell Solver for Curvilinear LES
 
Loading...
Searching...
No Matches
Troubleshooting by Symptom

Table of Contents

Type How-toFor Anyone with a failing runStatus Current behavior

Start from what you observed. Each symptom below runs cheapest checks first, so you spend a minute before you spend a cluster allocation.

This page complements Common Fatal Errors and Fixes, which is organized by error message. Use that page when you have an exact string to search for; use this one when you only have a behavior.

1. Validation Fails Before Anything Runs

Good news: this is the cheapest failure mode, and it is the one the system is designed to produce.

  1. Read the reported key path and file — validation names both.
  2. Check the value against the reference page for that role: Configuration Reference: Case YAML, Configuration Reference: Solver YAML, Configuration Reference: Monitor YAML, Configuration Reference: Postprocessor YAML.
  3. If it is a selector value, check it against the generated inventory for that family — an unknown value is rejected rather than defaulted.
  4. If it is a boundary condition, check the type/handler pairing at 2. Boundary Types; not every handler is legal for every type.
  5. Solver-specific blocks must match the selected solver. Setting a dual_time_picard_jameson_rk block while selecting Newton Krylov is an error, not an ignored stanza.

2. The Solver Does Not Converge

Establish where it fails before changing anything.

  1. Is the Poisson side healthy? Check the reported maximum divergence. If it is near machine zero, the pressure solve is fine and the problem is in momentum.
  2. Is it the momentum solve? Under Dual Time Picard Jameson RK, look for "reached N total attempts without convergence". That counts total attempts, accepted plus rejected.
  3. Is dtau pinned at its floor? A pseudo-CFL collapsed to pseudo_cfl.minimum with a residual ratio plateauing near 1 is the signature of a stalled controller rather than a hard case.
  4. Reduce the physical timestep before touching solver internals. Many apparent solver failures are a timestep the physics will not support.
  5. Try a different solver. p08_cap_newton_krylov gives true Newton convergence where pseudo-time stalls.

A periodic wall-bounded case is not a known cause: the stall recorded for it on 2026-08-24 was re-characterized on 2026-09-18 and does not reproduce (5.7 Known limitations).

3. The Solution Blows Up

  1. Check the initial condition is physically sensible — visualize step 0 before blaming the solver.
  2. Check inlet and outlet are consistent: an inlet with no outlet to conserve against will not behave.
  3. For Explicit RK4, a run that stops with "the step exceeds the explicit stability limit" has exceeded it; there is no recovery mechanism on that path. The message gives the viscous bound to size dt against. Reduce the timestep or change solver.
  4. Check the boundary conditions actually applied — the startup banner lists the resolved handler per face. A face you thought was a wall may not be.

4. Particles Are Lost or Behave Oddly

  1. Check the reported lost-particle count per step. Steady loss usually means particles are leaving through a boundary you did not intend.
  2. A freshly seeded particle has valid state: setup interpolates once at t=0 before the loop. If early behavior looks wrong, suspect the seeding, not the ordering.
  3. Isolate the mechanism using the verification family in Example Catalog — each isolates one term of the displacement equation, so a failure points at one thing.
  4. For search or migration anomalies, Search Robustness Metrics Reference defines the metrics to look at.

5. Output Is Missing or Unreadable

  1. Check the monitor configuration: output cadence controls whether anything is written at all.
  2. Check the postprocessor's own stream log, written to **<run.scheduler>/** as <run_id>_<prefix>.log — not <run.runtime_logs>/, which carries the solver's runtime logging. Solver success does not imply postprocessor success.
  3. Confirm the field you expect is one the run actually produced; Field Identity and Layout Catalog is authoritative on what exists.
  4. Remember Ucat is derived from Ucont. A visualization field is not the solver's evolved state — see Grid, Cell, and Variable Architecture Guide if a field looks inconsistent.

6. Restart Is Rejected or Behaves Unexpectedly

For a missing or incomplete PVD, first check post.yml -> io.paraview_series. Lineage indexing requires matching recipe IDs in accessible ancestor runs and physical times from committed checkpoints or retained PVDs. Changing post stride or fields creates a new recipe ID; an old directory is not automatically searched. Generate matching ancestor output, then repeat the child's post command. Open the child's .pvd, and reopen it after catch-up. A missing referenced VTK file cannot be recovered from the PVD itself; restore that run's visualization data.

  1. Confirm you want the operation you asked for: --continue resumes the same run; --restart-from seeds a new one. See Run Artifact Lifecycle Contract.
  2. Check that what you changed is compatible with continuing — 8. What You Can Change, And How To Continue.
  3. Restart is not bit-exact. A small difference immediately after restart is expected and documented in Maintenance Backlog and Low-Priority Fixes; it is a structural floor, not a tolerance you can tighten away.
  4. For initial_flux, a restart restores the latched target from the checkpoint. A log line saying it re-measured instead means the checkpoint predates that metadata.

7. The Job Fails on a Cluster but Works Locally

  1. Distinguish scheduler failure from solver failure: look at the scheduler .err file before the solver log.
  2. Confirm binaries and paths resolve on the compute node, not just the login node.
  3. Check the MPI rank count against the grid decomposition. A rank count that partitions the grid awkwardly can also limit how deeply the Poisson multigrid can coarsen, since each level must remain decomposable across the ranks it runs on. If the Poisson solve refuses your mglevels at one rank count but not another, this is usually why.
  4. Stage without submitting and inspect the generated scripts before blaming the solver: Cluster Run Guide (Slurm).
  5. Could not parse Slurm job id from sbatch output with nothing after it means sbatch accepted no job. Some sites wrap sbatch and exit 0 on a rejection, printing the reason only on stderr: run the staged script by hand (sbatch <run.scheduler>/solver.sbatch) to see it. An account that is no longer active is the usual cause; check the site's allocation command and resources.account in cluster.yml.
  6. undefined symbol: petscstack, or another symbol error as the executable starts, means it was built against a different PETSc than the job loads. picurv version shows both; see 2. High-Frequency Fatal Messages (Runtime Paths).

8. Results Look Numerically Suspicious

This is the hardest class, and the one where documentation helps least — but three checks are worth doing before anything else:

  1. Check units. Values are physical in YAML and non-dimensionalized before C. target_flux is a volumetric flux, not a velocity. A constant-factor error here produces plausible-looking wrong answers. See 5. Non-Dimensionalization Before C Input.
  2. Check the evidence for what you used. Capability Evidence Matrix shows what has actually been verified. An unverified capability is a candidate suspect.
  3. Check for known limitations on the capability entry for each selector you set.

9. LES Results Look Wrong

The closure is quiet when it misbehaves: a wrong coefficient still produces a run that completes and plots that look plausible. Turn the diagnostics on (les.diagnostics.enabled: true) and read <run.analysis.metrics>/les_coefficient.csv before anything else.

**cs_effective is zero for the whole run.** Expected for the first two steps of a run started from rest, because the procedure has no developed field to sample. Beyond that, check that the model is dynamic_smagorinsky; the constant model reports the coefficient you configured, not a measured one.

**cs_effective is far from 0.18-0.19 in isotropic turbulence.** That is what the dynamic procedure measured on the HOM02 benchmark at 64^3 with the box test filter (0.193-0.196 homogeneous, 0.181-0.183 local; 10. Status and Evidence). Lilly's 0.16-0.17 assumes a spectral cutoff in an inertial range. Check the grid first, not the model: at low Re_lambda a smaller coefficient is correct. Then check limited_fraction — if the clip is binding, the ceiling is setting the answer.

**cs_effective is noisy or oscillating.** Almost always averaging.mode: local, where the coefficient's denominator collapses wherever the resolved strain is briefly small. If the flow has a homogeneous direction, use averaging.mode: homogeneous.

**limited_fraction is large.** The max_cs ceiling is doing the modelling. Either the coefficient is genuinely diverging, which is a resolution or stability problem, or the ceiling is set too low for the flow.

**backscatter_fraction is large and the run is over-dissipative.** The clipping modes discard only the negative tail, which biases mean dissipation upward. Averaging over a homogeneous set lets forward and reverse transfer cancel the way the physics intends; clipping.mode: none keeps the sign and bounds the total viscosity instead.

**nu_t_over_nu_mean is far below one.** The model is contributing nothing. Check that LES is actually enabled, and that the grid is coarse enough for a subgrid model to be the right tool.

Validation rejected simpson_ik. That stencil averages over the central eta-plane only and assumes xi and zeta are homogeneous. The check requires both declared PERIODIC. Use volume_weighted_box on any other geometry.

More detail on every setting at Configuration Reference: Case YAML, and the reasoning at LES Turbulence Closure.

10. A Wall-Model Run Stops or Its Wall Stress Looks Wrong

Read <run.analysis.metrics>/wall_model.csv first: one row per step with the wall-face mean, spread and extremes of the friction velocity, the mean and maximum first-cell y+, and nu_wall_over_nu_mean.

The run stopped with "first-cell y+ has been outside ...". The runtime checks the wall-face mean y+ every step against the selected law's range (log_law 30-300, werner up to 300, cabot up to 1000) and stops after ten consecutive steps outside it. A log_law run started from a plug or laminar field often sits below 30 at first; werner has a linear branch and no lower bound. Otherwise resize the first cell.

**u_tau_mean collapses toward zero within a few steps**, with nu_wall_over_nu_mean falling to zero beside it. The flow has stopped, and the wall model is reporting it rather than causing it. Check that the inflow actually delivers flux: constant_velocity takes the face-normal speed from the component named after the face's own axis (vz on a -Zeta inlet), whatever the Cartesian direction of the inlet, and a speed given in another component is ignored.

**u_tau disagrees with the stress the flow carries.** In a stationary channel the viscous, resolved and subgrid shear together must fall linearly from the wall stress. A wall model whose u_tau^2 sits well above that line is delivering more stress than the law it reports, and the excess appears as Reynolds shear stress in the outer layer; that was the signature of the cell-integral Werner relation corrected on 2026-10-08 (wmles-channel-retau1000-werner-2026-10-09). On the corrected channel the two agree to about 2%.

11. Related Documentation

Generated by doxygen 1.9.8