PICurv 0.1.0
A Parallel Particle-In-Cell Solver for Curvilinear LES
 
Loading...
Searching...
No Matches
Testing and Validation Guide

This page documents PICurv's local testing model. The suite is intentionally split by intent so users can choose the smallest command that answers the question they actually have.

1. Testing Taxonomy

PICurv exposes four validation layers plus aggregate gates:

  1. Python/control-plane validation
  2. installation and PETSc provisioning validation
  3. isolated C unit/component validation
  4. executable entrypoint smoke validation
  5. full local validation sweep
  6. comprehensive MPI-inclusive validation sweep

Canonical commands:

  • make test-python
  • make coverage-python
  • make coverage-c
  • make coverage
  • make audit-build
  • make doctor
  • make unit
  • make unit-geometry
  • make unit-setup
  • make unit-solver
  • make unit-particles
  • make unit-io
  • make unit-logging
  • make unit-post
  • make unit-grid
  • make unit-metric
  • make unit-boundaries
  • make unit-poisson-rhs
  • make unit-runtime
  • make unit-simulation
  • make unit-mpi
  • make unit-periodic
  • make smoke
  • make smoke-mpi
  • make smoke-mpi-matrix
  • make smoke-stress
  • make smoke-periodic
  • make check
  • make check-mpi
  • make check-mpi-matrix
  • make check-full
  • make check-stress
  • make install-git-hooks

Compatibility aliases remain available:

  • make test -> make test-python
  • make install-check -> make doctor
  • make ctest* -> make unit*

2. Quick Start

Fast local checks:

python3 tests/tooling/audit_function_docs.py
python3 tests/tooling/audit_starter_content.py
make test
make coverage-python
make audit-build
make doctor
make unit-setup
make unit-simulation
make unit-io
make unit-runtime
make unit-mpi
make smoke
make smoke-periodic
make check
make check-full

Guidance:

  • Use python3 tests/tooling/audit_function_docs.py when changing C/Python function signatures, docstrings, or test helpers. It scans production code, generators, tests, and tooling; it rejects missing documentation and name-only descriptions such as “helper function” or “internal helper implementation.”
  • Use make test when working on picurv_cli/core.py, schemas, or repository metadata.
  • Use python3 tests/tooling/check_markdown_links.py when changing any Markdown in the repository; it scans every tracked Markdown file, not just docs/ and examples/.
  • Use python3 tests/tooling/audit_starter_content.py when changing examples/ or <repo>/config/. It checks the complete top-level template and configuration inventories, validates every declared composition, and verifies that picurv init faithfully copies each template without carrying a site-specific execution sample into the new case.
  • Use make audit-build when you want a clean compilation audit with <repo>/logs/build.log and <repo>/logs/build.warnings.log captured under <repo>/logs/.
  • Use make doctor after provisioning PETSc on a new machine.
  • Use make unit-setup when changing setup, teardown, initialization, or rank-info lifecycle code.
  • Use make unit-setup when changing FieldId, FieldDescriptor, runtime field views, or the typed ghost-update interface; it directly checks catalog completeness, bindings, failure behavior, and representative layouts.
  • Use make unit-simulation for the normal simulation-core debugging loop (unit-boundaries + unit-solver + unit-poisson-rhs + unit-runtime + unit-particles).
  • Use make unit-<area> while changing a subsystem in isolation.
  • Use make smoke after building binaries to execute tiny real solve/post/restart workflows.
  • Use make smoke-mpi-matrix when you need a rank-sweep MPI runtime sanity check.
  • Use make smoke-stress when you want the opt-in medium-budget runtime extension tier.
  • Use make unit-periodic and make smoke-periodic when changing geometric-periodic Eulerian behavior; both are included in the standard gates.
  • Use make coverage to enforce line-coverage floors for core Python scripts and C sources.
  • Use make check as the pre-merge gate at the end of a development cycle.
  • Use make check-mpi when multi-rank MPI behavior is in scope.
  • Use make check-full for comprehensive branch/CI/release validation that must cover all MPI layers.
  • Use make check-stress when you want the full default gate plus the stress tier in one pass.
  • Use make audit-agent-setup after changing shared instructions, skill discovery, or Claude-local configuration. Edit canonical skills under .agents/skills/, then use make sync-agent-skills to refresh and audit the materialized Claude copies.
  • Run make install-git-hooks once per clone to make make certify-docs an automatic pre-push gate for updates to remote main. The hook validates only the checked-out commit and blocks the push on failure; GitHub Actions then repeats the lighter structural documentation checks after the push.

2.1 Review Routing and Optional Source Xrefs

make review-packet is a bounded index into declared repository data. Its selectors are mutually exclusive:

make review-packet PAGE=44
make review-packet CONTRACT=field.identity_and_layout
make review-packet CAPABILITY=boundary.handler
make review-packet SUBSYSTEM=boundary.periodic
make review-packet SURFACE=boundary.system
make review-packet CHANGED=working-tree

The first five join registry identifiers to source symbols or watched files, pages, contracts, capability families, subsystem records, freshness state, and declared test evidence. Changed-set mode covers staged, unstaged, and untracked nonignored paths. Each routed production path lists the make targets its narrowest owner declares as evidence: the capability families that name the path, else the subsystem its freshness surface names, else every routed subsystem. An unrouted production path returns advisory status 3 with a nearest-guide fallback and a targeted search. Status 0 means the join is complete over declared registry data, not that code behavior is correct. Unknown identifiers return 2; known but unresolved or environment-unavailable routes return 3 instead of an empty success.

make docs-xref optionally adds docs_build/xref.json, distilled from Doxygen XML and stamped with current source bytes and Doxygen configuration. Review packets never build it implicitly and never display stale edges. Doxygen references are not a semantic call graph: callbacks, registry tables, function pointers, macros, PETSc dispatch, and runtime selection can require an intermediate or manual live-path trace. The registry route is usable on a fresh clone without this cache.

2.2 Main-Branch Pre-Push Certification

The tracked .githooks/pre-push hook enforces a commit-scoped documentation certificate before a local Git client updates refs/heads/main. Enable it in each clone with:

make install-git-hooks

Every update to main is also a release boundary. The repository-root VERSION file contains the semantic release version shared by the CLI, simulator, and postprocessor. Before allowing a main-branch push, the hook requires that the pushed range changes both VERSION and docs/CHANGELOG.md, that the version differs from the remote main version, and that the changelog contains a matching ## <version> heading. Feature-branch commits do not each need a version bump; the branch receives one coherent version and changelog entry when it is prepared for main. A concise commit-message summary is enough for the changelog when it accurately describes the user-visible change.

The hook runs make certify-docs, which requires a clean worktree and includes the PETSc/MPI runtime tier, whenever runtime-relevant files changed. For Markdown, documentation-site, hook, and workflow-only changes it instead runs make certify-docs-fast only if a full certificate for an ancestor of the target commit exists in local <repo>/logs/ and is no more than three days old. Missing or stale certificates always cause a full run. Override the interval for a single push with PICURV_FULL_CERT_MAX_AGE_DAYS=<days>.

The hook deliberately rejects a push that maps some other local commit directly to main, because it can only certify the exact checked-out HEAD. This is a local safeguard: Git permits bypassing hooks with git push --no-verify, so the post-push GitHub documentation gates remain a separate redundancy layer.

make certify-docs includes portable-agent-setup, Markdown-link, function-documentation, user-facing-reporting, starter-template/configuration, option-ingress, example/configuration regression, and zero-warning Doxygen gates before the PETSc/MPI make check-full tier. Therefore, with the tracked hook enabled, a normal push to main cannot publish the checked-out commit until all applicable local gates pass. The starter-content gate specifically rejects an unregistered example directory/YAML, an unregistered <repo>/config/ asset, a broken declared composition, or a template whose canonical relocation by picurv init loses or rewrites starter content incorrectly.

3. Python Suite (<tt>test-python</tt>)

Purpose:

  • validate picurv_cli/core.py
  • validate config translation and contract behavior
  • validate repository-level regression checks

This suite is implemented with pytest and does not require PETSc.

Current coverage includes:

  • CLI help/validate/dry-run smoke
  • config regression checks
  • case maintenance regressions
  • repository consistency scans
  • function-level documentation contract checks for C and Python executable code

For newly added or modified C test helpers, keep Doxygen blocks concise but descriptive: the @brief should state what the test or helper verifies rather than using a placeholder label.

Run locally:

make test-python

CI note:

3.1 Python Test File Matrix

Current Python files and primary responsibilities:

  • tests/test_cli_smoke.py
    • CLI help and argument contract checks
    • dry-run JSON schema assertions
    • staged Slurm workflow coverage for submit, cancel, and sweep
    • summarize configuration/health JSON/text output, time-history discovery/plot orchestration, standalone plot.gen, and failure-path checks
    • restart resolution/pathing checks
    • cluster no-submit manifest and sbatch-rank checks
    • grid-gen header/node-count checks
  • tests/test_case_maintenance.py
    • case-origin metadata behavior
    • real CLI wrapper coverage for build, sync-config, status-source, and pull-source
    • sync/pull/build source-root resolution behavior
    • template sync behavior (overwrite, prune)
    • status drift report behavior
  • tests/test_config_regressions.py
    • ingress-manifest drift checks
    • post-recipe alias compatibility checks
    • post-task validation guards and stats artifact prediction
  • tests/test_repo_consistency.py

These files are intentionally role-specific to keep failures actionable.

4. Installation Validation (<tt>doctor</tt>)

Purpose:

  • confirm PETSC_DIR is present and usable
  • confirm the configured toolchain can compile against PETSc
  • confirm a minimal PETSc-backed binary can initialize and create core PETSc objects

The doctor target builds and runs a small C smoke binary under tests/c/test_install_check.c.

It validates:

  • named case environment-visible for PETSC_DIR visibility
  • named case basic-petsc-objects for PetscInitialize, DMDA, Vec, DMSwarm, and PetscFinalize

This target answers: "Is this machine set up correctly for PICurv development?"

Run locally:

make doctor

5. Isolated C Unit Suites (<tt>unit-*</tt>)

Purpose:

  • support kernel and component development in isolation
  • reduce the blast radius while changing numerical code

Current suites:

  • make unit-geometry
  • make unit-setup
  • make unit-solver
  • make unit-particles
  • make unit-io
  • make unit-post
  • make unit-grid
  • make unit-metric
  • make unit-boundaries
  • make unit-poisson-rhs
  • make unit-runtime
  • make unit-mpi
  • make unit (all of the above)

Suite focus areas:

  • setup, initialization, and cleanup lifecycle coverage
  • geometry and interpolation invariants
  • solver utility kernels
  • particle location helpers
  • I/O pathing and field round-trips
  • logging contracts (level/allow-list/console snapshot cadence)
  • post-processing/statistics kernels and orchestration
  • grid decomposition and bounding-box exchange
  • metric tensor/face metric kernels
  • boundary factory/face-service checks
  • Poisson/RHS and diffusivity kernels
  • runtime helper kernels (initialization, particle helpers, wall models)

These tests use real PETSc objects and run single-rank by default. unit-mpi is the dedicated multi-rank suite (default TEST_MPI_NPROCS=2).

5.1 C Test File Matrix

Current C test files and their main purpose:

6. Executable Smoke (<tt>smoke</tt>)

Purpose:

  • confirm the compiled binaries still launch successfully
  • catch integration breakage that isolated unit tests may miss

The current smoke runner verifies:

  • bin/simulator launches and responds to -help
  • bin/postprocessor launches and responds to -help
  • bin/picurv init creates a case directory with origin metadata (no binary copies)
  • template matrix init/validate/dry-run checks across flat_channel, bent_channel, and brownian_motion
  • picurv run --dry-run --format json emits a valid solve/post execution plan
  • restart dry-run planning resolves --restart-from into the expected restart source directory
  • tiny end-to-end solve + post run (flat channel)
  • tiny end-to-end solve + post run (bent channel)
  • tiny end-to-end particle solve + post run (flat channel, default Trilinear interpolation)
  • tiny end-to-end particle solve + post run with corner-averaged interpolation (legacy regression)
  • restart execution with both particle modes (load and init)
  • restart-equivalence check for flat channel (continuous tiny run vs split restart tiny run continuity metric agreement)
  • tiny end-to-end analytical Brownian run with particle VTP + MSD CSV output
  • the examples/search_robustness/ family provides a dedicated end-to-end runtime characterization path for search and migration observability
  • multi-rank tiny solve + post runs for flat and bent channels (make smoke-mpi)
  • multi-rank flat particle runtime + restart (load/init) runs (make smoke-mpi)
  • rank-matrix MPI runtime sweep across flat+bent+flat-particle-restart (make smoke-mpi-matrix)
  • opt-in stress extensions for longer particle cycling, chained restarts, parabolic-inlet runtime coverage, periodic constant-flux validate/dry-run coverage, and extra-rank MPI particle runs (make smoke-stress)
  • geometric-periodic real runtime harness (make smoke-periodic, included in check)

These checks are intentionally tiny but execute real solver/postprocessor runtime paths. For the -help smoke checks, banner presence is authoritative: the local PETSc debug build may exit with code 62 (PETSC_ERR_ARG_WRONG) after printing help, and that still counts as a pass.

Run locally:

make smoke
make smoke-mpi
make smoke-mpi-matrix
make smoke-stress
make smoke-periodic

6.1 Useful Smoke Knobs

Environment controls used by the smoke layer:

  • KEEP_SMOKE_TMP=1:
    • preserves the smoke temporary workspace for post-failure inspection.
  • SMOKE_MPI_NPROCS=<n>:
    • rank count for make smoke-mpi.
  • SMOKE_MPI_MATRIX_NPROCS="<n1> <n2> ...":
    • rank matrix for make smoke-mpi-matrix.

Smoke orchestration lives in tests/smoke/run_smoke.sh; runtime profile mutation helpers there are part of the tested contract.

7. Aggregate Validation Gates (<tt>check*</tt>)

make check is the top-level local validation sweep.

It runs, in order:

  1. make test-python
  2. make doctor
  3. make unit
  4. make smoke

Use it when you want maximum local confidence before ending a development cycle.

make check-mpi extends this sweep by running make smoke-mpi and make unit-mpi afterward.

make check-mpi-matrix extends this sweep by running make smoke-mpi-matrix and make unit-mpi.

make check-full is the comprehensive MPI-inclusive gate. It runs make check, then make unit-mpi, make smoke-mpi, and make smoke-mpi-matrix.

make check-stress extends make check-full by running make smoke-stress afterward. make smoke-periodic runs as part of make check; make unit-periodic runs as part of make unit.

Use check-full for release candidates or CI workflows where both focused multi-rank tests and rank-matrix runtime checks are required in a single pass.

8. Coverage Gate Implementation Notes

Coverage gates are script-backed and checked in-repo:

Coverage artifacts:

  • Python summary: coverage/python/summary.txt
  • C summary: coverage/c/summary.txt

8.1 Coverage Follow-Up Snapshot (2026-07-31)

This dated snapshot records non-periodic C sources that still look thinly exercised after the 2026-07-31 local audit. Treat these numbers as test-readiness guidance for future work, not as evidence of confirmed product defects.

  • src/les.c: 29.46%. Add a direct dynamic-Smagorinsky fixture that executes the filtered-stress procedure, rather than only the constant-model and eddy-viscosity paths.
  • src/poisson.c: 47.61%. Add multigrid-cycle, periodic, and driven-flux correction cases.
  • src/momentum_newton_krylov.c: 62.50%. The focused residual/constraint/Jacobian/rollback suite is strong; extend it across less-common solver-option and nonlinear-error branches.
  • src/runloop.c: 66.46%. Add controlled live-run coverage for signal receipt and automatic walltime-guard shutdown/restart output.
  • src/postprocessor.c: 69.42%. Add malformed/empty/unknown pipeline-stage and optional-field combination coverage.

Next test work priorities from this snapshot:

  1. dynamic LES and Poisson multigrid/stencil cases,
  2. runtime signal and walltime-guard integration coverage,
  3. postprocessor pipeline error-path coverage,
  4. remaining Newton–Krylov option/error-path coverage.

9. Common Failure Modes

Common issues:

  • PETSC_DIR is unset:
    • make doctor, make unit*, make smoke*, and make check* will fail.
  • PETSC_ARCH points to the wrong build:
    • compilation may fail while including PETSc make variables.
  • mpicc or MPI runtime wrappers are missing:
    • PETSc-backed builds may fail even if Python tests still pass.
  • a unit test fails after writing temporary files:
    • inspect /tmp/picurv-test-* for preserved intermediate artifacts.
  • pytest is unavailable:
    • only the Python suite is blocked; PETSc-backed C tests are separate.
  • gcov is unavailable:
    • make coverage-c cannot produce C line-coverage reports.

10. Extending The Test Suite

When adding new tests:

  1. choose the narrowest valid layer (test-python, doctor, unit-*, or smoke)
  2. reuse tests/c/test_support.* for PETSc-backed fixtures; it now provides both a fast minimal fixture and a richer tiny-runtime fixture while mirroring the production da/fda/swarm contract instead of a same-size synthetic DM
  3. keep temporary artifacts under /tmp
  4. document new user-facing targets or workflows in the README and this guide

For detailed C test maintenance guidance, see C Test Suite Developer Guide.

11. Runtime Coverage Map

The smoke suite uses these named runtime sequences:

  • S0: template matrix init/validate/dry-run across flat_channel, bent_channel, and brownian_motion
  • S1: tiny flat-channel solve+post run
  • S1b: tiny bent-channel solve+post run
  • S2: tiny flat-channel solve+post with particles enabled (default Trilinear interpolation)
  • S2b: tiny flat-channel particle solve+post with CornerAveraged interpolation (legacy path regression)
  • S3: tiny restart runs from S2 with particle_restart_mode=load and particle_restart_mode=init
  • S4: tiny Brownian analytical solve+post with particle outputs and MSD statistics
  • S5: multi-rank tiny solve+post runs for flat and bent channels, plus flat particle base/restart (load and init) branches
  • S6: restart-equivalence run for flat channel (continuous vs split restart continuity metric agreement)
  • S7: periodic constant-flux validate + dry-run contract coverage in the opt-in stress tier
  • S8: geometric-periodic real runtime harness (make smoke-periodic, gating)

Runtime file coverage map (unit targets + runtime sequences):

12. Exhaustive-Readiness Backlog

P0 (implemented):

  • coverage gates: make coverage-python, make coverage-c, make coverage
  • restart-equivalence runtime smoke (S6)
  • rank-sweep MPI runtime smoke (make smoke-mpi-matrix)

P1 (implemented):

  • broadened MPI matrix to particle/restart branches
  • dedicated setup/runloop orchestration branch tests in unit-runtime
  • boundary-condition matrix expansion for handler/face/orientation combinations in unit-boundaries

P1 (next):

  • direct walking-search branch pinning for LocateParticleOrFindMigrationTarget:
    • boundary clamp
    • ghost-region handoff
    • tie-breaker
    • LOST and MIGRATING_OUT outcomes
  • explicit direction-complete and failure-path coverage for the GuessParticleOwnerWithBBox heuristic
  • non-restart MPI particle-migration tests for multi-pass handoff, newcomer flagging, and count conservation
  • direct positive-path momentum harnesses:
    • MomentumSolver_Explicit_RungeKutta4
    • one small invariant case for MomentumSolver_DualTime_Picard_JamesonRK
  • a direct periodic-stencil check for AssemblePoissonOperator/ProjectVelocity; the periodic branches are exercised end to end, not by a unit test
  • broader richer-runtime fixture variants so grid/setup/metric tests cover more than the tiny Cartesian baseline
  • broaden the MPI rank matrix to larger optional decompositions (for example SMOKE_MPI_MATRIX_NPROCS="2 3 4 6") in CI/nightly profiles
  • coverage follow-up from the 2026-07-31 audit: dynamic LES, Poisson multigrid/stencils, live walltime/signal shutdown, postprocessor pipeline errors, and remaining Newton–Krylov option/error paths (see section 8.1 for measured coverage and scope).

P2 (deeper hardening):

  • long-duration nightly/weekly stability suites (beyond tiny smoke budgets)
  • numerical oracle/golden-output tolerance checks for more physical scenarios
  • performance regression gates (runtime/memory envelopes)