Testing, Smoke, and Quality Gates Guide

Table of Contents

• 1. What This Guide Covers
• 2. Quick Smoke Matrix (Manual)
  • 2.1 Command Discovery Smoke
  • 2.2 Config-Only Validation Smoke
  • 2.3 Dry-Run Smoke (No File Writes)
• 3. Code Map (Where Behavior Lives)
• 4. Automated Smoke Tests
• 5. CI Quality Gate Behavior
• 6. Markdown Link Checking
• 7. How to Extend Smoke Coverage

This page explains how to verify the Iteration A CLI features end-to-end, how the smoke tests are structured, and how CI enforces quality gates.

1. What This Guide Covers

Iteration A introduced:

• pic.flow validate
• pic.flow run --dry-run and --format json
• standardized error envelope:
  • ERROR <CODE> | key=<...> | file=<...> | message=<...> | hint=<...>
• CI workflow for CLI smoke tests and markdown link checks

This guide documents:

1. manual smoke commands for users and maintainers,
2. automated smoke tests in tests/test_cli_smoke.py,
3. workflow behavior in .github/workflows/quality.yml,
4. extension patterns when adding future CLI features.

2. Quick Smoke Matrix (Manual)

Run from repository root.

2.1 Command Discovery Smoke

./scripts/pic.flow --help
./scripts/pic.flow run --help
./scripts/pic.flow validate --help

Expected:

• top-level help lists validate,
• run help lists --dry-run and --format {text,json},
• validate help lists file-role flags and --strict.

2.2 Config-Only Validation Smoke

Valid fixture set:

./scripts/pic.flow validate \
  --case tests/fixtures/valid/case.yml \
  --solver tests/fixtures/valid/solver.yml \
  --monitor tests/fixtures/valid/monitor.yml \
  --post tests/fixtures/valid/post.yml \
  --cluster tests/fixtures/valid/cluster.yml \
  --study tests/fixtures/valid/study.yml

Expected:

• exit code 0,
• [SUCCESS] Validation completed for ... file(s).

Invalid fixture example:

./scripts/pic.flow validate \
  --case tests/fixtures/invalid/case_missing_properties.yml \
  --solver tests/fixtures/valid/solver.yml \
  --monitor tests/fixtures/valid/monitor.yml

Expected:

• exit code 1,
• at least one structured error line with ERROR CFG_MISSING_SECTION.

2.3 Dry-Run Smoke (No File Writes)

Human-readable plan:

./scripts/pic.flow run --solve --post-process \
  --case tests/fixtures/valid/case.yml \
  --solver tests/fixtures/valid/solver.yml \
  --monitor tests/fixtures/valid/monitor.yml \
  --post tests/fixtures/valid/post.yml \
  --dry-run

Machine-readable plan:

./scripts/pic.flow run --solve --post-process \
  --case tests/fixtures/valid/case.yml \
  --solver tests/fixtures/valid/solver.yml \
  --monitor tests/fixtures/valid/monitor.yml \
  --post tests/fixtures/valid/post.yml \
  --dry-run --format json

Expected:

• no new run directories/files are created,
• JSON output parses cleanly and includes mode, launch_mode, stages, and artifacts.

3. Code Map (Where Behavior Lives)

Primary implementation in scripts/pic.flow:

• standardized error emitter: emit_structured_error(...)
• CLI usage failure exit code (2): fail_cli_usage(...)
• dry-run plan builder: build_run_dry_plan(args)
• dry-run renderer (text/json): render_run_dry_plan(plan, output_format=...)
• validate command implementation: validate_workflow(args)
• parser composition: build_main_parser()
• dispatch and argument combination checks: dispatch_command(args)

Related docs:

• The Conductor Script: pic.flow
• Common Fatal Errors and Fixes
• Configuration Contract (YAML -> Generated Artifacts -> Runtime)

4. Automated Smoke Tests

Automated smoke tests live in:

• tests/test_cli_smoke.py

Fixture packs:

• valid configs: tests/fixtures/valid/
• invalid configs: tests/fixtures/invalid/

Current test categories:

1. help output smoke,
2. validate pass/fail behavior,
3. structured error presence for invalid inputs,
4. dry-run no-write guarantee,
5. dry-run JSON schema sanity,
6. markdown link checker pass.

Run locally (when pytest is available):

pytest -q

5. CI Quality Gate Behavior

Workflow file:

• .github/workflows/quality.yml

Pipeline steps:

1. checkout,
2. setup Python,
3. install pytest, pyyaml, numpy,
4. run pytest -q,
5. run python scripts/check_markdown_links.py.

If either step fails, the workflow fails and blocks merge (for repositories using required checks).

Separate docs build/mirror workflow:

• .github/workflows/docs.yml
• handles Doxygen generation and docs artifact mirroring.

6. Markdown Link Checking

Link checker script:

• scripts/check_markdown_links.py

Coverage:

• README.md
• docs/**/*.md (excluding generated docs_build/)

It checks local relative links only and intentionally skips:

• http://...
• https://...
• mailto:...
• in-page anchors (#...)

7. How to Extend Smoke Coverage

When adding a new CLI option or subcommand:

1. add a help-output assertion in tests/test_cli_smoke.py,
2. add one success-path and one failure-path test,
3. add/update fixtures under tests/fixtures/,
4. document user-facing behavior in The Conductor Script: pic.flow,
5. update this page’s smoke matrix if the command is user-critical.

When adding new docs pages:

1. add nav entries in docs/DoxygenLayout.xml,
2. run python scripts/check_markdown_links.py,
3. ensure links from at least one entry page (mainpage/conductor/how-to).

9. Common First-Run Issues

Common local issues:

• pytest: command not found:
  • install pytest in your local Python environment,
  • or run tests in CI where dependencies are provisioned.
• No module named pytest:
  • use the Python interpreter/environment intended for development.
• network-restricted environments:
  • local package install may fail; rely on CI run for authoritative smoke status.

1. Reference Scales

• Pytest docs: https://docs.pytest.org/
• GitHub Actions workflow syntax: https://docs.github.com/actions/using-workflows/workflow-syntax-for-github-actions
• CommonMark link syntax reference: https://spec.commonmark.org/