|
PICurv 0.1.0
A Parallel Particle-In-Cell Solver for Curvilinear LES
|
Lifecycle-aware archival, offload, verification, and restore workflows. More...
Go to the source code of this file.
Data Structures | |
| class | picurv_cli.storage.StorageError |
| User-facing storage workflow failure. More... | |
Namespaces | |
| namespace | picurv_cli |
| namespace | picurv_cli.storage |
Functions | |
| str | picurv_cli.storage._utc_now () |
| Return a stable UTC timestamp for storage metadata. | |
| str | picurv_cli.storage._human_bytes (int value) |
| Format a byte count for concise command output. | |
| str | picurv_cli.storage._sha256_file (str path) |
| Calculate SHA-256 without loading a potentially large file into memory. | |
| None | picurv_cli.storage._atomic_write_json (str path, dict payload) |
| Atomically replace a JSON state or manifest file. | |
| picurv_cli.storage._read_json (str path) | |
| Read a JSON mapping when present, otherwise return None. | |
| picurv_cli.storage._find_upwards (str start, str filename) | |
| Find the nearest named file at or above a filesystem anchor. | |
| str | picurv_cli.storage.resolve_storage_config_path (str explicit_path=None, bool require=True) |
| Resolve an explicit or nearest workspace storage configuration. | |
| dict | picurv_cli.storage.load_storage_profile (str profile_name=None, str config_path=None) |
| Load and validate one non-secret rclone storage profile. | |
| str | picurv_cli.storage._remote_join (str remote, *str parts) |
| Join path components without corrupting rclone remote syntax. | |
| str | picurv_cli.storage._object_remote (dict profile, str archive_id, *str parts) |
| Return the remote path for one immutable archive object. | |
| subprocess.CompletedProcess | picurv_cli.storage._run_rclone (list arguments, bool check=True) |
| Invoke rclone through the same argv-based subprocess boundary as other PICurv tools. | |
| str | picurv_cli.storage._remote_sha256 (str remote_path) |
| Ask rclone to calculate or retrieve the SHA-256 of one remote object. | |
| dict | picurv_cli.storage._upload_verified (str local_path, str remote_path) |
| Upload one file, then verify its remote SHA-256. | |
| bytes | picurv_cli.storage._read_remote_bytes (str remote_path) |
| Read a small remote catalog object through rclone. | |
| dict | picurv_cli.storage._load_remote_manifest (dict profile, str archive_id, bool require_complete=True) |
| Fetch and validate one versioned remote storage manifest. | |
| str | picurv_cli.storage._state_path (str root_path) |
| Return the local storage state marker path for an artifact root. | |
| picurv_cli.storage.read_storage_state (str root_path) | |
| Read the nearest applicable storage marker for a run, study, or study member. | |
| bool | picurv_cli.storage.is_artifact_cold (str root_path) |
| Return whether a local artifact marker says payload data was pruned. | |
| list | picurv_cli.storage.cold_study_members (str study_path) |
| Return numbered study members whose local payload was pruned. | |
| None | picurv_cli.storage.require_storage_payload_local (str root_path, str operation, int checkpoint=None, checkpoints=None) |
| Reject a workflow that requires payload currently held in cold storage. | |
| dict | picurv_cli.storage.storage_state_summary (str root_path) |
| Return a compact storage status for summarize and status commands. | |
| str | picurv_cli.storage._validate_case_id (str case_id) |
| Validate a canonical numbered study-member identifier. | |
| list | picurv_cli.storage.resolve_local_storage_targets (str run_dir=None, str study_dir=None, case_ids=None) |
| Resolve explicit run/study selectors into concrete artifact descriptions. | |
| bool | picurv_cli.storage._path_is_within (str root, str candidate) |
| Return whether an absolute candidate remains within a root directory. | |
| str | picurv_cli.storage._resolve_configured_path (str root, str value) |
| Resolve runtime-directory syntax using the run directory as working directory. | |
| list | picurv_cli.storage._artifact_runtime_roots (str root) |
| Return run-like roots contained by a standalone run or whole study. | |
| list | picurv_cli.storage._discover_external_paths (str root) |
| Report configured data paths that escape the archived directory boundary. | |
| list | picurv_cli.storage._discover_dependencies (str root) |
| Discover absolute restart/source paths embedded in generated controls. | |
| list | picurv_cli.storage._walk_archive_entries (str root) |
| Enumerate archive entries without following symlinks. | |
| picurv_cli.storage._checkpoint_component (str relative_path) | |
| Return a checkpoint component token for a path inside a committed step bundle. | |
| str | picurv_cli.storage._classify_component (str relative_path) |
| Classify one artifact path for packaging and local retention. | |
| list | picurv_cli.storage._checkpoint_steps (list entries) |
| Return committed checkpoint steps represented in an inventory. | |
| list | picurv_cli.storage._find_incomplete_checkpoints (list entries) |
| Return incomplete checkpoint paths that make archival unsafe. | |
| bool | picurv_cli.storage._lock_owner_active (str metadata_path) |
| Conservatively determine whether a solver/post/storage owner marker is active. | |
| set | picurv_cli.storage._collect_job_ids (payload) |
| Recursively collect submitted Slurm job IDs from scheduler metadata. | |
| dict | picurv_cli.storage._slurm_activity (str root) |
| Query live Slurm state for every job recorded below an artifact scheduler directory. | |
| dict | picurv_cli.storage.inspect_artifact (dict target, bool query_scheduler=True) |
| Build a read-only inventory and lifecycle assessment for one artifact. | |
| None | picurv_cli.storage._assert_archive_safe (dict inventory) |
| Refuse to package a changing or scheduler-ambiguous artifact. | |
| picurv_cli.storage.storage_operation_lock (str root_path, str operation) | |
| Hold an exclusive local storage-operation marker for one artifact. | |
| picurv_cli.storage.runtime_stage_lock (str root_path, str stage) | |
| Mark a locally executed solver/post stage as active for storage safety. | |
| str | picurv_cli.storage._select_compression (str requested, int total_bytes, dict profile) |
| Resolve automatic or configured compression policy. | |
| str | picurv_cli.storage._chunk_extension (str compression) |
| Return the archive suffix for a compression policy. | |
| list | picurv_cli.storage._build_chunk_specs (dict inventory, int chunk_size_bytes) |
| Group archive entries into independently transferable component chunks. | |
| str | picurv_cli.storage._safe_component_name (str component) |
| Convert a component token into a portable archive filename fragment. | |
| None | picurv_cli.storage._write_tar_chunk (str root, dict spec, str destination, str compression) |
| Package explicitly inventoried entries without following symlinks. | |
| list | picurv_cli.storage._capture_study_context (dict target) |
| Embed small study control-plane files with an individually archived member. | |
| dict | picurv_cli.storage._config_fingerprints (str root) |
| Hash small canonical YAML inputs stored under an artifact. | |
| dict | picurv_cli.storage._git_provenance (str root) |
| Record best-effort current source revision and dirty state. | |
| dict | picurv_cli.storage._parse_tags (raw_tags) |
| Parse repeatable KEY=VALUE tags into deterministic metadata. | |
| dict | picurv_cli.storage.build_storage_plan (dict target, dict profile, str compression=None) |
| Build the read-only plan consumed by protect and offload. | |
| None | picurv_cli.storage._render_plan (dict plan) |
| Print a concise archive/offload plan. | |
| dict | picurv_cli.storage.archive_artifact (dict target, dict profile, str label=None, tags=None, str compression=None, bool prune_local=False) |
| Package, upload, verify, register, and optionally prune one artifact. | |
| None | picurv_cli.storage._prune_archived_payload (str root, dict inventory) |
| Remove only verified heavy payload while retaining control-plane files. | |
| list | picurv_cli.storage.list_remote_manifests (dict profile) |
| Enumerate completed archive manifests from the remote catalog. | |
| dict | picurv_cli.storage.verify_remote_archive (dict profile, str archive_id) |
| Verify the completion marker and every stored chunk checksum. | |
| None | picurv_cli.storage._validate_tar_members (tarfile.TarFile archive) |
| Reject archive members that could escape the restore destination. | |
| None | picurv_cli.storage._extract_chunk (str path, str destination) |
| Safely extract one verified tar chunk into a staging tree. | |
| None | picurv_cli.storage._restore_study_context (dict manifest, str case_destination) |
| Recreate missing study control-plane files around a restored member. | |
| None | picurv_cli.storage._merge_tree (str source, str destination) |
| Merge a verified restore tree into a known cold artifact skeleton. | |
| list | picurv_cli.storage._rebase_restored_text_paths (str root, list replacements) |
| Rebase known generated text artifacts after an explicit relocated restore. | |
| dict | picurv_cli.storage.restore_archive (dict profile, str archive_id, str destination=None, checkpoints=None, bool force=False) |
| Download, verify, extract, and materialize an archive or selected checkpoints. | |
| str | picurv_cli.storage._resolve_archive_id_from_args (args) |
| Resolve an explicit archive ID or a local artifact marker. | |
| None | picurv_cli.storage._render_status (dict inventory) |
| Render one artifact status row and safety details. | |
| None | picurv_cli.storage.storage_setup_workflow (args) |
| Create or update a non-secret workspace storage profile. | |
| None | picurv_cli.storage.storage_status_workflow (args) |
| Print local storage and lifecycle status for selected artifacts. | |
| None | picurv_cli.storage.storage_plan_workflow (args) |
| Render a read-only packaging and safety plan. | |
| None | picurv_cli.storage.storage_archive_workflow (args, bool prune_local) |
| Execute protect or offload for one or more explicit local targets. | |
| None | picurv_cli.storage.storage_restore_workflow (args) |
| Restore a remote archive by globally unique ID or local marker. | |
| None | picurv_cli.storage.storage_verify_workflow (args) |
| Verify all remote chunks for one archive. | |
| None | picurv_cli.storage.storage_list_workflow (args) |
| Search the remote manifest catalog without local artifact state. | |
| None | picurv_cli.storage.storage_show_workflow (args) |
| Show one complete remote archive manifest. | |
| None | picurv_cli.storage.storage_workflow (args) |
| Dispatch nested storage actions using existing PICurv workflow conventions. | |
| argparse.ArgumentParser | picurv_cli.storage.add_storage_parser (subparsers) |
| Attach the nested storage command parser to PICurv's top-level parser. | |
Lifecycle-aware archival, offload, verification, and restore workflows.
The storage layer deliberately remains independent of the numerical runtime. It packages immutable run/study artifacts, uses rclone as a transport boundary, and leaves a small local state marker whenever payload data is pruned.
Definition in file storage.py.
1.9.8