Add integration tests - #3
Merged
Merged
Conversation
Collaborator
Author
|
/ok to test 5da275b |
Collaborator
Author
|
/ok to test b36a1b9 |
Collaborator
Author
|
/ok to test 88ab705 |
Add an idempotent Python driver for the kind, NFS CSI, SSH, MariaDB, and Slinky fixture. Setup and start reconcile a complete running environment. Stop deletes the disposable cluster while preserving reusable host state and external NFS data. Capture the sanitized single-host feasibility study and implementation handoff. Add a bounded test action that builds and validates the deployment tarball, then runs one-node and two-node filesystem sweeps through SSH and Slurm. Run the SSH entry point on the host, and transfer the same archive to the Slinky login node for Slurm execution. Install the standard file utility in the test images, validate generated environments, retain diagnostic logs, and assert results and cleanup. Fix two SSH dispatch variable-shadowing bugs exposed by the regression cases.
Add full integration environment teardown Add an idempotent teardown lifecycle action that removes marker-owned NFS configuration and data while preserving installed tools. Track ownership for host state, loop devices, firewall rules, and locally built image tags. Use full teardown for CI cleanup and document its distinction from disposable stop. Run fixture lifecycle and sweep tests concurrently on amd64 and arm64, with a final job requiring both matrix legs to pass. Exercise idempotent setup and teardown, stop/start recovery, and enforce that sweeps run as the provisioned pre-sudo user instead of root. Validate exact one- and two-node workload metadata and ordered worker selection for SSH and Slurm. Log Slurm's per-cell host subset so the harness can prove prefix ordering. Copy completed results to the host, run extract-elbencho.sh, and require reports to expose both node counts. Document strengthened coverage and resource prerequisites, improve missing-command failure diagnostics, and retain bounded artifacts for troubleshooting.
Add an explicit sbx-shared storage backend alongside the existing NFS backend. Persist backend selection, limit automatic selection to recognized host capability gaps, and refuse backend changes until teardown. Run the SBX profile with the tested kind and Kubernetes versions, conditionally map the missing kmsg device, propagate the Docker SBX proxy CA, and expose a repository-backed directory through static RWX volumes. Prove cross-node and host visibility while retaining the same SSH, Slurm, tarball, sweep, result, and reporting coverage as the NFS profile. Use the digest-pinned upstream Elbencho image when release assets are unavailable in SBX. Package its dynamic runtime only in generated test artifacts, stage it for SSH workers, and keep third-party binaries out of the repository. Accept Elbencho's adjacent streamed JSON records and terminate loader-launched services by their listening port. Harden first-boot Slinky behavior by retrying only the recognized webhook startup race and refreshing the configless login client after accounting is ready. Add the reporting Python prerequisite and prevent shell startup noise from contaminating structured result discovery. Document backend selection, lifecycle and cleanup guarantees, environmental fidelity, and the Docker-SBX-specific compatibility behavior. Document the remaining filesystem sweep coverage gaps and rank future work across environment, command-line, and reporting behavior. Make the Docker SBX profile use a Kubernetes-compatible kubectl and reconcile Slinky values on every setup. Ignore terminating LoginSet pods during rollout and require ownership markers before mutating retained state or export directories. Package Elbencho with its discovered dynamic loader, use explicit SSH identities, and increase the bounded workload enough to produce measurable throughput. Handle valid all-zero scaling reports and add focused reporting and lifecycle regression tests. Restore configured Slurm worker order after allocation canonicalization so ORDER_NODES cell selection remains deterministic.
dbishop
force-pushed
the
add-integration-tests
branch
from
September 20, 2026 02:13
88ab705 to
84dd77f
Compare
Separate substrate and scenario selection, schedule scenarios deterministically, and cache deployment archives from one immutable source snapshot consumed by the existing archive builder. Add crash-recoverable SSH home transitions and fast regression coverage for selection, caching, reconciliation, and extensible assertions.
Exercise bounded Elbencho workloads through real SSH and Slurm paths on both CI architectures. Cover buffered and direct I/O, failure and resume, retained data, live capture, multidimensional sweeps, single-file and weighted-root behavior, shared SSH homes, and Slurm scheduling controls. Stage failure injection independently for each substrate and preserve its one-time marker through resume. Validate execution coordinates, lifecycle state, workload totals, native command semantics, reporting output, and scheduler evidence while allowing extensible artifacts and report fields. Isolate and clean scenario-owned storage, restage the SBX runtime after SSH home transitions, select NFS explicitly in CI, and document the expanded scenario catalog. Add specification and failure-staging regression tests.
Cover node-range parsing, Cartesian reification, configuration precedence, SSH host selection, destructive path guards, single-file constraints, weighted sizing, and Slurm argument boundaries with fast shell tests. Exercise reporting filters, CSV round trips, output modes, and live-report option validation without requiring a running fixture. Fix associative TEST_DIRS detection so legacy TEST_DIR cannot override an explicit map. Reject partially invalid node specifications before any execution is reified, and coerce cached report fields using resolved type annotations so numeric filters work after --from-csv. Make CSV input exclusive with raw result directories and reject malformed or no-match filters to prevent successful but misleading reports. Document the resulting reporting and fast-test contracts.
Keep host operator IDs out of pod-side storage setup and preserve all-squashed NFS ownership. Stage the SSH failure-injection delegate and runtime on the local coordinator so the wrapper can exercise the intended failure and resume lifecycle. Make fixture validation substrate-aware, including Slurm-only report extraction while SSH is unavailable. Remove only harness-owned NFS configuration during teardown and preserve pre-existing services and unrelated exports. Require workload metadata for bounded result contracts and validate aggregate and live report filtering independently. Make deployment cache mode coverage independent of umask and add focused regressions for these lifecycle and reporting boundaries. Document the integration invariants and require repository-pinned Python tooling for authoritative lint and test results. Run integration workloads as fixed non-root identity Provision a fixed tester UID/GID 2000 in the Slinky login and compute images, register its Slurm account, and require both the coordinator and two-node srun fan-out to prove the expected identity. Run Slurm sweeps, PVC staging, cleanup, and SSH worker staging as that account with restrictive umasks. Remove client-side ownership changes that fail against all-squashed NFS exports and keep host account IDs confined to local fixture state. Repair failure injection's cross-host staging location, avoid Slurm indices during SSH-only runs, and ignore terminating Slinky workers during rollout validation. Add sentinel-ID regression coverage, document ownership and teardown contracts, and validate the complete SSH and Slurm scenario catalog. Make Slurm account setup idempotent Query existing Slurm accounts, associations, and user defaults before mutating accounting state. Add only missing records, repair a stale default account, and verify the exact tester/storage-test relationship after reconciliation. Cover retained accounting state with a regression that runs the reconciler twice and proves the second pass performs no mutation. This allows repeated setup to accept the state created by its first pass. Run integration lifecycle as ordinary user Keep kubeconfig, keys, downloaded clients, caches, manifests, logs, and test runs user-owned from creation under the repository state tree. Reject root for every lifecycle action and let the NFS backend invoke sudo only for package installation and host-system operations. Make the Docker SBX path fully unprivileged. Use sticky shared roots instead of host-side chown, isolate Helm state from stale root-owned caches, and remove UID-mapped data through the private Docker engine. Install pinned client copies in the user state tree instead of globally. Run the GitHub lifecycle as the runner account and validate the root rejection boundary. Document the privilege model and cover idempotent state bootstrap, sudo-free SBX prerequisites, and diagnostics. Fix filesystem device validation Compare st_dev values from stat -c %d instead of interpreting statfs free-inode counts as filesystem identities. Apply the correction to SSH scriptlet/direct and Slurm sbatch/srun probes, preserve paths with spaces, and report the actual distinct-from-root contract. Add shell regressions that exercise every dispatch route against distinct and matching device identities. Align design, requirements, and repository context with the validation semantics. Direct agents to wrap future commit messages at about 72 characters. Create default integration state parent Allow the known repository-local state path to create its ignored tmp parent in a fresh checkout before bootstrapping user-owned state. Continue rejecting custom state paths unless their parent already exists, preserving the destructive-cleanup safety boundary. Cover clean-checkout bootstrap and custom-path rejection with focused driver safety tests, and document the distinction. Fix integration scenario validation and teardown Search standard sbin paths so ordinary-user NFS setup can find privileged utilities, and record NFS service ownership before package installation can activate it. Keep cleanup safe when private kind or exportfs clients are absent after interrupted or repeated teardown. Read Slurm worker-order evidence from copied execution logs and validate the real Elbencho binary before installing failure-injection wrappers. Use non-sticky writable SBX fixture directories so replacement pods can clean files when Docker SBX remaps bind-mounted ownership. Add focused regressions for tool lookup, ownership sequencing, copied Slurm evidence, wrapper ordering, SBX modes, and missing-client cleanup. Harden SSH fixture transition convergence Require StatefulSet generation and revision convergence before accepting ready SSH workers. Use unique home and RWX probes, retry bounded cross-node visibility, and clean probes from both pods on every exit. Guard NFS teardown when interrupted package installation leaves valid ownership state without exportfs, and cover both lifecycle boundaries with focused regression tests.
dbishop
force-pushed
the
add-integration-tests
branch
from
September 21, 2026 00:52
3ff260f to
cb8be2f
Compare
Make the single-host fixture safe to rerun, interrupt, and tear down. Use symlink-safe lifecycle and host locks, persist immutable ownership before mutation, reject unowned state, and fail closed on destructive probes. Restore only harness-owned host changes, validate retained cluster profiles, and bound Kubernetes API polling. Unify fixture capacity under a 4 GiB sparse NFS image. Grow retained images and filesystems safely, migrate data transactionally, retry and verify temporary unmounts, and check the actual fixture and Docker backing filesystems. Run eight NFS workers and restore the previous live worker count. Strengthen SSH and Slurm scenarios around failures and timeouts. Cancel and wait for exact Slurm allocations, preserve inherited traps, retry initial Elbencho service startup, and retain failure diagnostics. Protect partial scenario and failure-injection staging, and report cleanup errors without masking primary failures. Use a stable override hook and derive allocation limits from scenario deadlines. Make fixture images reproducible and resilient to registry failures. Verify cached and pulled digests and architectures, retain the official CSI registry fallback, journal fixture tags, and remove stale aliases. Pull pinned MariaDB and Alpine helpers through host Docker with bounded retries, preload private tags, and use Never pull policies for private workload images. Reuse deployment archives by exact content identity while invoking the normal zero-option build_tarball.sh from a disposable source snapshot. Include tracked contents, modes and deletions, the seeded Elbencho runtime, architecture, and recipe in cache invalidation. Isolate normal builder side effects from immutable inputs and reuse verified archives across scenarios and runs. Drive CI through one bounded non-root lifecycle wrapper on amd64 and arm64. Verify repeated setup, stop and restart, root rejection, and backend selection; preserve diagnostics, reserve cleanup headroom, attempt teardown twice, propagate cleanup failures, and support the same entry point with Docker SBX storage. Preserve positive sub-millisecond histogram bounds to avoid invalid log axes and runaway plot rendering. Add regression coverage for ownership, capacity, cache identity, image acquisition, signal handling, failure injection, cleanup, deadlines, and workload image contracts.
dbishop
force-pushed
the
add-integration-tests
branch
from
September 22, 2026 18:46
690ffb2 to
d3ef837
Compare
Pre-acquire digest-pinned kind node images through host Docker before SBX shared-path probes, cluster creation, and SBX cleanup. This keeps cold starts and teardown from bypassing host mirror configuration, digest and architecture checks, or the retry policy. Give image acquisition four bounded attempts with 60-second calls, 10/20/40-second exponential backoff plus jitter, and a shared 240-second deadline. Reserve a full pull-and-inspect window for each later registry so a hanging primary cannot starve the CSI fallback, and bound Docker metadata inspection within the same deadline. Correct the integration guide to document non-sticky SBX mode 0777 and its disposable-directory tradeoff. Describe the deployment cache's zero-argument builder and fixed recipe, and condense repository context around the durable fixture invariants. Add ordering and fake-clock coverage for cold kind pulls, direct SBX consumers, global deadlines, and fallback reservation.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First stab at both integration tests and trying to run them in our GitHub runners.