Skip to content

feat(preflight): add preflight validation role - #107

Merged
tech2734 merged 15 commits into
redhat-cop:v2from
vvaldez:preflight-role
Oct 16, 2026
Merged

tech2734 merged 15 commits into
redhat-cop:v2from
vvaldez:preflight-role

Conversation

@vvaldez

@vvaldez vvaldez commented Oct 12, 2026 •

Copy link
Copy Markdown

What

Adds a preflight role and vmf_preflight.yml playbook that validate the migration environment before seeding AAP:

  • OpenShift API authentication — SelfSubjectReview probe with the configured credential
  • OpenShift Virtualization (CNV) and MTV operator health — CSV phase checks
  • AAP access: /api/controller/v2/me/ probe (path configurable for AAP 2.4 and earlier)
  • vCenter connectivity and authentication — datacenter listing (vmware sources only)
  • VDDK init image configuration — non-empty and scheme-less (vmware sources only)

A single summary report fails the play if any check failed, so one run reports everything that needs attention.

Why

The migration playbooks currently fail late and deep — for example, a scheme'd VDDK image reaches the provider CR and Forklift rejects it with VDDKInitImageUnavailable two stages later. Preflight validates the environment up front with an actionable message per check.

Design notes

  • Credentials cascade from the same env-var / host-var chains as the existing roles (K8S_AUTH_API_KEY, VMWARE_HOST, AAP_*), falling back to the temporary API key alternative documented in the sample inventory. Namespace variables resolve through the target cluster host as well as extra vars.
  • Credential-bearing endpoints are rejected at pre-validation unless they use https and do not embed user:pass@.
  • no_log is controlled by preflight_secure_logging (default true) on all credential-bearing probes; hostnames and usernames appear in success/detail messages, never secrets.
  • The summary asserts only over the checks that ran, so isolated --tags debug runs end green when the targeted check passed. A source whose type is not vmware skips the vCenter and VDDK checks.

Testing

  • ansible-lint (production profile) passes clean across the role, playbook, and test files.
  • New preflight_report integration test (tests/integration/targets/preflight_report/, molecule scenario integration_preflight_report) covers the report summary paths: isolated tagged runs, all-passed full runs with mixed boolean forms, and a failed full run whose summary marks each failed check.
  • The integration target includes role tasks via FQCN include_role, which resolves under ansible-test's temp-dir layout. The recap shows ok=18 where include_tasks runs showed 24; coverage is unchanged (eight cases, three rescues, two skips).
  • Changelog fragment added.

@vvaldez
vvaldez requested a review from sabre1041 as a code owner October 12, 2026 16:33
@vvaldez
vvaldez deployed to external-ci October 12, 2026 16:34 — with GitHub Actions Active
Validates the migration environment before seeding AAP: OpenShift API
authentication, CNV and MTV operator CSV health, AAP access, vCenter
connectivity, and VDDK init image configuration. All checks run and
report a summary; the play fails if anything failed.

Connection variables cascade from AAP-injected environment variables to
inventory host variables resolved through preflight_target_name,
preflight_source_name, and preflight_aap_name, so the role runs
identically from an AAP job template and from a controller with the
sample inventory.

Automates the manual "Validate your environment" console checks from
the journey documentation. Includes the vmf_preflight.yml playbook and
an antsibull changelog fragment.
- Detect k8s probe failure via a missing resources key — failed_when:
  false clears the failed flag on the registered result, so the
  probe-failure detail must key off resources absence
- Add preflight_aap_api_path option: AAP 2.5 platform deployments serve
  the controller API at /api/controller/v2/; the default /api/v2/ 404s
  there (verified live)
- Assert only over checks that ran (ran flag on record tasks) so
  isolated --tags debug runs end green when the targeted check passed
- Cascade openshift_temporary_api_key and cluster-host cnv/mtv
  namespaces from the target host; pass convert_bool to the falsy
  filter so string-False defaults are caught
- Reject plaintext credential-bearing endpoints in pre-validation
- Add integration test scaffolding (tests/integration/targets,
  extensions/molecule)
@vvaldez
vvaldez changed the base branch from main to v2 October 12, 2026 16:58
@vvaldez vvaldez closed this Oct 12, 2026
@vvaldez vvaldez reopened this Oct 12, 2026
@vvaldez
vvaldez deployed to external-ci October 12, 2026 17:40 — with GitHub Actions Active
@vvaldez
vvaldez deployed to external-ci October 13, 2026 07:35 — with GitHub Actions Active
Comment thread roles/preflight/defaults/main.yml
Comment thread roles/preflight/defaults/main.yml
Comment thread roles/preflight/tasks/validate_aap.yml
Comment thread roles/preflight/tasks/validate_vcenter.yml Outdated
Comment thread roles/preflight/tasks/report.yml
@vvaldez
vvaldez deployed to external-ci October 14, 2026 18:11 — with GitHub Actions Active
@vvaldez
vvaldez requested a review from tech2734 October 14, 2026 18:14

@sabre1041 sabre1041 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good start to a preflight check list. A few items:

  1. Running to the following issue as it relates to verifying vmware
    "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1792093923.5295908-74-157229115925727/AnsiballZ_vcenter_datacenter_info.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1792093923.5295908-74-157229115925727/AnsiballZ_vcenter_datacenter_info.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1792093923.5295908-74-157229115925727/AnsiballZ_vcenter_datacenter_info.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.vmware.vmware_rest.plugins.modules.vcenter_datacenter_info', init_globals=dict(_module_fqn='ansible_collections.vmware.vmware_rest.plugins.modules.vcenter_datacenter_info', _modlib_path=modlib_path),\n  File \"<frozen runpy>\", line 226, in run_module\n  File \"<frozen runpy>\", line 98, in _run_module_code\n  File \"<frozen runpy>\", line 88, in _run_code\n  File \"/tmp/ansible_vmware.vmware_rest.vcenter_datacenter_info_payload_zx9mk_bn/ansible_vmware.vmware_rest.vcenter_datacenter_info_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_datacenter_info.py\", line302, in <module>\n  File \"/usr/lib64/python3.12/asyncio/base_events.py\", line 691, in run_until_complete\n    return future.result()\n           ^^^^^^^^^^^^^^^\n  File \"/tmp/ansible_vmware.vmware_rest.vcenter_datacenter_info_payload_zx9mk_bn/ansible_vmware.vmware_rest.vcenter_datacenter_info_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_datacenter_info.py\", line 248, in main\n  File \"/tmp/ansible_vmware.vmware_rest.vcenter_datacenter_info_payload_zx9mk_bn/ansible_vmware.vmware_rest.vcenter_datacenter_info_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_datacenter_info.py\", line 275, in entry_point\n  File \"/usr/local/lib64/python3.12/site-packages/aiohttp/client_reqrep.py\", line 777, in json\n    raise ContentTypeError(\naiohttp.client_exceptions.ContentTypeError: 404, message='Attempt to decode JSON with unexpected mimetype: text/plain; charset=utf-8', url='https://vcenter.apps.ocpv07.rhdp.net/api/vcenter/datacenter'\nUnclosed client session\nclient_session: <aiohttp.client.ClientSession object at 0xffffa49088f0>\nUnclosed connector\nconnections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0xffffa3596e40>, 40212.0821937)])']\nconnector: <aiohttp.connector.TCPConnector object at 0xffffa483ce90>\n",
  1. Whats the best way to ignore certain types of checks. Skipping tags?
  2. As @tech2734 mentioned, should default preflight_aap_api_path=/api/controller/v2/
  3. Provides a suggestion to resolve an issue faced when turning off certificate verification to OpenShift

Comment thread roles/preflight/defaults/main.yml Outdated
Assisted-by: opencode:publishers/prelude-maas/models/glm-53-flash
@vvaldez
vvaldez deployed to external-ci October 15, 2026 20:25 — with GitHub Actions Active
@vvaldez

vvaldez commented Oct 15, 2026

Copy link
Copy Markdown
Author

Both reviews addressed on 7d2a394 — one line each:

@tech2734

  • __preflight_source_vddk moved from defaults to vars/main.yml — fixed in c8ac8f4
  • preflight_aap_api_path now defaults to /api/controller/v2/ — fixed in 7d2a394 (AAP 2.4 and earlier set /api/v2/)
  • validate_aap results[0] guard — skipped: verified empirically that json.results[0].username | default('unknown', true) handles every response shape without erroring (404 body, non-JSON response, empty results: [], and ok) — chained access propagates undefined, so the suggested guard would be a no-op
  • validate_vcenter when converted to list form — fixed in c8ac8f4
  • report.yml unran 'False' string defaults are real false booleans — fixed in c8ac8f4

@sabre1041

  • vCenter module crash aborting the run — fixed in 7d2a394: the probe now carries ignore_errors so a module exception (e.g. aiohttp ContentTypeError on the non-JSON 404 you hit) flows into the record task's graceful failure detail instead of killing the play
  • ignoring certain checks — --tags <check> / --skip-tags <check>; the summary asserts only over checks that ran, so a partial run ends green with the skipped checks reported in the table
  • AAP API path default — fixed in 7d2a394 (matches your and tech2734's suggestion)
  • verify_ssl suggestion — applied in 7d2a394; the | bool cast was already present, so the delta drops the inner default(true, true) truthy flag (an empty hostvar now resolves false)

CI green on 7d2a394 (pre-commit, ansible-lint, gitleaks, PR validation).

@vvaldez
vvaldez requested a review from sabre1041 October 15, 2026 20:27
@vvaldez

vvaldez commented Oct 15, 2026

Copy link
Copy Markdown
Author

@sabre1041 — the traceback is tolerated as of 7d2a394: the probe now carries ignore_errors, so a module exception flows into the record task and the summary reports the vCenter check as failed (with the re-run-alone guidance) instead of killing the play. A preflight run can no longer abort on this.

What the traceback itself means: the endpoint at vcenter.apps.ocpv07.rhdp.net answered HTTP 404 with Content-Type: text/plain, and vmware.vmware_rest feeds the response straight into aiohttp .json() without tolerating a non-JSON body — so the module crashes (the unclosed-session lines are crash debris) rather than failing cleanly. A 404 — rather than 401/403 — means the request reached something that does not serve the API; credentials are not the problem.

On that cluster: vcenter.apps.ocpv07.rhdp.net is an apps. route in front of vCenter, and whatever answers there is not serving /api/vcenter/datacenter. Worth a direct probe:

curl -sk -u '<user>:<pass>' https://vcenter.apps.ocpv07.rhdp.net/api/vcenter/datacenter

A plain-text/HTML 404 page confirms the route (or an intermediate proxy) is not forwarding the path to vCenter — fix the route backend, or point preflight_vcenter_hostname at the vCenter FQDN directly. Once the endpoint serves the REST API, the check goes green.

The crash itself is upstream vmware.vmware_rest behavior (aiohttp ContentTypeError on non-JSON responses) — the preflight tolerates it either way.

The certified vmware.vmware_rest collection requires vCenter 7.0+ and
cannot reach 6.x at all (no /api/ tree) — the reporter's cluster runs a
6.5-era vCenter behind a proxy that maps only /sdk and /rest. The
fallback uses community.vmware.vmware_datacenter_info (pyVmomi, 6.5+,
already a collection dependency) and runs only when the REST probe
cannot return data.

Module tier justification (community content): certified successor
vmware.vmware does not yet ship a datacenter_info module, so this is
the only content covering the 6.x path.

Assisted-by: opencode:publishers/prelude-maas/models/glm-53-flash
@vvaldez
vvaldez deployed to external-ci October 15, 2026 22:34 — with GitHub Actions Active
The proxied REST path on 6.x vCenters can answer with an empty
datacenter list instead of failing — the fallback now also triggers on
that, and the merge prefers a result with datacenters so the check
reports the real count via the SOAP SDK.

Verified against a 6.5-era vCenter behind a path-mapping proxy: the
check reports connectivity + authentication via the SOAP fallback where
the certified REST collection cannot reach the API at all.

Assisted-by: opencode:publishers/prelude-maas/models/glm-53-flash
Assisted-by: opencode:publishers/prelude-maas/models/glm-53-flash
@vvaldez
vvaldez deployed to external-ci October 15, 2026 22:43 — with GitHub Actions Active
@vvaldez

vvaldez commented Oct 15, 2026

Copy link
Copy Markdown
Author

@sabre1041 — the vCenter check now passes against your environment. Verified with a read-only run of the role against vcenter.apps.ocpv07.rhdp.net:

"detail": " HTTP session established at vcenter.apps.ocpv07.rhdp.net (1 datacenters visible via the SOAP SDK)",
"passed": true

What the probes found on that cluster:

  • The route's gateway is a Go-based proxy that maps /sdk and /rest but not /api — an authenticated GET /api/vcenter/datacenter returns Go's default 404 page not found, and its unauthenticated 401 mimics vCenter's error body (which is why the unauth probe looks like REST is served).
  • The backend is a 6.5-era vCenter — /sdk/vimServiceVersions.xml advertises urn:vim25 6.5 — so the /api/ REST tree doesn't exist there at all; that's 7.0+ only. The certified vmware.vmware_rest collection cannot reach it on any hostname.
  • The provider works because govmomi speaks SOAP via /sdk, which is mapped and supported by 6.x — no exposure change would have fixed the REST path short of an upgrade.

The fix (fa9a1be + 4a36004): the check runs the certified REST probe first, then falls back to community.vmware.vmware_datacenter_info (pyVmomi, vCenter 6.5+, already a collection dependency) whenever REST returns no data or an empty list — the proxied REST path answers with an empty list rather than failing, which the first pass of the fallback missed. The merge prefers a result with datacenters, so the check reports the real count via whichever path found them. No admin changes needed.

Module tier note: community.vmware is Tier-3 content, so the justification is documented in the task header — the certified successor vmware.vmware does not yet ship a datacenter_info module and the certified REST collection cannot reach 6.x, making this the only content covering that path. The SOAP fallback runs only when the REST probe cannot return data.

Under ansible-test the target runs from a temp-dir copy where the
repo-relative {{ role_path }}/../../../.. no longer reaches the
collection's roles/ directory. include_role with tasks_from resolves
through the installed collection in both layouts, matching how the
sibling cleanup targets already reference collection content.

Assisted-by: opencode:publishers/prelude-maas/models/glm-53-flash
@vvaldez
vvaldez deployed to external-ci October 16, 2026 01:15 — with GitHub Actions Active
sabre1041
sabre1041 previously approved these changes Oct 16, 2026

@sabre1041 sabre1041 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tech2734
tech2734 previously approved these changes Oct 16, 2026

@tech2734 tech2734 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a bug during live testing against the homelab environment: the validate_certs cascade silently flips false → true due to the default(true, true) pattern. Two suggestions below.

Comment thread roles/preflight/defaults/main.yml
Comment thread roles/preflight/defaults/main.yml
The inner default(true, true) treated a legitimate false from the
inventory as undefined and replaced it with true, silently re-enabling
TLS verification for the vCenter and AAP checks. The inner default
now catches only an undefined key, matching the openshift_verify_ssl
pattern.

Assisted-by: opencode:publishers/prelude-maas/models/glm-53-flash
@vvaldez
vvaldez dismissed stale reviews from tech2734 and sabre1041 via 166a5cc October 16, 2026 13:49
@vvaldez

vvaldez commented Oct 16, 2026

Copy link
Copy Markdown
Author

Fixed in 166a5cc.

  • validate_certs cascade: the inner default(true, true) treated an explicit false from the inventory as undefined and replaced it with the fallback true, for both the vCenter and AAP checks. The inner default now catches only an undefined key, matching the openshift_verify_ssl pattern: an inventory false survives, the env var wins when set, and a missing key still defaults to true.
  • Verified red-green: pre-fix both flags resolved true with an inventory false; post-fix all six cascade cases pass (inventory false survives, env var wins when set, missing key defaults to true).

Thanks John for the find, Andy for the pointer.

@vvaldez
vvaldez deployed to external-ci October 16, 2026 13:49 — with GitHub Actions Active

@sabre1041 sabre1041 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tech2734
tech2734 merged commit b2cdf26 into redhat-cop:v2 Oct 16, 2026
22 checks passed

This branch was successfully deployed

1 active deployment
external-ci — 166a5ccc Deployed Oct 16, 2026 by vvaldez via external-approval #739
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants