Add CI job for lightspeed operator deployment - #65
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: malingatembo The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited) Review profile: CHILL Plan: Team Run ID: Comment |
|
/recheck |
|
Build failed (check pipeline). Post Warning: |
|
/recheck |
|
recheck |
1 similar comment
|
recheck |
0c3b371 to
cecc1eb
Compare
|
Unable to freeze job graph: Job lightspeed-operator-deployment-crc does not specify a run playbook |
|
Build failed (check pipeline). Post ❌ lightspeed-operator-deployment-crc FAILURE in 29m 14s |
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc |
There was a problem hiding this comment.
question (non-blocking): Does this job deploy OpenStack as well? I think it does not but I am not 100 % certain about it. In a long term it would be good if the job deployed OpenStack as well because then we can do some basic checks that involve MCP.
There was a problem hiding this comment.
No no, this one only deploys OpenStack Lightspeed operator using the CRC ( so no full deployment of OpenStack control plane). for full OpenStack deployment + MCP integration testing, i think we'd need to add that to a seperate job or enhance this one - maybe we can keep track and have a seperate ticket for it
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc | ||
| attempts: 1 |
There was a problem hiding this comment.
suggestion (blocking): I think you want to add openstack-k8s-operators-content-provider job as a dependency [1]. This job builds the operator and then provides it to the "child" jobs that depend on it. This allows you to test the operator from a content in a PR.
There was a problem hiding this comment.
Thanks for the pointer to this feature.. Yeah, I agree we should require a build from pr. I have make the change in ebf2560
| --- | ||
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc |
There was a problem hiding this comment.
suggestion (non-blocking): If I were working on this I would be using autohold to investigate the job from the inside.
|
Build failed (check pipeline). Post ❌ lightspeed-operator-deployment-crc FAILURE in 20m 52s |
|
Unable to freeze job graph: Job lightspeed-operator-deployment-crc depends on openstack-k8s-operators-content-provider which was not run. |
|
Build failed (check pipeline). Post ❌ openstack-k8s-operators-content-provider FAILURE in 7m 04s |
| @@ -0,0 +1,17 @@ | |||
| --- | |||
There was a problem hiding this comment.
@malingatembo just FYI, you want to set the CONTAINER_TOOL to podman in Makefile. I've encountered it too:).
Lines 81 to 85 in 8eded89
Because of the failure in the job:
2026-08-26 17:01:59.168382 | controller | TASK [operator_build : lightspeed-operator - Call docker-build dry_run={{ cifmw_operator_build_dryrun|bool }}, chdir={{ operator.src }}, output_dir={{ cifmw_operator_build_basedir }}/artifacts, script=make docker-build, extra_args={'IMG': '{{ operator_img }}'}] ***
2026-08-26 17:01:59.168387 | controller | Wednesday 26 August 2026 17:01:59 +0000 (0:00:00.025) 0:02:31.959 ******
2026-08-26 17:01:59.168391 | controller | Wednesday 26 August 2026 17:01:59 +0000 (0:00:00.025) 0:02:31.958 ******
2026-08-26 17:01:59.315000 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_lightspeed_operator_call_docke.log
2026-08-26 17:01:59.315047 | controller | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
2026-08-26 17:01:59.315054 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/operator_build/tasks/build.yml:163
2026-08-26 17:01:59.315059 | controller | fatal: [localhost]: FAILED! =>
2026-08-26 17:01:59.315066 | controller | changed: true
2026-08-26 17:01:59.315072 | controller | msg: non-zero return code
2026-08-26 17:01:59.315077 | controller | rc: 2
2026-08-26 17:01:59.315082 | controller | stderr: ''
2026-08-26 17:01:59.315088 | controller | stderr_lines: []
2026-08-26 17:01:59.315093 | controller | stdout: |
2026-08-26 17:01:59.315098 | controller | ~/src/github.com/openstack-k8s-operators/lightspeed-operator ~/ci-framework-data/artifacts
2026-08-26 17:01:59.315104 | controller | docker build --build-arg GOMAXPROCS= -t 38.129.56.96:5001/openstack-k8s-operators/lightspeed-operator:011ef9691d76dd9e4f494b1c25294ed98df09a39 .
2026-08-26 17:01:59.315110 | controller | bash: line 1: docker: command not found
2026-08-26 17:01:59.315116 | controller | make: *** [Makefile:184: docker-build] Error 127
2026-08-26 17:01:59.315122 | controller | stdout_lines:
2026-08-26 17:01:59.315126 | controller | - ~/src/github.com/openstack-k8s-operators/lightspeed-operator ~/ci-framework-data/artifacts
2026-08-26 17:01:59.315131 | controller | - docker build --build-arg GOMAXPROCS= -t 38.129.56.96:5001/openstack-k8s-operators/lightspeed-operator:011ef9691d76dd9e4f494b1c25294ed98df09a39
2026-08-26 17:01:59.315137 | controller | .
2026-08-26 17:01:59.315142 | controller | - 'bash: line 1: docker: command not found'
2026-08-26 17:01:59.315146 | controller | - 'make: *** [Makefile:184: docker-build] Error 127'
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 50m 09s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 55m 18s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 57m 53s |
63b4ae6 to
b30b91e
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
b30b91e to
fadb45e
Compare
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
Creates lightspeed-operator-deployment-crc job that: - Deploys operator using ci-framework playbook - Uses example LLM endpoint (no real credentials) - Expected to fail on auth verification - Inherits from cifmw-base-crc for CRC cluster provisioning JIRA: OSPRH-33344
The deployment job depends on OpenStack-k8s-operators-content-provider but it was not configured to run in the pipeline. this cased the error: 'Job depends on openstack-k8s-operators-content-provider which was not run' Added content-provider to the jobs list so it runs before deployment, building the operator from PR code and making it available for testing. Fixes: CONFIG_ERROR - dependency not satisfied Related: OSPRH-33344
The run_hook role requires cifmw.general.ci_script module but can't find it because our custom run playbook (in lightspeed-operator repo) doesn't have access to the collection installed in ci-framework. Error from Zuul build ff62866e450d4e1e850f22ee6eb10dd1: ANSIBLE PARSE ERROR ERROR! couldn't resolve module/action 'cifmw.general.ci_script' The error appears to be in '.../ci-framework/roles/run_hook/tasks/playbook.yml': line 90 Build: https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/ff62866e450d4e1e850f22ee6eb10dd1 Added the symlink setup step that all ci-framework run playbooks use: - name: Make a symlink to local .ansible collection dir ansible.builtin.include_role: name: cifmw_helpers tasks_from: symlink_cifmw_collection.yml This creates a symlink making the cifmw collection plugins available for nested Ansible execution. Pattern from: https://github.com/openstack-k8s-operators/ci-framework/blob/main/ci/playbooks/edpm/run.yml#L18-L23 Module usage in run_hook role: https://github.com/openstack-k8s-operators/ci-framework/blob/main/roles/run_hook/tasks/playbook.yml#L90-L115 Fixes: couldn't resolve module/action 'cifmw.general.ci_script' Related: OSPRH-33344
fadb45e to
7d24a2a
Compare
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 52m 18s |
The cifmw_helpers role's internal conditional logic was preventing symlink creation even when cifmw_helpers_project_dir was set correctly. Replace role call with direct ansible.builtin.file tasks that: 1. Ensure parent directory exists 2. Create symlink forcefully This bypasses the role's broken conditionals and ensures the cifmw collection is always available for install-openstack-lightspeed.yml
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 54m 44s |
|
@malingatembo: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
fix:
1. Symlink: Remove existing cifmw directory before creating symlink
to prevent 'directory is not empty' error
2. Missing catalog image: Add cifmw_openstack_lightspeed_catalog_imag
pointing to content-provider built catalog image
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 24m 01s |
1. Fix cifmw collection symlink (Approach 2)
- ci/playbooks/lightspeed/run.yml: Use pre-built collection
- Symlink to:
ci-framework/ci/playbooks/collections/ansible_collections/cifmw
- Simpler approach with complete cifmw.general structure
2. Enable autohold debugging support
- ci/playbooks/lightspeed/pre-run.yaml: Inject SSH keys from GitHub
- Allows SSH access to failed nodes for debugging
- Fetches keys from: https://github.com/malingatembo.keys
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 14m 14s |
|
recheck Probably caused by: |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 58m 12s |
Co-authored-by: Lukas Piwowarski <33072434+lpiwowar@users.noreply.github.com>
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 52m 51s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 52m 08s |
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 58m 44s |
Zuul generates its own ansible.cfg, ignoring repo's version. Use host-vars to set ansible_collections_paths environment variable. Changes: - Remove ansible.cfg (unused by Zuul) - Add host-vars.controller.ansible_collections_paths - Points to both symlink location and ci-framework original Addresses: cifmw.general.ci_script module resolution across ansible-playbook process boundaries
Per feedback: Ensures Zuul clones ci-framework into workspace before running playbooks, making collection available at standard path. Changes: - Add openstack-k8s-operators/ci-framework to required-projects - Remove broken host-vars.ansible_collections_paths (Ansible expects ANSIBLE_COLLECTIONS_PATH env var, not inventory var) Symlink in pre-run still needed to expose collection at ~/.ansible/collections
|
@malingatembo you are missing |
| # CI job for deploying OpenStack Lightspeed operator | ||
| - job: | ||
| name: lightspeed-operator-deployment-crc | ||
| parent: cifmw-base-crc-openstack |
There was a problem hiding this comment.
@malingatembo I'm still wondering whether the cifmw-base-crc-openstack is the right base job for us. If I was working on this I would probably try to use this parent job [1]. And then I would try to use the pre_tests_*: to execute your playbook. It has
- the benefit of OpenStack already being deployed in that job (simplifies stuff for us later when we want to do the MCP testing) +
- we have already working template we can follow [1] +
- we can simply configure the test_operator to run the tests using config from Jan later (until we do that we can simply set
cifmw_run_tests: false).
But I'm assuming you picked the job for a reason. This is just my thinking.
There was a problem hiding this comment.
You are right, [1] use a parent that already has open stack, that way we can just deploy lightspeed operator in pre_test hook. On it!
|
Build failed (check pipeline). Post ✔️ openstack-k8s-operators-content-provider SUCCESS in 21m 01s |
Follow pattern from test-operator: use parent job that provides fully deployed OpenStack environment, then deploy our operator on top via pre_tests hook. Depends-On: openstack-k8s-operators/ci-framework#4108
3 participants
Creates lightspeed-operator-deployment-crc job that:
JIRA: OSPRH-33344