From 5c5fce0c39f02b0b92b7efce2501c156d7869ba0 Mon Sep 17 00:00:00 2001 From: Or Balayla Date: Thu, 1 Oct 2026 10:28:57 +0300 Subject: [PATCH 1/6] ci: split the PR pipeline into a launcher and small jobs The PR pipeline ran every check in one Jenkins job. As in NIXL and UCC, Blossom now triggers a launcher that runs one small job per stage in parallel. Each leaf posts its own PR status, and the launcher fails if any leaf fails. CloudAI/PR/cloudai-pr .ci/pr/Jenkinsfile.launcher CloudAI/PR/cloudai-pr-copyright .ci/pr/copyright_matrix.yaml CloudAI/PR/cloudai-pr-secrets .ci/pr/secrets_matrix.yaml CloudAI/PR/cloudai-pr-coverity .ci/pr/coverity_matrix.yaml CloudAI/PR/cloudai-pr-coverage .ci/pr/coverage_matrix.yaml The release pipeline is moving into this repository too, so each pipeline gets its own .ci/ directory, CloudAI/ subfolder, job prefix and JJB project. Stage bodies are unchanged. Leaves reuse .ci/Jenkinsfile. The launcher's webhook token is cloudai-pr, so the CI_SERVER secret must move to @cloudai-pr. Do that once this is merged. Also fix the webhook trigger's logging keys, which JJB silently dropped. Refs: HPCINFRA-4889 Signed-off-by: Or Balayla --- .ci/README.md | 76 ++++++++++------ .ci/header-check.yml | 2 +- .ci/job_matrix.yaml | 164 ----------------------------------- .ci/pr/Jenkinsfile.launcher | 74 ++++++++++++++++ .ci/pr/copyright_matrix.yaml | 50 +++++++++++ .ci/pr/coverage_matrix.yaml | 62 +++++++++++++ .ci/pr/coverity_matrix.yaml | 62 +++++++++++++ .ci/pr/secrets_matrix.yaml | 42 +++++++++ .ci/proj_jjb.yaml | 110 +++++++++++++++++------ 9 files changed, 421 insertions(+), 221 deletions(-) delete mode 100644 .ci/job_matrix.yaml create mode 100644 .ci/pr/Jenkinsfile.launcher create mode 100644 .ci/pr/copyright_matrix.yaml create mode 100644 .ci/pr/coverage_matrix.yaml create mode 100644 .ci/pr/coverity_matrix.yaml create mode 100644 .ci/pr/secrets_matrix.yaml diff --git a/.ci/README.md b/.ci/README.md index b33fe4d47..0e20356ca 100644 --- a/.ci/README.md +++ b/.ci/README.md @@ -1,51 +1,71 @@ -# CloudAI PR CI +# CloudAI Jenkins pipelines -Blossom Jenkins pipeline that runs on every pull request. +Blossom Jenkins pipelines for CloudAI. Each pipeline gets its own directory +here, its own subfolder of the `CloudAI` Jenkins folder and its own job prefix: + +| Pipeline | Directory | Jenkins jobs | Trigger | +| --- | --- | --- | --- | +| PR | `pr/` | `CloudAI/PR/cloudai-pr*` | `/build` comment on a pull request | + +A new pipeline `` follows the same scheme: `/`, +`CloudAI//` and the `cloudai-` prefix. + +Shared by all pipelines: | File | Role | | --- | --- | -| `proj_jjb.yaml` | Job definition, applied with `jenkins-jobs update` | -| `Jenkinsfile` | Thin entry point; reports commit status, hands off to ci-demo | -| `job_matrix.yaml` | The stages that actually run | +| `proj_jjb.yaml` | Job definitions, applied with `jenkins-jobs update` | +| `Jenkinsfile` | Runs one ci-demo matrix; reports commit status when started for a PR | | `header-check.yml` | Copyright-header policy for `header_check.py` | -Resulting job: `CloudAI` / `cloudai-ci`, alongside the existing -`cloudai-release` job. +## PR pipeline -## Flow +| File | Role | +| --- | --- | +| `pr/Jenkinsfile.launcher` | Launcher; runs the leaf jobs in parallel | +| `pr/_matrix.yaml` | The stage one leaf job runs | + +Resulting jobs, in `CloudAI/PR`: the launcher `cloudai-pr` and one leaf job +`cloudai-pr-` per matrix in `pr/`. + +### Flow `.github/workflows/blossom-ci.yml` authorizes the requester, runs Blossom's own -vulnerability scan, then triggers this job via the `CI_SERVER` secret -(`@cloudai-ci`). Jenkins checks out the PR merged into its base -branch and runs the matrix. +vulnerability scan, then triggers the launcher via the `CI_SERVER` secret +(`@cloudai-pr`). The launcher resolves the PR merged into its base +branch to one commit and starts every leaf job on it in parallel. Each leaf +reports its own commit status on the PR; the launcher's `cloudai-pr` status +fails if any leaf fails. -## Stages +### Stages -- **Check copyrights** — `header_check.py` against `header-check.yml`, limited - to files touched this calendar year. -- **Secret scan** — `secret_scan.py` over the working tree. -- **Coverity scan** — `--all-security`, reports only. Gating on findings needs - a Coverity stream to diff against; without one every pre-existing defect - would block every PR. -- **Coverage** — `pytest --cov`. Containers run as root, so the two - permission-sensitive tests are run separately with their exit code inverted. +One leaf job per stage, each with its own matrix in `pr/`: + +| Leaf job | Stage | +| --- | --- | +| `cloudai-pr-copyright` | **Check copyrights** — `header_check.py` against `header-check.yml`, limited to files touched this calendar year. | +| `cloudai-pr-secrets` | **Secret scan** — `secret_scan.py` over the working tree. | +| `cloudai-pr-coverity` | **Coverity scan** — `--all-security`; fails on any defect, since the codebase reports zero. | +| `cloudai-pr-coverage` | **Coverage** — `pytest --cov`. Containers run as root, so the two permission-sensitive tests are run separately with their exit code inverted. | Blackduck and the antivirus scan stay in the release pipeline: Blossom already -Blackducks the PR before this job starts, and the antivirus stage only acts on -published release tarballs. +Blackducks the PR before this pipeline starts, and the antivirus stage only +acts on published release tarballs. GitHub Actions (`.github/workflows/ci.yml`) still owns linting, type checking, the docs build and the install smoke test. ## Changing CI -Everything this job does lives in `job_matrix.yaml`, so changing CI behaviour -is an ordinary pull request. Only three things need Jenkins-side action, and -all are one-time: +Everything the leaf jobs do lives in their matrices, so changing what an +existing leaf runs is an ordinary pull request. Only three things need +Jenkins-side action, and all are one-time: -1. `jenkins-jobs update .ci/proj_jjb.yaml` to create or update the job +1. `jenkins-jobs update .ci/proj_jjb.yaml` to create or update the jobs. A new + PR leaf needs this: add `pr/_matrix.yaml`, list the leaf under `jobs` + in `proj_jjb.yaml` and in `pr/Jenkinsfile.launcher`, then update. 2. installing credentials 3. adding namespace egress rules when a stage needs a new host -Do not edit the job through the Jenkins web UI — changes are overwritten on the -next update. +Do not edit the jobs through the Jenkins web UI — changes are overwritten on +the next update. diff --git a/.ci/header-check.yml b/.ci/header-check.yml index 19735e077..89be9c235 100644 --- a/.ci/header-check.yml +++ b/.ci/header-check.yml @@ -11,7 +11,7 @@ general: # Jenkins pipeline definitions carry no license header by convention. # The checker fails on any file it has no rule for, so this must be an # explicit exclusion rather than simply absent from the include list. - - "Jenkinsfile$" + - "Jenkinsfile(\\..+)?$" apache-2.0: validate-spdx-license: true diff --git a/.ci/job_matrix.yaml b/.ci/job_matrix.yaml deleted file mode 100644 index 541ce31a1..000000000 --- a/.ci/job_matrix.yaml +++ /dev/null @@ -1,164 +0,0 @@ -# CloudAI PR CI matrix (ci-demo). -# -# The per-PR half of the HPCINFRA-4786 split. Derived from the release-scan -# pipeline, which is de facto mostly -# CI: header check, secret scan, Coverity and coverage are all per-commit -# checks and are carried over here more or less verbatim. -# -# Two stages are deliberately NOT carried over: -# -# Blackduck - the Blossom workflow that triggers this job already runs a -# Blackduck scan in its own Vulnerability-scan stage before Jenkins starts, -# so a second one per PR is redundant. The release job's copy also rolls -# results up under a vX.Y.beta PROJECT_VERSION, which has no meaning for a -# pull request. -# Antivirus - gated on CLOUDAI_SHA matching a vX.Y.* release tag and scans -# the published GitHub tarball. It exits 0 on anything that is not a -# release, so on a PR it would only ever be a no-op. -# -# Unlike the release job, this pipeline lives in the repository it tests, so -# pipeline-scm has already checked the pull request out into ${WORKSPACE}. -# There is no clone step and no CLOUDAI_SHA - the merge ref arrives as sha1. -# -# Named by the conf_file build parameter in .ci/proj_jjb.yaml. ---- -job: cloudai-ci - -failFast: false -timeout_minutes: 60 - -kubernetes: - cloud: il-ipp-blossom-prod - namespace: hpcx - nodeSelector: 'kubernetes.io/arch=amd64' - # Container resources are set per image (requests/limits on each - # runs_on_dockers entry below), same as the release pipeline - HPCINFRA-4329. - -runs_on_dockers: - # Same toolbox images and tags the release pipeline pins, with its - # requests/limits carried over unchanged. - - url: 'harbor.mellanox.com/toolbox/header_check:0.0.65' - name: 'header_check' - tag: '0.0.65' - arch: 'x86_64' - requests: '{memory: 128Mi, cpu: 100m}' - limits: '{memory: 512Mi, cpu: 1000m}' - - url: 'harbor.mellanox.com/toolbox/secret_scan:0.0.31' - name: 'secret_scan' - tag: '0.0.31' - arch: 'x86_64' - requests: '{memory: 128Mi, cpu: 500m}' - limits: '{memory: 512Mi, cpu: 3000m}' - - url: 'harbor.mellanox.com/hpcx/x86_64/coverity_2023:until_2028' - name: 'coverity_scan' - tag: 'base' - arch: 'x86_64' - requests: '{memory: 1Gi, cpu: 1000m}' - limits: '{memory: 4Gi, cpu: 4000m}' - - url: 'ghcr.io/astral-sh/uv:debian' - name: 'uv' - tag: 'debian' - arch: 'x86_64' - requests: '{memory: 1Gi, cpu: 1000m}' - limits: '{memory: 4Gi, cpu: 3000m}' - -steps: - - name: Check copyrights - containerSelector: '{name:"header_check"}' - run: | - set -eEx - set -o pipefail - - # --since limits the check to files touched in the current calendar - # year. Without it (e.g. --path on the repo root) the tool walks every - # file and demands the current year in its copyright line, which forces - # a 150+ file rebump every January even for files nobody touched. The - # policy we owe is "files modified this year must declare this year" - - # which is exactly what --since enforces. Files untouched this year - # keep whatever years their header already lists. - /opt/nvidia/header_check.py \ - --config ${WORKSPACE}/.ci/header-check.yml \ - --since "$(date +%Y)-01-01" \ - --git-repo ${WORKSPACE} \ - 2>&1 | tee ${WORKSPACE}/header_check.log - - archiveArtifacts: 'header_check.log' - - - name: Secret scan - containerSelector: '{name:"secret_scan"}' - run: | - set -eEx - set -o pipefail - - /opt/nvidia/secret_scan.py \ - --path . \ - --git-repo ${WORKSPACE} \ - 2>&1 | tee ${WORKSPACE}/secret_scan.log - - archiveArtifacts: 'secret_scan.log' - - - name: Coverity scan - containerSelector: '{name:"coverity_scan"}' - run: | - set -eEx - set -o pipefail - - # The release job emits to ${WORKSPACE}/coverity-scan because there the - # sources sit in a ${WORKSPACE}/CLOUDAI subdirectory. Here the sources - # ARE ${WORKSPACE}, so the emit directory is kept outside the scanned - # tree to stop cov-capture walking its own output. - cov_dir=/tmp/coverity-scan - - cov-capture --project-dir ${WORKSPACE} --dir ${cov_dir} - cov-analyze --dir ${cov_dir} --all-security --aggressiveness-level high \ - 2>&1 | tee ${WORKSPACE}/coverity-analyze.log - cov-format-errors --dir ${cov_dir} --text-output-style oneline \ - 2>&1 | tee ${WORKSPACE}/coverity.log - - # Fail on any finding; the codebase reports zero, so anything here was - # introduced by the change under test. Resolve false positives in the - # code, not by loosening this - coverity[] annotations are not honoured - # for Python here. - # `|| true` keeps grep's no-match from tripping `set -e`; an absent - # summary means the log was unreadable, which must not read as zero. - summary=$(grep -oE 'Defect occurrences found[[:space:]]*:[[:space:]]*[0-9]+' \ - ${WORKSPACE}/coverity-analyze.log || true) - [ -n "${summary}" ] || { echo "Coverity summary not found"; exit 1; } - - defects=$(printf '%s\n' "${summary}" | grep -oE '[0-9]+' | tail -1) - [ "${defects}" -eq 0 ] || { echo "Coverity reported ${defects} defect(s)"; exit 1; } - - archiveArtifacts: 'coverity.log,coverity-analyze.log' - - - name: Coverage - containerSelector: '{name:"uv"}' - run: | - set -eEx - set -o pipefail - - cd ${WORKSPACE} - uv sync --locked --extra dev - # Run all tests except the ones that fail when running as root - uv run python -m pytest --cov -k "not test_not_writable and not test_parent_wo_access" - # Run the root-sensitive tests separately (expected to fail as root, so invert exit code) - expect_pytest_failure() { - test_name="$1" - - if uv run python -m pytest --cov --cov-append -k "${test_name}"; then - echo "Expected failure did not occur for ${test_name}" - exit 1 - else - rc=$? - if [ "${rc}" -ne 1 ]; then - echo "Unexpected pytest exit code for ${test_name}: ${rc}" - exit "${rc}" - fi - echo "EXPECTED FAILURE (${test_name}, running as root) - PASSED" - fi - } - - expect_pytest_failure "test_parent_wo_access" - expect_pytest_failure "test_not_writable" - uv run coverage report 2>&1 | tee ${WORKSPACE}/coverage.log - - archiveArtifacts: 'coverage.log' diff --git a/.ci/pr/Jenkinsfile.launcher b/.ci/pr/Jenkinsfile.launcher new file mode 100644 index 000000000..5737e0c88 --- /dev/null +++ b/.ci/pr/Jenkinsfile.launcher @@ -0,0 +1,74 @@ +#!/usr/bin/groovy + +// Entry point of the PR pipeline. Blossom triggers this job; it runs the small +// leaf jobs below in parallel, each of which runs .ci/Jenkinsfile with its own +// matrix from .ci/pr/ and reports its own commit status. + +// Blossom's helper, used to report commit status back to the pull request. +@Library('blossom-github-lib@master') + +import ipp.blossom.* + +// Leaf jobs, keyed by parallel-branch name. Declared in .ci/proj_jjb.yaml. +def leafJobs = [ + 'copyright': 'cloudai-pr-copyright', + 'secrets': 'cloudai-pr-secrets', + 'coverity': 'cloudai-pr-coverity', + 'coverage': 'cloudai-pr-coverage', +] + +def githubHelper = null +def pipelineOverviewUrl = "" +// A manual run hands its own sha1 to every leaf. +def sha1 = params.sha1 + +// Injected by the Blossom workflow for PR-triggered runs; absent when a +// maintainer starts the job by hand, so every use is guarded. +if (params.githubData) { + withCredentials([usernamePassword(credentialsId: 'svcnbu-swx-cloudai-github-token', + passwordVariable: 'GIT_PASSWORD', + usernameVariable: 'GIT_USERNAME')]) { + githubHelper = GithubHelper.getInstance("${GIT_PASSWORD}", params.githubData) + } + pipelineOverviewUrl = "${env.BUILD_URL}/pipeline-overview/" + githubHelper.updateCommitStatus(pipelineOverviewUrl, "CI started", + GitHubCommitState.PENDING, env.JOB_BASE_NAME) + currentBuild.description = githubHelper.getBuildDescription() +} + +try { + if (params.githubData) { + // Resolve refs/pull//merge once, so every leaf tests the same + // commit even if the base branch moves while they start. Inside the + // try, so a failed lookup still closes the pending status. + sha1 = githubHelper.getMergedSHA() + if (!sha1) { + error "Could not resolve the merge commit of the pull request" + } + } + + // A failing leaf does not stop its siblings; parallel rethrows once all + // of them finish, so this job fails if any leaf failed. + parallel(leafJobs.collectEntries { branch, job -> + [(branch): { + build job: job, parameters: [ + string(name: 'sha1', value: sha1), + string(name: 'githubData', value: params.githubData), + ] + }] + }) +} catch (Exception ex) { + currentBuild.result = 'FAILURE' + println ex + throw ex +} finally { + if (params.githubData) { + githubHelper.updateCommitStatus( + pipelineOverviewUrl, + "CI completed", + currentBuild.resultIsBetterOrEqualTo('SUCCESS') ? GitHubCommitState.SUCCESS + : GitHubCommitState.FAILURE, + env.JOB_BASE_NAME + ) + } +} diff --git a/.ci/pr/copyright_matrix.yaml b/.ci/pr/copyright_matrix.yaml new file mode 100644 index 000000000..d624257c8 --- /dev/null +++ b/.ci/pr/copyright_matrix.yaml @@ -0,0 +1,50 @@ +# CloudAI PR pipeline - copyright header check (ci-demo), run by the cloudai-pr-copyright job. +# +# pipeline-scm has already checked the pull request out into ${WORKSPACE}; +# the merge commit arrives as sha1. +# +# Named by the conf_file build parameter in .ci/proj_jjb.yaml. +--- +job: cloudai-pr-copyright + +failFast: false +timeout_minutes: 60 + +kubernetes: + cloud: il-ipp-blossom-prod + namespace: hpcx + nodeSelector: 'kubernetes.io/arch=amd64' + # Container resources are set per image (requests/limits on each + # runs_on_dockers entry below), same as the release pipeline - HPCINFRA-4329. + +runs_on_dockers: + # Same toolbox images and tags the release pipeline pins, with its + # requests/limits carried over unchanged. + - url: 'harbor.mellanox.com/toolbox/header_check:0.0.65' + name: 'header_check' + tag: '0.0.65' + arch: 'x86_64' + requests: '{memory: 128Mi, cpu: 100m}' + limits: '{memory: 512Mi, cpu: 1000m}' + +steps: + - name: Check copyrights + containerSelector: '{name:"header_check"}' + run: | + set -eEx + set -o pipefail + + # --since limits the check to files touched in the current calendar + # year. Without it (e.g. --path on the repo root) the tool walks every + # file and demands the current year in its copyright line, which forces + # a 150+ file rebump every January even for files nobody touched. The + # policy we owe is "files modified this year must declare this year" - + # which is exactly what --since enforces. Files untouched this year + # keep whatever years their header already lists. + /opt/nvidia/header_check.py \ + --config ${WORKSPACE}/.ci/header-check.yml \ + --since "$(date +%Y)-01-01" \ + --git-repo ${WORKSPACE} \ + 2>&1 | tee ${WORKSPACE}/header_check.log + + archiveArtifacts: 'header_check.log' diff --git a/.ci/pr/coverage_matrix.yaml b/.ci/pr/coverage_matrix.yaml new file mode 100644 index 000000000..6014c1022 --- /dev/null +++ b/.ci/pr/coverage_matrix.yaml @@ -0,0 +1,62 @@ +# CloudAI PR pipeline - unit tests with coverage (ci-demo), run by the cloudai-pr-coverage job. +# +# pipeline-scm has already checked the pull request out into ${WORKSPACE}; +# the merge commit arrives as sha1. +# +# Named by the conf_file build parameter in .ci/proj_jjb.yaml. +--- +job: cloudai-pr-coverage + +failFast: false +timeout_minutes: 60 + +kubernetes: + cloud: il-ipp-blossom-prod + namespace: hpcx + nodeSelector: 'kubernetes.io/arch=amd64' + # Container resources are set per image (requests/limits on each + # runs_on_dockers entry below), same as the release pipeline - HPCINFRA-4329. + +runs_on_dockers: + # Same toolbox images and tags the release pipeline pins, with its + # requests/limits carried over unchanged. + - url: 'ghcr.io/astral-sh/uv:debian' + name: 'uv' + tag: 'debian' + arch: 'x86_64' + requests: '{memory: 1Gi, cpu: 1000m}' + limits: '{memory: 4Gi, cpu: 3000m}' + +steps: + - name: Coverage + containerSelector: '{name:"uv"}' + run: | + set -eEx + set -o pipefail + + cd ${WORKSPACE} + uv sync --locked --extra dev + # Run all tests except the ones that fail when running as root + uv run python -m pytest --cov -k "not test_not_writable and not test_parent_wo_access" + # Run the root-sensitive tests separately (expected to fail as root, so invert exit code) + expect_pytest_failure() { + test_name="$1" + + if uv run python -m pytest --cov --cov-append -k "${test_name}"; then + echo "Expected failure did not occur for ${test_name}" + exit 1 + else + rc=$? + if [ "${rc}" -ne 1 ]; then + echo "Unexpected pytest exit code for ${test_name}: ${rc}" + exit "${rc}" + fi + echo "EXPECTED FAILURE (${test_name}, running as root) - PASSED" + fi + } + + expect_pytest_failure "test_parent_wo_access" + expect_pytest_failure "test_not_writable" + uv run coverage report 2>&1 | tee ${WORKSPACE}/coverage.log + + archiveArtifacts: 'coverage.log' diff --git a/.ci/pr/coverity_matrix.yaml b/.ci/pr/coverity_matrix.yaml new file mode 100644 index 000000000..eeca3a22a --- /dev/null +++ b/.ci/pr/coverity_matrix.yaml @@ -0,0 +1,62 @@ +# CloudAI PR pipeline - Coverity scan (ci-demo), run by the cloudai-pr-coverity job. +# +# pipeline-scm has already checked the pull request out into ${WORKSPACE}; +# the merge commit arrives as sha1. +# +# Named by the conf_file build parameter in .ci/proj_jjb.yaml. +--- +job: cloudai-pr-coverity + +failFast: false +timeout_minutes: 60 + +kubernetes: + cloud: il-ipp-blossom-prod + namespace: hpcx + nodeSelector: 'kubernetes.io/arch=amd64' + # Container resources are set per image (requests/limits on each + # runs_on_dockers entry below), same as the release pipeline - HPCINFRA-4329. + +runs_on_dockers: + # Same toolbox images and tags the release pipeline pins, with its + # requests/limits carried over unchanged. + - url: 'harbor.mellanox.com/hpcx/x86_64/coverity_2023:until_2028' + name: 'coverity_scan' + tag: 'base' + arch: 'x86_64' + requests: '{memory: 1Gi, cpu: 1000m}' + limits: '{memory: 4Gi, cpu: 4000m}' + +steps: + - name: Coverity scan + containerSelector: '{name:"coverity_scan"}' + run: | + set -eEx + set -o pipefail + + # The release job emits to ${WORKSPACE}/coverity-scan because there the + # sources sit in a ${WORKSPACE}/CLOUDAI subdirectory. Here the sources + # ARE ${WORKSPACE}, so the emit directory is kept outside the scanned + # tree to stop cov-capture walking its own output. + cov_dir=/tmp/coverity-scan + + cov-capture --project-dir ${WORKSPACE} --dir ${cov_dir} + cov-analyze --dir ${cov_dir} --all-security --aggressiveness-level high \ + 2>&1 | tee ${WORKSPACE}/coverity-analyze.log + cov-format-errors --dir ${cov_dir} --text-output-style oneline \ + 2>&1 | tee ${WORKSPACE}/coverity.log + + # Fail on any finding; the codebase reports zero, so anything here was + # introduced by the change under test. Resolve false positives in the + # code, not by loosening this - coverity[] annotations are not honoured + # for Python here. + # `|| true` keeps grep's no-match from tripping `set -e`; an absent + # summary means the log was unreadable, which must not read as zero. + summary=$(grep -oE 'Defect occurrences found[[:space:]]*:[[:space:]]*[0-9]+' \ + ${WORKSPACE}/coverity-analyze.log || true) + [ -n "${summary}" ] || { echo "Coverity summary not found"; exit 1; } + + defects=$(printf '%s\n' "${summary}" | grep -oE '[0-9]+' | tail -1) + [ "${defects}" -eq 0 ] || { echo "Coverity reported ${defects} defect(s)"; exit 1; } + + archiveArtifacts: 'coverity.log,coverity-analyze.log' diff --git a/.ci/pr/secrets_matrix.yaml b/.ci/pr/secrets_matrix.yaml new file mode 100644 index 000000000..846a1484d --- /dev/null +++ b/.ci/pr/secrets_matrix.yaml @@ -0,0 +1,42 @@ +# CloudAI PR pipeline - secret scan (ci-demo), run by the cloudai-pr-secrets job. +# +# pipeline-scm has already checked the pull request out into ${WORKSPACE}; +# the merge commit arrives as sha1. +# +# Named by the conf_file build parameter in .ci/proj_jjb.yaml. +--- +job: cloudai-pr-secrets + +failFast: false +timeout_minutes: 60 + +kubernetes: + cloud: il-ipp-blossom-prod + namespace: hpcx + nodeSelector: 'kubernetes.io/arch=amd64' + # Container resources are set per image (requests/limits on each + # runs_on_dockers entry below), same as the release pipeline - HPCINFRA-4329. + +runs_on_dockers: + # Same toolbox images and tags the release pipeline pins, with its + # requests/limits carried over unchanged. + - url: 'harbor.mellanox.com/toolbox/secret_scan:0.0.31' + name: 'secret_scan' + tag: '0.0.31' + arch: 'x86_64' + requests: '{memory: 128Mi, cpu: 500m}' + limits: '{memory: 512Mi, cpu: 3000m}' + +steps: + - name: Secret scan + containerSelector: '{name:"secret_scan"}' + run: | + set -eEx + set -o pipefail + + /opt/nvidia/secret_scan.py \ + --path . \ + --git-repo ${WORKSPACE} \ + 2>&1 | tee ${WORKSPACE}/secret_scan.log + + archiveArtifacts: 'secret_scan.log' diff --git a/.ci/proj_jjb.yaml b/.ci/proj_jjb.yaml index d9f60939f..fce577879 100644 --- a/.ci/proj_jjb.yaml +++ b/.ci/proj_jjb.yaml @@ -1,20 +1,33 @@ -# Jenkins Job Builder definition for the CloudAI PR CI pipeline. +# Jenkins Job Builder definition for CloudAI's Jenkins jobs. # # Applied to the Blossom Jenkins instance that hosts the CloudAI folder. -# Do NOT edit the resulting job through the Jenkins web UI. +# Do NOT edit the resulting jobs through the Jenkins web UI. # Changes will be overwritten on the next jenkins-jobs update according to this file. # -# Resulting job: CloudAI / cloudai-ci -# Pipeline body: .ci/Jenkinsfile + .ci/job_matrix.yaml (ci-demo). +# Each pipeline gets its own subfolder of CloudAI, its own job prefix and its +# own directory under .ci/: +# CloudAI/PR/cloudai-pr - launcher; Blossom triggers it, it runs the +# leaves below (.ci/pr/Jenkinsfile.launcher) +# CloudAI/PR/cloudai-pr- - one small pipeline per .ci/pr/_matrix.yaml +# (.ci/Jenkinsfile, ci-demo) # -# The CloudAI folder itself is declared alongside the existing cloudai-release -# job, so it is deliberately not redeclared here - doing so would overwrite that -# folder's description on every update from this side. +# The CloudAI folder itself and the release job are still declared outside this +# repository, so neither is redeclared here - doing so would overwrite them on +# every update from this side. +# The PR pipeline's subfolder. Nested under CloudAI so its jobs inherit that +# folder's credentials and permissions. - job-template: - name: "{jjb_proj}" + name: "{jjb_folder}/PR" + project-type: folder + description: &description | +

Do NOT edit this job through the Web GUI !

+

Source of truth: cloudai/.ci/proj_jjb.yaml. Edit there and re-run jenkins-jobs update.

+ +- job-template: + name: "{jjb_proj}-pr" project-type: pipeline - folder: "{jjb_folder}" + folder: "{jjb_folder}/PR" concurrent: true properties: - github: @@ -25,10 +38,8 @@ - inject: keep-system-variables: true properties-content: | - jjb_proj={jjb_proj} - description: | -

Do NOT edit this job through the Web GUI !

-

Source of truth: cloudai/.ci/proj_jjb.yaml. Edit there and re-run jenkins-jobs update.

+ jjb_proj={jjb_proj}-pr + description: *description parameters: - string: name: "sha1" @@ -38,23 +49,19 @@ name: "githubData" default: "" description: "[blossom] Raw webhook payload; set automatically on PR-triggered runs. Leave empty for a manual run - the pipeline then skips reporting commit status. Paste a captured payload only to replay a build onto a specific PR." - - string: - name: "conf_file" - default: ".ci/job_matrix.yaml" - description: "[tools] CI Demo job matrix" - string: name: "script" - default: ".ci/Jenkinsfile" - description: "[tools] CI Demo Jenkinsfile to initialise job" + default: ".ci/pr/Jenkinsfile.launcher" + description: "[tools] Jenkinsfile that starts the leaf jobs" triggers: - generic-webhook-trigger: # Also the right-hand side of the CI_SERVER GitHub secret: - # @cloudai-ci. These tokens are matched across the + # @cloudai-pr. These tokens are matched across the # whole instance rather than per folder, so two jobs sharing one # would both fire - keep it unique. - token: "{jjb_proj}" - print-contributors: true - print-posted-content: true + token: "{jjb_proj}-pr" + print-contrib-var: true + print-post-content: true silent-response: false cause: "Blossom PR trigger" post-content-params: @@ -66,7 +73,7 @@ - type: JSONPath key: sha1 value: '$.concat("refs/pull/", $.issue.number, "/merge")' - pipeline-scm: + pipeline-scm: &pipeline_scm scm: - git: url: "{jjb_git}" @@ -81,12 +88,50 @@ wipe-workspace: true script-path: "$script" +# A leaf job. No trigger of its own: only the launcher above starts it, passing +# down the commit to test and the Blossom payload so it can report status. +- job-template: + name: "{jjb_proj}-pr-{leaf}" + project-type: pipeline + folder: "{jjb_folder}/PR" + concurrent: true + properties: + - github: + url: "{jjb_github_url}" + - build-discarder: + days-to-keep: 14 + num-to-keep: 100 + - inject: + keep-system-variables: true + properties-content: | + jjb_proj={jjb_proj}-pr-{leaf} + description: *description + parameters: + - string: + name: "sha1" + default: "{jjb_branch}" + description: "Commit of CloudAI to be checked. Set by the launcher." + - string: + name: "githubData" + default: "" + description: "[blossom] Raw webhook payload, passed down by the launcher. Leave empty for a manual run - the pipeline then skips reporting commit status." + - string: + name: "conf_file" + default: ".ci/pr/{leaf}_matrix.yaml" + description: "[tools] CI Demo job matrix" + - string: + name: "script" + default: ".ci/Jenkinsfile" + description: "[tools] CI Demo Jenkinsfile to initialise job" + pipeline-scm: *pipeline_scm + +# One project per pipeline, so each can be added or moved on its own. - project: - name: cloudai-ci - jjb_proj: 'cloudai-ci' + name: cloudai-pr + jjb_proj: 'cloudai' jjb_folder: 'CloudAI' jjb_branch: 'main' - # Unlike cloudai-release, this pipeline lives in the repository it tests, + # Unlike cloudai-release, these pipelines live in the repository they test, # so the SCM clone URL and the GitHub project URL are the same repo. jjb_git: 'git@github.com:NVIDIA/cloudai.git' jjb_github_url: 'https://github.com/NVIDIA/cloudai/' @@ -94,4 +139,13 @@ # the other jobs on this instance already use. jjb_gh_cred_id: 'svcnbu-swx-hpcx_ssh_with_key' jobs: - - "{jjb_proj}" + - "{jjb_folder}/PR" + - "{jjb_proj}-pr" + - "{jjb_proj}-pr-{leaf}": + leaf: copyright + - "{jjb_proj}-pr-{leaf}": + leaf: secrets + - "{jjb_proj}-pr-{leaf}": + leaf: coverity + - "{jjb_proj}-pr-{leaf}": + leaf: coverage From c0f260cd5069a014b06e3621d131a81e2e7744ec Mon Sep 17 00:00:00 2001 From: Or Balayla Date: Thu, 1 Oct 2026 14:59:47 +0300 Subject: [PATCH 2/6] ci: report the PR launcher result on blossom-ci The Blossom workflow sets blossom-ci to pending and nothing closed it, because the launcher reported under its own cloudai-pr status. Drop the context so the launcher reports on blossom-ci, as the NIXL dispatcher does. Leaf jobs keep their cloudai-pr- statuses. Refs: HPCINFRA-4884 Signed-off-by: Or Balayla --- .ci/README.md | 5 +++-- .ci/pr/Jenkinsfile.launcher | 7 ++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/.ci/README.md b/.ci/README.md index 0e20356ca..26fd228c7 100644 --- a/.ci/README.md +++ b/.ci/README.md @@ -34,8 +34,9 @@ Resulting jobs, in `CloudAI/PR`: the launcher `cloudai-pr` and one leaf job vulnerability scan, then triggers the launcher via the `CI_SERVER` secret (`@cloudai-pr`). The launcher resolves the PR merged into its base branch to one commit and starts every leaf job on it in parallel. Each leaf -reports its own commit status on the PR; the launcher's `cloudai-pr` status -fails if any leaf fails. +reports its own `cloudai-pr-` commit status on the PR. The launcher +reports the overall result on `blossom-ci`, the status the workflow leaves +pending, which fails if any leaf fails. ### Stages diff --git a/.ci/pr/Jenkinsfile.launcher b/.ci/pr/Jenkinsfile.launcher index 5737e0c88..392ef39bb 100644 --- a/.ci/pr/Jenkinsfile.launcher +++ b/.ci/pr/Jenkinsfile.launcher @@ -31,8 +31,10 @@ if (params.githubData) { githubHelper = GithubHelper.getInstance("${GIT_PASSWORD}", params.githubData) } pipelineOverviewUrl = "${env.BUILD_URL}/pipeline-overview/" + // No context: the launcher reports on blossom-ci, the status the Blossom + // workflow leaves pending, so the PR shows the overall result there. githubHelper.updateCommitStatus(pipelineOverviewUrl, "CI started", - GitHubCommitState.PENDING, env.JOB_BASE_NAME) + GitHubCommitState.PENDING) currentBuild.description = githubHelper.getBuildDescription() } @@ -67,8 +69,7 @@ try { pipelineOverviewUrl, "CI completed", currentBuild.resultIsBetterOrEqualTo('SUCCESS') ? GitHubCommitState.SUCCESS - : GitHubCommitState.FAILURE, - env.JOB_BASE_NAME + : GitHubCommitState.FAILURE ) } } From 760f9e74f143dfc27d773931aa8b0afa3ddb6646 Mon Sep 17 00:00:00 2001 From: Or Balayla Date: Thu, 1 Oct 2026 15:20:06 +0300 Subject: [PATCH 3/6] ci: keep blossom-ci green when only a leaf fails A failed leaf already shows on its own cloudai-pr- status. Failing blossom-ci as well left it stale after re-running just that leaf, since only the launcher writes it. Start leaves with propagate: false and paint a failed one's branch red instead, so blossom-ci fails only when dispatching breaks, as in the NIXL dispatcher. Refs: HPCINFRA-4884 Signed-off-by: Or Balayla --- .ci/README.md | 7 ++++--- .ci/pr/Jenkinsfile.launcher | 13 ++++++++++--- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/.ci/README.md b/.ci/README.md index 26fd228c7..a720dacc3 100644 --- a/.ci/README.md +++ b/.ci/README.md @@ -34,9 +34,10 @@ Resulting jobs, in `CloudAI/PR`: the launcher `cloudai-pr` and one leaf job vulnerability scan, then triggers the launcher via the `CI_SERVER` secret (`@cloudai-pr`). The launcher resolves the PR merged into its base branch to one commit and starts every leaf job on it in parallel. Each leaf -reports its own `cloudai-pr-` commit status on the PR. The launcher -reports the overall result on `blossom-ci`, the status the workflow leaves -pending, which fails if any leaf fails. +reports its own `cloudai-pr-` commit status on the PR, which is where a +failure shows. The launcher closes `blossom-ci`, the status the workflow leaves +pending, and fails it only if dispatching breaks. A failed leaf can be re-run +on its own in Jenkins (same parameters); its status then updates the PR. ### Stages diff --git a/.ci/pr/Jenkinsfile.launcher b/.ci/pr/Jenkinsfile.launcher index 392ef39bb..4b090588b 100644 --- a/.ci/pr/Jenkinsfile.launcher +++ b/.ci/pr/Jenkinsfile.launcher @@ -49,14 +49,21 @@ try { } } - // A failing leaf does not stop its siblings; parallel rethrows once all - // of them finish, so this job fails if any leaf failed. + // Each leaf reports its own status, so a failed leaf only paints its + // branch red: blossom-ci fails only if dispatching breaks, and re-running + // just that leaf clears the PR. propagate: false keeps a missing leaf + // job fatal, unlike catching every error. parallel(leafJobs.collectEntries { branch, job -> [(branch): { - build job: job, parameters: [ + def run = build job: job, propagate: false, parameters: [ string(name: 'sha1', value: sha1), string(name: 'githubData', value: params.githubData), ] + if (run.result != 'SUCCESS') { + catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') { + error "${job} #${run.number} finished ${run.result}" + } + } }] }) } catch (Exception ex) { From 5b1b9233d88d43fd19a7b3e74d7600291d5694c2 Mon Sep 17 00:00:00 2001 From: Or Balayla Date: Sun, 4 Oct 2026 10:07:31 +0300 Subject: [PATCH 4/6] ci: keep the PR pipeline files in the shared .ci layout Follow the .ci layout the other ci-demo projects use instead of a per-pipeline directory: the leaf matrices move from .ci/pr/ to .ci/pipeline/ and the launcher to .ci/Jenkinsfile.launcher. Jenkins job names, the CloudAI/PR folder and the commit-status contexts are unchanged; only the script and conf_file defaults in proj_jjb.yaml point at the new paths. Refs: HPCINFRA-4889 Signed-off-by: Or Balayla --- .ci/{pr => }/Jenkinsfile.launcher | 2 +- .ci/README.md | 31 +++++++++------------- .ci/{pr => pipeline}/copyright_matrix.yaml | 0 .ci/{pr => pipeline}/coverage_matrix.yaml | 0 .ci/{pr => pipeline}/coverity_matrix.yaml | 0 .ci/{pr => pipeline}/secrets_matrix.yaml | 0 .ci/proj_jjb.yaml | 11 ++++---- 7 files changed, 19 insertions(+), 25 deletions(-) rename .ci/{pr => }/Jenkinsfile.launcher (98%) rename .ci/{pr => pipeline}/copyright_matrix.yaml (100%) rename .ci/{pr => pipeline}/coverage_matrix.yaml (100%) rename .ci/{pr => pipeline}/coverity_matrix.yaml (100%) rename .ci/{pr => pipeline}/secrets_matrix.yaml (100%) diff --git a/.ci/pr/Jenkinsfile.launcher b/.ci/Jenkinsfile.launcher similarity index 98% rename from .ci/pr/Jenkinsfile.launcher rename to .ci/Jenkinsfile.launcher index 4b090588b..0c545c844 100644 --- a/.ci/pr/Jenkinsfile.launcher +++ b/.ci/Jenkinsfile.launcher @@ -2,7 +2,7 @@ // Entry point of the PR pipeline. Blossom triggers this job; it runs the small // leaf jobs below in parallel, each of which runs .ci/Jenkinsfile with its own -// matrix from .ci/pr/ and reports its own commit status. +// matrix from .ci/pipeline/ and reports its own commit status. // Blossom's helper, used to report commit status back to the pull request. @Library('blossom-github-lib@master') diff --git a/.ci/README.md b/.ci/README.md index a720dacc3..485901a22 100644 --- a/.ci/README.md +++ b/.ci/README.md @@ -1,32 +1,27 @@ # CloudAI Jenkins pipelines -Blossom Jenkins pipelines for CloudAI. Each pipeline gets its own directory -here, its own subfolder of the `CloudAI` Jenkins folder and its own job prefix: +Blossom Jenkins pipelines for CloudAI. Each pipeline gets its own subfolder of +the `CloudAI` Jenkins folder and its own job prefix: -| Pipeline | Directory | Jenkins jobs | Trigger | -| --- | --- | --- | --- | -| PR | `pr/` | `CloudAI/PR/cloudai-pr*` | `/build` comment on a pull request | +| Pipeline | Jenkins jobs | Trigger | +| --- | --- | --- | +| PR | `CloudAI/PR/cloudai-pr*` | `/build` comment on a pull request | -A new pipeline `` follows the same scheme: `/`, -`CloudAI//` and the `cloudai-` prefix. - -Shared by all pipelines: +A new pipeline `` follows the same scheme: `CloudAI//` and the +`cloudai-` prefix. | File | Role | | --- | --- | | `proj_jjb.yaml` | Job definitions, applied with `jenkins-jobs update` | | `Jenkinsfile` | Runs one ci-demo matrix; reports commit status when started for a PR | +| `Jenkinsfile.launcher` | PR pipeline launcher; runs the leaf jobs in parallel | +| `pipeline/_matrix.yaml` | The stage one leaf job runs | | `header-check.yml` | Copyright-header policy for `header_check.py` | ## PR pipeline -| File | Role | -| --- | --- | -| `pr/Jenkinsfile.launcher` | Launcher; runs the leaf jobs in parallel | -| `pr/_matrix.yaml` | The stage one leaf job runs | - Resulting jobs, in `CloudAI/PR`: the launcher `cloudai-pr` and one leaf job -`cloudai-pr-` per matrix in `pr/`. +`cloudai-pr-` per matrix in `pipeline/`. ### Flow @@ -41,7 +36,7 @@ on its own in Jenkins (same parameters); its status then updates the PR. ### Stages -One leaf job per stage, each with its own matrix in `pr/`: +One leaf job per stage, each with its own matrix in `pipeline/`: | Leaf job | Stage | | --- | --- | @@ -64,8 +59,8 @@ existing leaf runs is an ordinary pull request. Only three things need Jenkins-side action, and all are one-time: 1. `jenkins-jobs update .ci/proj_jjb.yaml` to create or update the jobs. A new - PR leaf needs this: add `pr/_matrix.yaml`, list the leaf under `jobs` - in `proj_jjb.yaml` and in `pr/Jenkinsfile.launcher`, then update. + PR leaf needs this: add `pipeline/_matrix.yaml`, list the leaf under + `jobs` in `proj_jjb.yaml` and in `Jenkinsfile.launcher`, then update. 2. installing credentials 3. adding namespace egress rules when a stage needs a new host diff --git a/.ci/pr/copyright_matrix.yaml b/.ci/pipeline/copyright_matrix.yaml similarity index 100% rename from .ci/pr/copyright_matrix.yaml rename to .ci/pipeline/copyright_matrix.yaml diff --git a/.ci/pr/coverage_matrix.yaml b/.ci/pipeline/coverage_matrix.yaml similarity index 100% rename from .ci/pr/coverage_matrix.yaml rename to .ci/pipeline/coverage_matrix.yaml diff --git a/.ci/pr/coverity_matrix.yaml b/.ci/pipeline/coverity_matrix.yaml similarity index 100% rename from .ci/pr/coverity_matrix.yaml rename to .ci/pipeline/coverity_matrix.yaml diff --git a/.ci/pr/secrets_matrix.yaml b/.ci/pipeline/secrets_matrix.yaml similarity index 100% rename from .ci/pr/secrets_matrix.yaml rename to .ci/pipeline/secrets_matrix.yaml diff --git a/.ci/proj_jjb.yaml b/.ci/proj_jjb.yaml index fce577879..595da0ceb 100644 --- a/.ci/proj_jjb.yaml +++ b/.ci/proj_jjb.yaml @@ -4,11 +4,10 @@ # Do NOT edit the resulting jobs through the Jenkins web UI. # Changes will be overwritten on the next jenkins-jobs update according to this file. # -# Each pipeline gets its own subfolder of CloudAI, its own job prefix and its -# own directory under .ci/: +# Each pipeline gets its own subfolder of CloudAI and its own job prefix: # CloudAI/PR/cloudai-pr - launcher; Blossom triggers it, it runs the -# leaves below (.ci/pr/Jenkinsfile.launcher) -# CloudAI/PR/cloudai-pr- - one small pipeline per .ci/pr/_matrix.yaml +# leaves below (.ci/Jenkinsfile.launcher) +# CloudAI/PR/cloudai-pr- - one small pipeline per .ci/pipeline/_matrix.yaml # (.ci/Jenkinsfile, ci-demo) # # The CloudAI folder itself and the release job are still declared outside this @@ -51,7 +50,7 @@ description: "[blossom] Raw webhook payload; set automatically on PR-triggered runs. Leave empty for a manual run - the pipeline then skips reporting commit status. Paste a captured payload only to replay a build onto a specific PR." - string: name: "script" - default: ".ci/pr/Jenkinsfile.launcher" + default: ".ci/Jenkinsfile.launcher" description: "[tools] Jenkinsfile that starts the leaf jobs" triggers: - generic-webhook-trigger: @@ -117,7 +116,7 @@ description: "[blossom] Raw webhook payload, passed down by the launcher. Leave empty for a manual run - the pipeline then skips reporting commit status." - string: name: "conf_file" - default: ".ci/pr/{leaf}_matrix.yaml" + default: ".ci/pipeline/{leaf}_matrix.yaml" description: "[tools] CI Demo job matrix" - string: name: "script" From e2d30f070d176d5cd23af804fdd6979e8908dea2 Mon Sep 17 00:00:00 2001 From: Or Balayla Date: Sun, 4 Oct 2026 12:33:28 +0300 Subject: [PATCH 5/6] ci: name the PR pipeline CI and drop the script parameter The team calls this pipeline CI, not PR: the Jenkins subfolder becomes CloudAI/CI, the launcher cloudai-ci (webhook token cloudai-ci) and the leaf jobs and their commit statuses cloudai-ci-. The script parameter only fed script-path and nobody overrides it, so each job now sets its Jenkinsfile in script-path directly. Refs: HPCINFRA-4889 Signed-off-by: Or Balayla --- .ci/Jenkinsfile.launcher | 10 +++--- .ci/README.md | 24 ++++++------- .ci/pipeline/copyright_matrix.yaml | 4 +-- .ci/pipeline/coverage_matrix.yaml | 4 +-- .ci/pipeline/coverity_matrix.yaml | 4 +-- .ci/pipeline/secrets_matrix.yaml | 4 +-- .ci/proj_jjb.yaml | 56 +++++++++++++----------------- 7 files changed, 50 insertions(+), 56 deletions(-) diff --git a/.ci/Jenkinsfile.launcher b/.ci/Jenkinsfile.launcher index 0c545c844..a76b88351 100644 --- a/.ci/Jenkinsfile.launcher +++ b/.ci/Jenkinsfile.launcher @@ -1,6 +1,6 @@ #!/usr/bin/groovy -// Entry point of the PR pipeline. Blossom triggers this job; it runs the small +// Entry point of the CI pipeline. Blossom triggers this job; it runs the small // leaf jobs below in parallel, each of which runs .ci/Jenkinsfile with its own // matrix from .ci/pipeline/ and reports its own commit status. @@ -11,10 +11,10 @@ import ipp.blossom.* // Leaf jobs, keyed by parallel-branch name. Declared in .ci/proj_jjb.yaml. def leafJobs = [ - 'copyright': 'cloudai-pr-copyright', - 'secrets': 'cloudai-pr-secrets', - 'coverity': 'cloudai-pr-coverity', - 'coverage': 'cloudai-pr-coverage', + 'copyright': 'cloudai-ci-copyright', + 'secrets': 'cloudai-ci-secrets', + 'coverity': 'cloudai-ci-coverity', + 'coverage': 'cloudai-ci-coverage', ] def githubHelper = null diff --git a/.ci/README.md b/.ci/README.md index 485901a22..6eb777ae9 100644 --- a/.ci/README.md +++ b/.ci/README.md @@ -5,7 +5,7 @@ the `CloudAI` Jenkins folder and its own job prefix: | Pipeline | Jenkins jobs | Trigger | | --- | --- | --- | -| PR | `CloudAI/PR/cloudai-pr*` | `/build` comment on a pull request | +| CI | `CloudAI/CI/cloudai-ci*` | `/build` comment on a pull request | A new pipeline `` follows the same scheme: `CloudAI//` and the `cloudai-` prefix. @@ -14,22 +14,22 @@ A new pipeline `` follows the same scheme: `CloudAI//` and the | --- | --- | | `proj_jjb.yaml` | Job definitions, applied with `jenkins-jobs update` | | `Jenkinsfile` | Runs one ci-demo matrix; reports commit status when started for a PR | -| `Jenkinsfile.launcher` | PR pipeline launcher; runs the leaf jobs in parallel | +| `Jenkinsfile.launcher` | CI pipeline launcher; runs the leaf jobs in parallel | | `pipeline/_matrix.yaml` | The stage one leaf job runs | | `header-check.yml` | Copyright-header policy for `header_check.py` | -## PR pipeline +## CI pipeline -Resulting jobs, in `CloudAI/PR`: the launcher `cloudai-pr` and one leaf job -`cloudai-pr-` per matrix in `pipeline/`. +Resulting jobs, in `CloudAI/CI`: the launcher `cloudai-ci` and one leaf job +`cloudai-ci-` per matrix in `pipeline/`. ### Flow `.github/workflows/blossom-ci.yml` authorizes the requester, runs Blossom's own vulnerability scan, then triggers the launcher via the `CI_SERVER` secret -(`@cloudai-pr`). The launcher resolves the PR merged into its base +(`@cloudai-ci`). The launcher resolves the PR merged into its base branch to one commit and starts every leaf job on it in parallel. Each leaf -reports its own `cloudai-pr-` commit status on the PR, which is where a +reports its own `cloudai-ci-` commit status on the PR, which is where a failure shows. The launcher closes `blossom-ci`, the status the workflow leaves pending, and fails it only if dispatching breaks. A failed leaf can be re-run on its own in Jenkins (same parameters); its status then updates the PR. @@ -40,10 +40,10 @@ One leaf job per stage, each with its own matrix in `pipeline/`: | Leaf job | Stage | | --- | --- | -| `cloudai-pr-copyright` | **Check copyrights** — `header_check.py` against `header-check.yml`, limited to files touched this calendar year. | -| `cloudai-pr-secrets` | **Secret scan** — `secret_scan.py` over the working tree. | -| `cloudai-pr-coverity` | **Coverity scan** — `--all-security`; fails on any defect, since the codebase reports zero. | -| `cloudai-pr-coverage` | **Coverage** — `pytest --cov`. Containers run as root, so the two permission-sensitive tests are run separately with their exit code inverted. | +| `cloudai-ci-copyright` | **Check copyrights** — `header_check.py` against `header-check.yml`, limited to files touched this calendar year. | +| `cloudai-ci-secrets` | **Secret scan** — `secret_scan.py` over the working tree. | +| `cloudai-ci-coverity` | **Coverity scan** — `--all-security`; fails on any defect, since the codebase reports zero. | +| `cloudai-ci-coverage` | **Coverage** — `pytest --cov`. Containers run as root, so the two permission-sensitive tests are run separately with their exit code inverted. | Blackduck and the antivirus scan stay in the release pipeline: Blossom already Blackducks the PR before this pipeline starts, and the antivirus stage only @@ -59,7 +59,7 @@ existing leaf runs is an ordinary pull request. Only three things need Jenkins-side action, and all are one-time: 1. `jenkins-jobs update .ci/proj_jjb.yaml` to create or update the jobs. A new - PR leaf needs this: add `pipeline/_matrix.yaml`, list the leaf under + leaf needs this: add `pipeline/_matrix.yaml`, list the leaf under `jobs` in `proj_jjb.yaml` and in `Jenkinsfile.launcher`, then update. 2. installing credentials 3. adding namespace egress rules when a stage needs a new host diff --git a/.ci/pipeline/copyright_matrix.yaml b/.ci/pipeline/copyright_matrix.yaml index d624257c8..a9bda6fd8 100644 --- a/.ci/pipeline/copyright_matrix.yaml +++ b/.ci/pipeline/copyright_matrix.yaml @@ -1,11 +1,11 @@ -# CloudAI PR pipeline - copyright header check (ci-demo), run by the cloudai-pr-copyright job. +# CloudAI CI pipeline - copyright header check (ci-demo), run by the cloudai-ci-copyright job. # # pipeline-scm has already checked the pull request out into ${WORKSPACE}; # the merge commit arrives as sha1. # # Named by the conf_file build parameter in .ci/proj_jjb.yaml. --- -job: cloudai-pr-copyright +job: cloudai-ci-copyright failFast: false timeout_minutes: 60 diff --git a/.ci/pipeline/coverage_matrix.yaml b/.ci/pipeline/coverage_matrix.yaml index 6014c1022..144c206fc 100644 --- a/.ci/pipeline/coverage_matrix.yaml +++ b/.ci/pipeline/coverage_matrix.yaml @@ -1,11 +1,11 @@ -# CloudAI PR pipeline - unit tests with coverage (ci-demo), run by the cloudai-pr-coverage job. +# CloudAI CI pipeline - unit tests with coverage (ci-demo), run by the cloudai-ci-coverage job. # # pipeline-scm has already checked the pull request out into ${WORKSPACE}; # the merge commit arrives as sha1. # # Named by the conf_file build parameter in .ci/proj_jjb.yaml. --- -job: cloudai-pr-coverage +job: cloudai-ci-coverage failFast: false timeout_minutes: 60 diff --git a/.ci/pipeline/coverity_matrix.yaml b/.ci/pipeline/coverity_matrix.yaml index eeca3a22a..b96754ce0 100644 --- a/.ci/pipeline/coverity_matrix.yaml +++ b/.ci/pipeline/coverity_matrix.yaml @@ -1,11 +1,11 @@ -# CloudAI PR pipeline - Coverity scan (ci-demo), run by the cloudai-pr-coverity job. +# CloudAI CI pipeline - Coverity scan (ci-demo), run by the cloudai-ci-coverity job. # # pipeline-scm has already checked the pull request out into ${WORKSPACE}; # the merge commit arrives as sha1. # # Named by the conf_file build parameter in .ci/proj_jjb.yaml. --- -job: cloudai-pr-coverity +job: cloudai-ci-coverity failFast: false timeout_minutes: 60 diff --git a/.ci/pipeline/secrets_matrix.yaml b/.ci/pipeline/secrets_matrix.yaml index 846a1484d..fa3f66b82 100644 --- a/.ci/pipeline/secrets_matrix.yaml +++ b/.ci/pipeline/secrets_matrix.yaml @@ -1,11 +1,11 @@ -# CloudAI PR pipeline - secret scan (ci-demo), run by the cloudai-pr-secrets job. +# CloudAI CI pipeline - secret scan (ci-demo), run by the cloudai-ci-secrets job. # # pipeline-scm has already checked the pull request out into ${WORKSPACE}; # the merge commit arrives as sha1. # # Named by the conf_file build parameter in .ci/proj_jjb.yaml. --- -job: cloudai-pr-secrets +job: cloudai-ci-secrets failFast: false timeout_minutes: 60 diff --git a/.ci/proj_jjb.yaml b/.ci/proj_jjb.yaml index 595da0ceb..8c5ae3f1d 100644 --- a/.ci/proj_jjb.yaml +++ b/.ci/proj_jjb.yaml @@ -5,28 +5,28 @@ # Changes will be overwritten on the next jenkins-jobs update according to this file. # # Each pipeline gets its own subfolder of CloudAI and its own job prefix: -# CloudAI/PR/cloudai-pr - launcher; Blossom triggers it, it runs the +# CloudAI/CI/cloudai-ci - launcher; Blossom triggers it, it runs the # leaves below (.ci/Jenkinsfile.launcher) -# CloudAI/PR/cloudai-pr- - one small pipeline per .ci/pipeline/_matrix.yaml +# CloudAI/CI/cloudai-ci- - one small pipeline per .ci/pipeline/_matrix.yaml # (.ci/Jenkinsfile, ci-demo) # # The CloudAI folder itself and the release job are still declared outside this # repository, so neither is redeclared here - doing so would overwrite them on # every update from this side. -# The PR pipeline's subfolder. Nested under CloudAI so its jobs inherit that +# The CI pipeline's subfolder. Nested under CloudAI so its jobs inherit that # folder's credentials and permissions. - job-template: - name: "{jjb_folder}/PR" + name: "{jjb_folder}/CI" project-type: folder description: &description |

Do NOT edit this job through the Web GUI !

Source of truth: cloudai/.ci/proj_jjb.yaml. Edit there and re-run jenkins-jobs update.

- job-template: - name: "{jjb_proj}-pr" + name: "{jjb_proj}-ci" project-type: pipeline - folder: "{jjb_folder}/PR" + folder: "{jjb_folder}/CI" concurrent: true properties: - github: @@ -37,7 +37,7 @@ - inject: keep-system-variables: true properties-content: | - jjb_proj={jjb_proj}-pr + jjb_proj={jjb_proj}-ci description: *description parameters: - string: @@ -48,17 +48,13 @@ name: "githubData" default: "" description: "[blossom] Raw webhook payload; set automatically on PR-triggered runs. Leave empty for a manual run - the pipeline then skips reporting commit status. Paste a captured payload only to replay a build onto a specific PR." - - string: - name: "script" - default: ".ci/Jenkinsfile.launcher" - description: "[tools] Jenkinsfile that starts the leaf jobs" triggers: - generic-webhook-trigger: # Also the right-hand side of the CI_SERVER GitHub secret: - # @cloudai-pr. These tokens are matched across the + # @cloudai-ci. These tokens are matched across the # whole instance rather than per folder, so two jobs sharing one # would both fire - keep it unique. - token: "{jjb_proj}-pr" + token: "{jjb_proj}-ci" print-contrib-var: true print-post-content: true silent-response: false @@ -72,8 +68,8 @@ - type: JSONPath key: sha1 value: '$.concat("refs/pull/", $.issue.number, "/merge")' - pipeline-scm: &pipeline_scm - scm: + pipeline-scm: + scm: &scm - git: url: "{jjb_git}" credentials-id: "{jjb_gh_cred_id}" @@ -85,14 +81,14 @@ browser: githubweb browser-url: "{jjb_github_url}" wipe-workspace: true - script-path: "$script" + script-path: ".ci/Jenkinsfile.launcher" # A leaf job. No trigger of its own: only the launcher above starts it, passing # down the commit to test and the Blossom payload so it can report status. - job-template: - name: "{jjb_proj}-pr-{leaf}" + name: "{jjb_proj}-ci-{leaf}" project-type: pipeline - folder: "{jjb_folder}/PR" + folder: "{jjb_folder}/CI" concurrent: true properties: - github: @@ -103,7 +99,7 @@ - inject: keep-system-variables: true properties-content: | - jjb_proj={jjb_proj}-pr-{leaf} + jjb_proj={jjb_proj}-ci-{leaf} description: *description parameters: - string: @@ -118,15 +114,13 @@ name: "conf_file" default: ".ci/pipeline/{leaf}_matrix.yaml" description: "[tools] CI Demo job matrix" - - string: - name: "script" - default: ".ci/Jenkinsfile" - description: "[tools] CI Demo Jenkinsfile to initialise job" - pipeline-scm: *pipeline_scm + pipeline-scm: + scm: *scm + script-path: ".ci/Jenkinsfile" # One project per pipeline, so each can be added or moved on its own. - project: - name: cloudai-pr + name: cloudai-ci jjb_proj: 'cloudai' jjb_folder: 'CloudAI' jjb_branch: 'main' @@ -138,13 +132,13 @@ # the other jobs on this instance already use. jjb_gh_cred_id: 'svcnbu-swx-hpcx_ssh_with_key' jobs: - - "{jjb_folder}/PR" - - "{jjb_proj}-pr" - - "{jjb_proj}-pr-{leaf}": + - "{jjb_folder}/CI" + - "{jjb_proj}-ci" + - "{jjb_proj}-ci-{leaf}": leaf: copyright - - "{jjb_proj}-pr-{leaf}": + - "{jjb_proj}-ci-{leaf}": leaf: secrets - - "{jjb_proj}-pr-{leaf}": + - "{jjb_proj}-ci-{leaf}": leaf: coverity - - "{jjb_proj}-pr-{leaf}": + - "{jjb_proj}-ci-{leaf}": leaf: coverage From 5c63a37f2537c45ccd83c030f9644b71d9755531 Mon Sep 17 00:00:00 2001 From: Or Balayla Date: Sun, 4 Oct 2026 13:24:58 +0300 Subject: [PATCH 6/6] ci: load ci-demo through the Jenkins global library Use the ci-demo library the Jenkins instance defines instead of pulling a GitHub ref through pipeline-github-lib. The stable_hpcx ref was copied from cloudaix. On hpcx-01 the global library resolves to ci-demo@master. Refs: HPCINFRA-4889 Signed-off-by: Or Balayla --- .ci/Jenkinsfile | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.ci/Jenkinsfile b/.ci/Jenkinsfile index 2b1b9f05c..cf930224a 100644 --- a/.ci/Jenkinsfile +++ b/.ci/Jenkinsfile @@ -1,8 +1,7 @@ #!/usr/bin/groovy -// load pipeline functions -// Requires pipeline-github-lib plugin to load library from github -@Library('github.com/Mellanox/ci-demo@stable_hpcx') +// ci-demo pipeline functions, from the library the Jenkins instance defines. +@Library('ci-demo') // Blossom's helper, used to report commit status back to the pull request. @Library('blossom-github-lib@master')