ci: split the PR pipeline into a launcher and small jobs

.ci/Jenkinsfile

@@ -1,8 +1,7 @@
 #!/usr/bin/groovy
 
-// load pipeline functions
-// Requires pipeline-github-lib plugin to load library from github
-@Library('github.com/Mellanox/ci-demo@stable_hpcx')
+// ci-demo pipeline functions, from the library the Jenkins instance defines.
+@Library('ci-demo')
 // Blossom's helper, used to report commit status back to the pull request.
 @Library('blossom-github-lib@master')
 

.ci/Jenkinsfile.launcher

@@ -0,0 +1,82 @@
+#!/usr/bin/groovy
+
+// Entry point of the CI pipeline. Blossom triggers this job; it runs the small
+// leaf jobs below in parallel, each of which runs .ci/Jenkinsfile with its own
+// matrix from .ci/pipeline/ and reports its own commit status.
+
+// Blossom's helper, used to report commit status back to the pull request.
+@Library('blossom-github-lib@master')
+
+import ipp.blossom.*
+
+// Leaf jobs, keyed by parallel-branch name. Declared in .ci/proj_jjb.yaml.
+def leafJobs = [
+    'copyright': 'cloudai-ci-copyright',
+    'secrets':   'cloudai-ci-secrets',
+    'coverity':  'cloudai-ci-coverity',
+    'coverage':  'cloudai-ci-coverage',
+]
+
+def githubHelper = null
+def pipelineOverviewUrl = ""
+// A manual run hands its own sha1 to every leaf.
+def sha1 = params.sha1
+
+// Injected by the Blossom workflow for PR-triggered runs; absent when a
+// maintainer starts the job by hand, so every use is guarded.
+if (params.githubData) {
+    withCredentials([usernamePassword(credentialsId: 'svcnbu-swx-cloudai-github-token',
+                                      passwordVariable: 'GIT_PASSWORD',
+                                      usernameVariable: 'GIT_USERNAME')]) {
+        githubHelper = GithubHelper.getInstance("${GIT_PASSWORD}", params.githubData)
+    }
+    pipelineOverviewUrl = "${env.BUILD_URL}/pipeline-overview/"
+    // No context: the launcher reports on blossom-ci, the status the Blossom
+    // workflow leaves pending, so the PR shows the overall result there.
+    githubHelper.updateCommitStatus(pipelineOverviewUrl, "CI started",
+                                    GitHubCommitState.PENDING)
+    currentBuild.description = githubHelper.getBuildDescription()
+}
+
+try {
+    if (params.githubData) {
+        // Resolve refs/pull/<n>/merge once, so every leaf tests the same
+        // commit even if the base branch moves while they start. Inside the
+        // try, so a failed lookup still closes the pending status.
+        sha1 = githubHelper.getMergedSHA()
+        if (!sha1) {
+            error "Could not resolve the merge commit of the pull request"
+        }
+    }
+
+    // Each leaf reports its own status, so a failed leaf only paints its
+    // branch red: blossom-ci fails only if dispatching breaks, and re-running
+    // just that leaf clears the PR. propagate: false keeps a missing leaf
+    // job fatal, unlike catching every error.
+    parallel(leafJobs.collectEntries { branch, job ->
+        [(branch): {
+            def run = build job: job, propagate: false, parameters: [
+                string(name: 'sha1', value: sha1),
+                string(name: 'githubData', value: params.githubData),
+            ]
+            if (run.result != 'SUCCESS') {
+                catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
+                    error "${job} #${run.number} finished ${run.result}"
+                }
+            }
+        }]
+    })
+} catch (Exception ex) {
+    currentBuild.result = 'FAILURE'
+    println ex
+    throw ex
+} finally {
+    if (params.githubData) {
+        githubHelper.updateCommitStatus(
+            pipelineOverviewUrl,
+            "CI completed",
+            currentBuild.resultIsBetterOrEqualTo('SUCCESS') ? GitHubCommitState.SUCCESS
+                                                            : GitHubCommitState.FAILURE
+        )
+    }
+}

.ci/README.md

@@ -1,51 +1,68 @@
-# CloudAI PR CI
+# CloudAI Jenkins pipelines
 
-Blossom Jenkins pipeline that runs on every pull request.
+Blossom Jenkins pipelines for CloudAI. Each pipeline gets its own subfolder of
+the `CloudAI` Jenkins folder and its own job prefix:
+
+| Pipeline | Jenkins jobs | Trigger |
+| --- | --- | --- |
+| CI | `CloudAI/CI/cloudai-ci*` | `/build` comment on a pull request |
+
+A new pipeline `<name>` follows the same scheme: `CloudAI/<Name>/` and the
+`cloudai-<name>` prefix.
 
 | File | Role |
 | --- | --- |
-| `proj_jjb.yaml` | Job definition, applied with `jenkins-jobs update` |
-| `Jenkinsfile` | Thin entry point; reports commit status, hands off to ci-demo |
-| `job_matrix.yaml` | The stages that actually run |
+| `proj_jjb.yaml` | Job definitions, applied with `jenkins-jobs update` |
+| `Jenkinsfile` | Runs one ci-demo matrix; reports commit status when started for a PR |
+| `Jenkinsfile.launcher` | CI pipeline launcher; runs the leaf jobs in parallel |
+| `pipeline/<leaf>_matrix.yaml` | The stage one leaf job runs |
 | `header-check.yml` | Copyright-header policy for `header_check.py` |
 
-Resulting job: `CloudAI` / `cloudai-ci`, alongside the existing
-`cloudai-release` job.
+## CI pipeline
+
+Resulting jobs, in `CloudAI/CI`: the launcher `cloudai-ci` and one leaf job
+`cloudai-ci-<leaf>` per matrix in `pipeline/`.
 
-## Flow
+### Flow
 
 `.github/workflows/blossom-ci.yml` authorizes the requester, runs Blossom's own
-vulnerability scan, then triggers this job via the `CI_SERVER` secret
-(`<jenkins-url>@cloudai-ci`). Jenkins checks out the PR merged into its base
-branch and runs the matrix.
+vulnerability scan, then triggers the launcher via the `CI_SERVER` secret
+(`<jenkins-url>@cloudai-ci`). The launcher resolves the PR merged into its base
+branch to one commit and starts every leaf job on it in parallel. Each leaf
+reports its own `cloudai-ci-<leaf>` commit status on the PR, which is where a
+failure shows. The launcher closes `blossom-ci`, the status the workflow leaves
+pending, and fails it only if dispatching breaks. A failed leaf can be re-run
+on its own in Jenkins (same parameters); its status then updates the PR.
 
-## Stages
+### Stages
 
-- **Check copyrights** — `header_check.py` against `header-check.yml`, limited
-  to files touched this calendar year.
-- **Secret scan** — `secret_scan.py` over the working tree.
-- **Coverity scan** — `--all-security`, reports only. Gating on findings needs
-  a Coverity stream to diff against; without one every pre-existing defect
-  would block every PR.
-- **Coverage** — `pytest --cov`. Containers run as root, so the two
-  permission-sensitive tests are run separately with their exit code inverted.
+One leaf job per stage, each with its own matrix in `pipeline/`:
+
+| Leaf job | Stage |
+| --- | --- |
+| `cloudai-ci-copyright` | **Check copyrights** — `header_check.py` against `header-check.yml`, limited to files touched this calendar year. |
+| `cloudai-ci-secrets` | **Secret scan** — `secret_scan.py` over the working tree. |
+| `cloudai-ci-coverity` | **Coverity scan** — `--all-security`; fails on any defect, since the codebase reports zero. |
+| `cloudai-ci-coverage` | **Coverage** — `pytest --cov`. Containers run as root, so the two permission-sensitive tests are run separately with their exit code inverted. |
 
 Blackduck and the antivirus scan stay in the release pipeline: Blossom already
-Blackducks the PR before this job starts, and the antivirus stage only acts on
-published release tarballs.
+Blackducks the PR before this pipeline starts, and the antivirus stage only
+acts on published release tarballs.
 
 GitHub Actions (`.github/workflows/ci.yml`) still owns linting, type checking,
 the docs build and the install smoke test.
 
 ## Changing CI
 
-Everything this job does lives in `job_matrix.yaml`, so changing CI behaviour
-is an ordinary pull request. Only three things need Jenkins-side action, and
-all are one-time:
+Everything the leaf jobs do lives in their matrices, so changing what an
+existing leaf runs is an ordinary pull request. Only three things need
+Jenkins-side action, and all are one-time:
 
-1. `jenkins-jobs update .ci/proj_jjb.yaml` to create or update the job
+1. `jenkins-jobs update .ci/proj_jjb.yaml` to create or update the jobs. A new
+   leaf needs this: add `pipeline/<leaf>_matrix.yaml`, list the leaf under
+   `jobs` in `proj_jjb.yaml` and in `Jenkinsfile.launcher`, then update.
 2. installing credentials
 3. adding namespace egress rules when a stage needs a new host
 
-Do not edit the job through the Jenkins web UI — changes are overwritten on the
-next update.
+Do not edit the jobs through the Jenkins web UI — changes are overwritten on
+the next update.

.ci/header-check.yml

@@ -11,7 +11,7 @@ general:
     # Jenkins pipeline definitions carry no license header by convention.
     # The checker fails on any file it has no rule for, so this must be an
     # explicit exclusion rather than simply absent from the include list.
-    - "Jenkinsfile$"
+    - "Jenkinsfile(\\..+)?$"
 
 apache-2.0:
   validate-spdx-license: true