STAC-25498 Add the arm64 build lane to the GitHub Actions pipeline

[LouisParkin]

Jira: STAC-25498 (sub-task of STAC-25142)

Ports the GitLab agent-arm lane. GitLab fans .gitlab-ci-agent.yml out twice from .gitlab-ci.yml (agent-x86 / agent-arm) with per-arch variables; only the amd64 half had been ported. The arm64-xlarge-public ARC scale set landing on argocd-apps main (STAC-25492) removes the blocker, so that fan-out becomes an arch matrix.

This is a hard prerequisite for the arm64 half of STAC-25457 — merge_docker_manifest_* cannot build a multi-arch manifest from one architecture.

The cache-collision bug this had to fix first

agent-godeps-cache-metadata.sh hashed the module graph plus BASE_IMAGE_TAG. Both datadog_build_linux_x64 and datadog_build_linux_arm64 carry the same tag 7af9194f, so the two arches would have computed an identical cache tag — and one arch's image would have silently satisfied the other's existence check, handing arm64 jobs an amd64 module cache.

Fixed by folding the base image name into the hash and the arch into the tag (godeps-<arch>-<hash>), which makes the collision impossible by construction and keeps the arch readable off the ref.

What changed

File	Change
.github/scripts/agent-godeps-cache-metadata.sh	Requires BASE_IMAGE_NAME + ARCH; hashes base:<name>:<tag>; tag is godeps-<arch>-<hash>
.github/workflows/godeps-cache.yml	Required arch input; selects the base image; builds natively per arch
.github/workflows/build-binaries.yml	arch matrix on build-agent, build-cluster-agent, build-cluster-agent-image; per-arch artifacts
.github/workflows/build-deb.yml	arch matrix on build-deb, test-deb-renaming, build-agent-image; per-arch artifacts
.github/workflows/lint-and-unit-tests.yml	arch: amd64 on its cache call (no arm64 lane, by design)

Design notes

• Native builds, not QEMU. The arm64 cache image is built on an arm64 runner; binfmt emulation would make the Dockerfile's module extraction unusably slow. The arches are independent images, not a multi-arch manifest — consumers reference them from container.image, which resolves against the runner's own architecture anyway.
• fail-fast: false everywhere. An arm64 regression must not abort the amd64 leg; amd64 has been green for weeks and during arm64 bring-up its result is the signal.
• Per-arch artifact names. Matrix legs upload concurrently and actions/upload-artifact v4 rejects a second upload to an existing name, so a shared name would have failed the arm64 leg outright.
• Stagger ownership (STAC-25494). One designated cache builder per arch: lint-and-unit-tests owns amd64 (delay 0), build-binaries owns arm64 (delay 0); every other caller waits 900s.
• test-deb-renaming stays on amd64 for both legs. ar + grep read the arm64 .deb fine, so burning a scarce arm64 XL runner on a grep would buy nothing.
• Both arches run on PRs, matching GitLab, where agent-x86 and agent-arm shared the same commit-or-MR rule.
• GitLab per-arch variables. Only ARCH carries over. PKG_ARCH_SUFFIX is the same value and had exactly one use (selecting the built .deb). PACKAGE_ARCH / DD_PKG_ARCH / STS_PKG_ARCH are set by build_deb but nothing in the StackState omnibus path reads them, and omnibus derives the package architecture from the native build host. DOCKER_ARCH is deliberately not ported — its only consumer is publish_image.sh. Note for STAC-25457: on arm64 it is aarch64, not arm64, so it cannot simply be aliased to matrix.arch.

Not in scope

Lint and unit tests gain no arm64 lane — GitLab ran them in both child pipelines, but the directive is to run each suite once on the path to master.

Validation

• actionlint v1.7.7 — only the expected unknown-self-hosted-label notes (docker-public, xlarge-public, arm64-xlarge-public); no expression, needs or matrix errors
• zizmor v1.28.0 — no findings
• shellcheck — clean
• Runner label verified against argocd-apps main: runnerScaleSetName: arm64-xlarge-public, runnerGroup: public, nodeSelector kubernetes.io/arch: arm64, DinD sidecar present

[LouisParkin]

First CI run — arm64 lane is live and green

Binary builds: completed / success, all 10 jobs, both architectures.

[success] Go dependency cache image (amd64) / Compute amd64 cache image tag
[success] Go dependency cache image (arm64) / Compute arm64 cache image tag
[success] Go dependency cache image (amd64) / Build and publish missing amd64 cache image
[success] Go dependency cache image (arm64) / Build and publish missing arm64 cache image
[success] Build agent binary (branded / StackState, amd64)
[success] Build agent binary (branded / StackState, arm64)
[success] Build cluster-agent binary (branded / StackState, amd64)
[success] Build cluster-agent binary (branded / StackState, arm64)
[success] Build cluster-agent container image (docker build, no push on PR, amd64)
[success] Build cluster-agent container image (docker build, no push on PR, arm64)

What that confirms end to end:

• The arm64-xlarge-public scale set schedules — the arm64 jobs picked up runners rather than queueing forever, which was the main label risk.
• The arm64 cache image builds natively on an arm64 runner and publishes to quay.
• The two arches computed distinct cache tags, so the 7af9194f shared-tag collision is genuinely resolved — each arch pulled its own image as container.image and started.
• The arm64 agent and cluster-agent binaries compile, and the arm64 cluster-agent image builds and passes its runtime smoke test on the arm64 DinD sidecar.
• The STAC-25494 stagger behaved: delay-0 designated builders built, delay-900 callers polled and exited early once the image appeared.

Both arches rebuilt their cache image on this run, as expected — the tag scheme changed (base image name in the hash, arch in the tag), so no pre-existing image matched. One-time cost.

Still running: DEB package build (cold omnibus, both arches — the long pole) and Lint and unit tests. Will follow up when they land.

[LouisParkin]

Rebased on the integration branch — and it needed a fix

Merged STAC-25142-agent-lint-unit (now including #449 / STAC-25461) into this branch. Git reported zero conflicts, but the merged result was broken.

#449 added a terminal cerberus-notify job to build-binaries.yml and build-deb.yml with:

needs:
  - godeps-cache

This PR splits that job into godeps-cache-amd64 / godeps-cache-arm64. The two changes touch different regions of the file, so the textual merge succeeded — and produced a workflow that will not parse:

build-binaries.yml:335:3: job "cerberus-notify" needs job "godeps-cache" which does not exist in this workflow [job-needs]
build-deb.yml:407:3:      job "cerberus-notify" needs job "godeps-cache" which does not exist in this workflow [job-needs]

Fixed in 41f575b by listing both per-arch jobs. That also preserves #449's stated intent — needs lists every job rather than just the leaves, so an early cache failure that skips its dependents still reaches Slack.

Worth noting as a merge-order hazard for the remaining migration PRs: a semantically clean git merge is not evidence that the workflow graph is intact. actionlint catches this class immediately.

Side effect: arm64 now covers the release branch too

#449 also added push: branches: [stackstate-7.78.2] to these workflows. Combined with this PR, arm64 now runs on every PR and on release-branch pushes — full GitLab parity plus release coverage, and any arm64 failure on the release branch reports to Slack via Cerberus.

Re-validated after the merge: actionlint clean (only the expected unknown-self-hosted-label notes), zizmor no findings.