Jenkins: delete build dir and clean workspace after each stage

[causten]

Motivation

Some jobs create 130GB of data and can fill up drive space unnecessarily

Technical Details

Delete the build/ directory in a finally block after each slave's Docker container exits, and call cleanWs() in a top-level post { always } block to wipe the primary agent checkout. Stash and archiveArtifacts for HIP Clang Release still run inside body() before cleanup, so .deb artifacts are preserved.

[pfultz2]

I wonder if its better to do this in the cmake_build function since this is the function that creates the build directory in the first place.

[causten]

Yeah I can make that change.

[pfultz2]

call cleanWs() in a top-level post { always } block to wipe the primary agent checkout.

If we are calling cleanWs() at the end, then why do we need to delete the build directory?

[naveenrajm7]

call cleanWs() in a top-level post { always } block to wipe the primary agent checkout.

If we are calling cleanWs() at the end, then why do we need to delete the build directory?

cleanWs() at pipeline level only wipes the primary agent's workspace. The parallel stage agents (gpu-2, gpu-4, gpu-6, gpu-7) have their own separate workspaces, those are where the 134G build/ dirs accumulate and cleanWs() never reaches them.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #5155      +/-   ##
===========================================
- Coverage    93.26%   93.13%   -0.13%     
===========================================
  Files          623      625       +2     
  Lines        33097    33252     +155     
===========================================
+ Hits         30866    30967     +101     
- Misses        2231     2285      +54     

see 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[causten]

call cleanWs() in a top-level post { always } block to wipe the primary agent checkout.

If we are calling cleanWs() at the end, then why do we need to delete the build directory?

cleanWs() at pipeline level only wipes the primary agent's workspace. The parallel stage agents (gpu-2, gpu-4, gpu-6, gpu-7) have their own separate workspaces, those are where the 134G build/ dirs accumulate and cleanWs() never reaches them.

I do still need the logs to examine any issues. Hopefully cleanWs() doesn't kill the log Artifacts. I'll still need to let it run to completion and validate.