Harden the CRAB backend with the DSProd production lessons

[kandrosov]

Summary

Ports the CRAB lessons from the DSProd 115k-job production (cms-flaf/DSProd #5, #6, #7, #11, #16, #17, #18, #19) into FLAF's central CRAB layer, and fixes two related defects on the shared HTCondor path found while auditing against them.

Changes

Status and polling (FLAFCrabJobManager, new)

• An unreadable crab status response is retried (3×, 15 s apart), then that task's jobs are reported pending — one message per task instead of one error per job — and the workflow raises only after 10 consecutive unreadable polls. Previously a single such response failed every job of the task, skipped the whole poll iteration (no resubmission), and killed the workflow after poll_fails = 5 occurrences.
• The parse error now attaches the head of what crab actually returned (the multi-MB JSON is excluded), so the next occurrence is diagnosable.
• CRAB polls now default to 5 minutes: multi-workflow tasks inherit HTCondor's 2-minute poll_interval through the MRO, so the CrabWorkflow class default never won (same trap parallel_jobs had). crab.poll_interval / --poll-interval override.

Credentials and files

• Every crab invocation runs with HOME=$TMPDIR/flaf_crab_home_<uid>: CRAB rewrites ~/.crab3 on every command, and with $HOME on AFS a multi-day production dies the moment the AFS token lapses. A generated crab wrapper additionally runs every subcommand except submit from that home, so crab.log stays out of the working area (submit keeps its cwd — the generated config names scriptExe/inputFiles relative to it).
• The CMSSW sandbox law runs crab in is now built in setup_job_manager — once, with an actionable error — instead of lazily inside each submission, where a failure surfaces as a per-job dummy_job_id / "unknown job id" storm half an hour later. --print-status and completeness checks do not build the sandbox.
• CreateVomsProxy no longer deletes a short-lived proxy at instantiation: on a CRAB worker that deleted the ~23:59 h delegated proxy and failed every remote-storage call in the job. The renewal policy lives in complete(); batch nodes accept any valid delegated proxy and refuse to create one.
• The poll-loop kinit renewal is throttled to hourly and made verbose, so a later credential failure can be diagnosed.

Submission at scale

• The wave gate now aggregates on jobs waiting, not free slots: jobs (retries included) are held while a full wave of refill_fraction × parallel_jobs is still achievable, and released immediately once running + waiting can no longer fill one. The free-slot rule was permanently open in any part-filled pool, so each poll's retry handful became its own tiny CRAB task. --no-poll bypasses the gate (its documented one-shot resubmission must not be parked).
• max_runtime / n_cpus are excluded from req() propagation: a requiring task's resources no longer silently cap what it requires (e.g. a 2 h / 1 CPU plot task capping the 5 h / 2 CPU merge jobs). CLI per-task values still reach that task's branches (workflow↔branch conversion is unaffected), and explicit req(..., max_runtime=...) pins keep working.
• A worker never requires (and can therefore never rebuild and overwrite) a live bundle: _uses_bundles() returns False under LAW_JOB_HOME. Grouped jobs evaluate workflow_requires() on the worker with the forwarded --bundle flag; a transient false-incomplete could previously trigger a rebuild of the tarball other jobs were downloading.

Site selection (run_tools/crab_sites.py, new)

• CRAB gives Site.whitelist precedence over Site.blacklist, so with the default T1_*/T2_*/T3_* globs every configured exclusion was silently void. Excluded sites are now removed from the whitelist itself; a glob covering an excluded site is expanded from the CRIC processing-site list (computeunits entries — cvmfs SITECONF also lists storage endpoints, which the CRAB server refuses). The list is cached 24 h per analysis; CRIC is only consulted when something is actually excluded. Blacklist entries may be globs too.
• Automatic site quarantine: a rolling per-site record of job outcomes (harvested from crab status, keyed by the per-attempt job id — law's poll attaches per-job extra positionally, so site_history in job_data can sit on the wrong job when several CRAB projects are live). A site whose recent jobs mostly fail, measured over jobs sent (ended + in flight) and judged against the other sites' record, is kept out of the next CRAB task. Tunable via crab.auto_blacklist; one black-hole worker node otherwise eats a production at ~250 jobs/hour.
• crab.ignore_global_blacklist (opt-in) waives CMS's own blacklist of known-broken sites.
• If CRIC is unreachable with no cache mid-production, a quarantine-only exclusion degrades with a warning; a user-configured blacklist still aborts (it must not be silently defeated).

Testing

• test/test_crab_hardening.py: 56 unit tests, each encoding a production failure mode (unreadable status, retry parking, wave-gate scenarios incl. the tail, glob blacklists, black-hole quarantine with baseline exclusion, delegated-proxy survival, req() isolation, worker bundle guard).
• Adversarial verification against the installed law 0.1.20: the real BaseRemoteWorkflowProxy.poll()/submit() loop was driven with the new proxy through 600 randomized scenarios (pool sizes 1–50, refill 0.0–1.0, failures, retries, resumed runs, mid-run parallel_jobs changes) — no hangs, outcomes identical to stock law with fewer, larger CRAB tasks.
• Real runs (all to final plots on storage): full run_ci_test.sh chain on CRAB (HH_bbtautau, Run3_2022EE — 5-minute cadence, no crab.log outside /tmp, unsubmitted: 0 throughout, whitelist globs intact) and on HTCondor for HH_bbtautau, HH_bbWW and H_mumu, with per-task resource isolation verified in the generated JDLs (AnaTupleFileTask 40 h/4 CPU, AnaTupleMergeTask 48 h/2 CPU, HistTupleProducerTask 5 h/4 CPU — each its own default, no leakage), plus a HelloWorld CRAB smoke covering the sandbox preflight.
• mkdocs build --strict passes; docs updated in this PR (docs/workflow/crab.md documented the old blacklist semantics backwards, docs/workflow/arguments.md described a refill guarantee the old rule never delivered).

Notes

• The HH_bbWW HTCondor CI currently fails on main independently of this PR: defineControlRegions (HH_bbWW Implement plotting task #111) references TwoStageDNN_M{mp}_* columns that only the TwoStageDNN payload produces, so any config without that payload crashes at HistTuple production. Validated here with that hunk temporarily reverted.
• Known accepted trade-offs, matching the production-proven DSProd behaviour: while a task's status is degraded to pending, law resubmits nothing for it (bounded by the 10-poll tolerance); site-stats records are per analysis and last-writer-wins across concurrent law processes (advisory data); CRAB-internal retry failures are attributed to the final site only (errs toward quarantining later, never falsely).