Ride out a CRAB status response that cannot be read

[kandrosov]

What the message actually means

The 4763 errors are not the new-task case. law already handles that: when the per-job JSON is
missing it accepts the response if the CRAB server status is one of HOLDING/NEW/QUEUED on command SUBMIT or SUBMITTED, and marks the jobs pending. Here it reported but got 'None' — the value is
literally None, meaning law could not find the Status on the CRAB server line at all. The
response was unreadable, not merely early.

It is transient: both of the running tasks parse cleanly now.

Status on the CRAB server:  SUBMITTED     Status on the scheduler:  RUNNING

Why one bad response costs so much

Two amplifiers in law turn a single hiccup into what you saw:

• query_group maps one group failure onto every job of the task, hence 4763 identical lines;
• any query error makes the poll loop continue (law/workflow/remote.py:1157) — the rest of that
  iteration is abandoned: no status line, no resubmission of the retry jobs, and the other CRAB
  task's perfectly good data thrown away with it. poll_fails consecutive occurrences kill the
  workflow outright.

So with two tasks in flight, one flaky status response wastes the whole poll.

The change

A DSProdCrabJobManager (via law's crab_job_manager_cls hook) that:

1. retries the status query 3 times, 15 s apart — the transient case then disappears entirely
   and nothing is printed;
2. if it still cannot be read, reports the task's jobs as pending — exactly what law does for a
   task with no per-job info yet — and prints one line for the task, not one per job, so the
   poll completes normally and the other task keeps making progress;
3. raises if one task stays unreadable for 10 consecutive polls, because a production that
   quietly stalls is worse than one that stops.

On adding a distinct "new" status

Considered and not done. law's status vocabulary is a fixed set (pending / running / finished /
retry / failed) that the status line and the poll accounting are built on, and a freshly submitted
task legitimately is waiting — pending says that correctly. A sixth bucket would change the
display without touching the actual problem here, which is that an unreadable response is treated as
a fatal per-job error. Tolerating the response fixes the noise and the lost poll iterations together.

Testing

Nine checks against a fake status query:

case	result
fails twice then succeeds	retried, nothing reported, 3 attempts
... leaves no state behind	counter cleared
never succeeds	jobs returned pending, not errors
... attempts bounded	exactly query_retries + 1
... counted once per poll for that task	{task: 1}
job_ids omitted	derived from the project directory
unreadable for 11 consecutive polls	raises, naming the task and the last error
... not before the tolerance is used up	raises only on the 11th
a task that recovers	leaves no state

mkdocs build --strict passes; run_tools/apply_format.sh is a no-op on the diff.

Not yet exercised against a real flaky response — the condition is transient and I cannot
reproduce it on demand. The retry path is what production will exercise first.