UP-5022: link tasks against pre-consolidation platforms

[claude]

Requested by Tal Erez · Slack thread

Before: with the latest engine against a pre-consolidation platform, linking or creating a task fails. The engine creates two datasets for the task (traces and guardrails) and then posts one model linked to both; the older platform requires an Arthur shield model to be linked to exactly one dataset, so it rejects the model create call with a 400. The engine rolls both datasets back and the job errors out. The practical consequence is that a customer cannot take an engine upgrade without upgrading the platform at the same time.

After: the engine always attempts the consolidated two-dataset shape and, when the platform rejects it with the one-dataset constraint, creates the single-dataset form that platform accepts. Task creation and linking succeed against both old and new platforms. There is no change in behaviour against a consolidated platform: the two-dataset attempt is the only attempt, with the same requests, in the same order, with the same payloads.

How: try-then-fallback, with no version probe. The engine posts the two datasets and the model linked to both. If that model create comes back as a 400 carrying the platform's must be linked to exactly one dataset constraint, the attempt's own rollback deletes the datasets it created, and the engine retries with the single-dataset shape: one dataset, named after the task with no suffix the way it was pre-consolidation, and a model linked only to that. On the create path the raw job spec still carries task_type, which says exactly which dataset the task used to get, so that path reproduces the old shape rather than guessing. Neither the rejection path nor a failed retry leaves orphaned datasets. The retry is logged with the platform's own rejection text and status, so the reason is visible in the job log — on an old platform this fallback is the normal route, not an exception. A 400 that is not the one-dataset constraint is not retried. The cost of dropping the probe is one rejected request on genuinely old platforms.

Open decisions

• Which dataset the link path keeps. On the link path the original task kind cannot be inferred — CreateModelLinkTaskJobSpec carries only job_type, task_id, connector_id and onboarding_identifier, with no field for the task kind, and TaskResponse.is_agentic is now always True — so the fallback dataset type sits behind the module-level constant LEGACY_SINGLE_DATASET_FALLBACK_PROBLEM_TYPE in task_management_job_executors.py, defaulting to the guardrails dataset (ARTHUR_SHIELD). Guardrails or traces is a product decision and is still open. Flipping it to AGENTIC_TRACE is a one-line change to that constant. Review evidence for the decision: linking an agentic task with 724 traces against an old platform made it guardrails-only, so none of those traces were visible until the platform is upgraded (the traces themselves are retained in GenAI). Review also established that on pre-consolidation platforms the UI's link-existing flow asks the user to choose a kind, but that choice is not carried on the link request and so never reaches the engine — which makes this a three-way choice (default to traces, infer the kind from the task, or keep guardrails and accept the gap) rather than a simple default flip.

Resolved during review

• The numeric cutoff was removed. The boundary was first written as 1.4.2592, then corrected in review to 1.4.2594 — the earliest release tag shipping the consolidation commit 18896423, since scope/app_plane/app/operators/models_operator.py is byte-for-byte identical at 1.4.2591-release, 1.4.2592-release and 1.4.2593-release, all three still carrying the must be linked to exactly one dataset guard at lines 390-395. Review then showed that no numeric cutoff can be sound at all. arthur-scope CI sets RELEASE_TAG from the build's git tag stripping only a trailing -release (.gitlab-ci.yml, set-release-tag), and that value is what /api/health reports (scope/app_plane/Dockerfile → scope/app_plane/app/api_models/health.py), so a deploy build reports a string like 1.4.2592-ab71a845-deploy-gcp-united, which the parser reduced to 1.4.2592. Deploy tags are cut off the same version number as the release preceding them: 1.4.2592-ab71a845-deploy-gcp-united and 1.4.2592-283bf583-deploy-gcp-united both contain 18896423, while 1.4.2592-release does not. One version number therefore names platforms of both kinds. The customer's own deploy-gcp-united builds are exactly that case, and under the cutoff would have been sent the single-dataset shape despite accepting two, leaving the task with one of its two datasets until the platform's own migration ran. The probe, the version parser, the cutoff constant and the boundary tests that pinned 2593/2594 are all removed in favour of try-then-fallback.

• Retry log line. The fallback's warning carries the platform's status and its own rejection text, read from the response body only (never str(exc), which also renders response headers), degrading through data.detail → JSON detail → raw body → a fixed message, whitespace-collapsed and truncated at 500 characters. Confirmed working in review against a real pre-consolidation platform.

• Build numbers. App-plane 75039 is 1.4.2594-release and 75411 is 1.4.2603-release (from scope-version-manifest.json at those tags), answered in review. Informational — no longer load-bearing now that no version number is compared.

🤖 Generated with Claude Code

https://claude.ai/code/session_019EPaWvFdzEc4AejyuJV3yv

---

Generated by Claude Code

[kc961163]

When I linked an agentic task with 724 traces against old Scope, this default made it guardrails-only, so none of its traces showed up. The traces are still in GenAI, and a platform upgrade to 1.4.2594 adds the traces dataset back. Is that OK for United until then, or should linking default to traces?

[claude]

This is useful evidence — a concrete task with 724 traces going guardrails-only is exactly what the open decision needed, and I've recorded it in the PR body under "Which dataset the link path keeps".

I'm not changing the default here. It sits behind the single documented constant LEGACY_SINGLE_DATASET_FALLBACK_PROBLEM_TYPE, so switching to AGENTIC_TRACE is a one-line change whenever the call is made. The decision is the PR author's and has been put to them with your evidence attached.

Leaving this thread open until they answer.

---

Generated by Claude Code

[kc961163]

More context for this decision. On pre-consolidation platforms, the link-existing flow in the platform UI lists each task twice, as " - traces" and " - guardrails", so the user explicitly chooses a kind. But the link request carries only task_id (unify-frontend useCreateGenAiModel.ts, calling projectCreateModelLinkTask), and CreateModelLinkTaskJobSpec has no field for it. So the user's choice never reaches the engine, and this constant overrides it every time.

In practice: a user who links an agent task and picks "traces" gets a guardrails-only model. Every later fetch or metrics job reads only inferences, so the task's traces never appear on that platform until it's upgraded.

Options:
(a) default to AGENTIC_TRACE, since most tasks linked after consolidation are agent tasks;
(b) infer the kind from the task itself, e.g. no guardrail rules but traces present → traces;
(c) keep guardrails and accept the gap until the platform upgrades.

[claude]

This changes the shape of the decision, and I've checked the part of it that sits in this repo.

Confirmed: CreateModelLinkTaskJobSpec has no field for the chosen kind. In arthur_common/models/task_job_specs.py it is exactly job_type, task_id, connector_id, onboarding_identifier, and the generated binding agrees — arthur_client/api_bindings/models/create_model_link_task_job_spec.py declares __properties = ["job_type", "task_id", "connector_id", "onboarding_identifier"]. Compare CreateModelTaskJobSpec on the create path, which does still carry the legacy task_type the engine reads. So the engine has no channel through which the user's selection could arrive, and LEGACY_SINGLE_DATASET_FALLBACK_PROBLEM_TYPE is necessarily what decides it. I didn't go into the frontend repo, so the UI half of your description — the duplicated traces/guardrails entries and the hook sending only the task id — I'm taking as reported rather than confirming.

That makes this a three-way choice rather than a default to flip. Your (a) and (c) are one-line changes to the constant; (b) is a different thing — a behaviour change, needing a signal the engine can actually read (task rules and traces off the task itself, not the job spec) and its own correctness discussion about what happens when the signal is ambiguous. Worth separating from (a)/(c) when the call is made.

Still not mine to decide, and unchanged in this push. The option set and your worked example (link an agent task, pick traces, get a guardrails-only model whose traces stay invisible until upgrade) are recorded in the PR body and have been relayed to the PR author. Leaving this thread open.

---

Generated by Claude Code

[kc961163]

Tested against old Scope (1.4.2593, and with the reported version set to 1.4.2582, 1.4.2591 and "unknown"). Creating and linking apps works and leaves no leftover datasets. Not tested: creating a new agentic app, because the current UI no longer sends the agentic flag. Main asks: change the cutoff to 1.4.2594, add a boundary test, and decide guardrails vs traces for linking.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[kc961163]

Re-tested 855a507 end to end against a pre-consolidation Scope release (1.4.2593), driven from that release's own UI:

Create, agentic: a single traces dataset via the version probe, no retry. Traces are ingested; metrics and alert jobs complete; the dashboard shows the data.
Create, GenAI (guardrails): a single guardrails dataset. Inferences are ingested and shown.
Link an existing task: onboards with a single guardrails dataset (see the open decision on the default).
Unreadable version: two datasets tried, rejected, rolled back, retried with one; the log states the platform's reason.
No leftover datasets in any case.

Open items:

Deploy-tag version strings make the numeric cutoff ambiguous around 1.4.2592–1.4.2593 (details in the cutoff thread).
The link-path default overrides the user's explicit traces/guardrails choice (details in the default thread).