Skip to content

Add pre-task baseline verifier check to warn on org contamination - #8

Open
laurahanu wants to merge 1 commit into
SalesforceAIResearch:mainfrom
laurahanu:pre-task-verifier-warning
Open

Add pre-task baseline verifier check to warn on org contamination#8
laurahanu wants to merge 1 commit into
SalesforceAIResearch:mainfrom
laurahanu:pre-task-verifier-warning

Conversation

@laurahanu

Copy link
Copy Markdown

Summary

Runs the SCUBA verifier against the starting org before each task begins and logs a warning if the baseline score is > 0. A clean org should score 0, so a non-zero baseline signals leftover artifacts from a previous run that could inflate the final score. This is a defensive layer to catch contamination before the agent starts.

  • utils.py: shared run_baseline_check() helper; skips QA templates (they need an agent answer) and never raises so a check failure can't break the run
  • args.py: --pretask_baseline_check flag (default on; --no- to disable)
  • main_bu.py: check runs before agent.run()
  • lib_eval_single_task.py: check runs in both CUA workers after env reset

Test plan

  • Run a task on a deliberately dirtied org and confirm the warning fires
  • Confirm QA-template tasks are skipped
  • Confirm --no-pretask_baseline_check disables the check

@salesforce-cla

Copy link
Copy Markdown

Thanks for the contribution! Before we can merge this, we need @laurahanu to sign the Salesforce Inc. Contributor License Agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant