[API] Support background experiment execution

[podkidyshev]

Summary

• Add wait=False to return a pending experiment and execute it in a separate worker process.
• Persist the worker request and controller log in the experiment directory. Record setup and execution failures in experiment output.
• Extend existing API tests and documentation for background execution. Synchronous execution remains the default.

Test Plan

• Automated CI.
• Manual macOS test: launched two one-second standalone Sleep experiments from a short-lived Python process, using Path inputs for one and TOML text for the other. Both returned pending snapshots. After the caller exited, a separate process verified both completed, each with one successful run, timestamps, controller log, and saved request.

PASS: both background workers completed after caller exit; one successful run and timestamps each

Additional Notes

Based on #1061. This is the final split of the original API implementation. Workers load installed CloudAI plugins in a fresh process; caller-only in-memory registrations are not transferred.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}