Skip to content

[ci] Surface Anyscale logs for failed jobs - #2045

Open
bvolpato wants to merge 1 commit into
NovaSky-AI:mainfrom
bvolpato:bvolpato/anyscale-failure-logs
Open

[ci] Surface Anyscale logs for failed jobs#2045
bvolpato wants to merge 1 commit into
NovaSky-AI:mainfrom
bvolpato:bvolpato/anyscale-failure-logs

Conversation

@bvolpato

@bvolpato bvolpato commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Print a bounded remote log tail when an Anyscale job fails after its entrypoint starts.
  • Keep log retrieval best-effort so diagnostics cannot mask the original job result.
  • Allow the line count to be tuned with ANYSCALE_LOG_TAIL_LINES; default invalid values to 100 lines with a warning.

Why

Scheduled GPU failures currently stop at the Anyscale terminal state. Test output remains in Anyscale, so GitHub CI cannot distinguish a product regression from a remote environment failure. This adds failure context to the existing workflow log without changing successful jobs or capacity retry decisions.

Validation

  • Mocked real-failure path: prints remote tail and exits 1.
  • Mocked log-fetch failure: keeps original exit 1.
  • Mocked capacity path: preserves retry behavior and does not print a misleading test tail.
  • Mocked invalid line-count override: warns, falls back to 100, prints the failure tail, and exits 1.
  • bash -n ci/submit_anyscale_job.sh
  • ShellCheck
  • gitleaks pre-commit hook
  • uv lock --check
  • git diff --check

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds best-effort remote log tail printing to the Anyscale job submission script when a job fails after its entrypoint has started. This is achieved by introducing a new print_remote_log_tail function and a configurable ANYSCALE_LOG_TAIL_LINES environment variable. The review feedback suggests validating that LOG_TAIL_LINES is a valid positive integer to prevent command failures, and appending || true to the print_remote_log_tail call to ensure that any unexpected failure during log retrieval does not interfere with the script's final exit status.

Comment thread ci/submit_anyscale_job.sh
Comment thread ci/submit_anyscale_job.sh Outdated
- Fetch a bounded remote tail after real failures and timeouts.

- Keep log retrieval best-effort and preserve capacity retry decisions.
@bvolpato
bvolpato force-pushed the bvolpato/anyscale-failure-logs branch from 3a23ed8 to e263a57 Compare August 16, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant