Skip to content

workflows: start and inform a review from the @codex review comment - #30

Merged
idy merged 3 commits into
mainfrom
claude/issue-29-comment-trigger
Sep 3, 2026
Merged

workflows: start and inform a review from the @codex review comment#30
idy merged 3 commits into
mainfrom
claude/issue-29-comment-trigger

Conversation

@idy

@idy idy commented Sep 3, 2026

Copy link
Copy Markdown
Member

Result

Two defects on the same path — the PR comment that asks for a review. It now
starts a review when the mention sits on its own line anywhere in the comment,
and its content actually reaches the code review turn.

1. The comment did not start a review

resolve gated issue_comment events with
/^@codex(?:\s+review(?:\s+[\s\S]+)?)?\s*$/i against the whole trimmed comment.
The ^ anchor meant the comment had to begin with @codex. A comment that
described what was pushed and then ended with @codex review never matched, so
eligible was false and start, review, publish, and finalize were all
skipped. Nothing failed and nothing was posted back, so from the requester's
side the workflow looked like it simply had not run.

Verified on GizClaw/gizclaw#1084: runs 33785352754, 33785379065, and
33786877584 each show review / resolve succeeding with review / start
skipped. The PR was open and not a draft at the time and issue_comment is a
trusted base event, so the trigger expression is the only remaining cause. Both
real comment bodies were replayed through the old and new predicates — old
false, new true. Two pushes went unreviewed.

The predicate moved out of the inline github-script block into
.github/scripts/review-request/, alongside issue-review, pr-readiness,
and pr-review.

stripQuotedText blanks the regions where the trigger is quoted rather than
requested — fenced code blocks (backtick or tilde, including an unterminated
one), inline code spans, and block quotes. Lines are replaced with empty lines
rather than removed, so stripping cannot join two unrelated lines into one
apparent command. This PR's own description quotes the trigger and must not
start a review because of it.

REVIEW_REQUEST_COMMAND is /^ {0,3}@codex(?:[ \t]+review\b.*?)?[ \t]*$/im.
The m flag lets the request sit on any line. Requiring the whole line to be
the command is the deliberate tightening decision: a mid-sentence mention
does not start a review
, and neither does @codex please look or
@codex reviewing. At most three leading spaces are accepted, CommonMark's
indented-code-block boundary, so a four-space-indented mention is inert too.
Bare @codex keeps working unchanged. isReviewRequestComment also rejects a
comment whose author type is Bot or whose login ends in [bot]; that guard
did not exist before.

resolve gains two issue_comment-only steps: a sparse checkout of the module
from job.workflow_repository at job.workflow_sha — the same pinning review
and finalize already use, so the trigger logic always matches the workflow
revision that is running — and a node evaluate.mjs step that receives the
comment through the environment and sets a requested output. The comment body
is never interpolated into a shell command.

2. The comment never reached the reviewer

Fixing the trigger alone would have started a review that still could not see
the comment. The discussion step already collected the last 20 PR comments and
trigger_comment_id into PR_CONTEXT_FILE, but nothing read
context.comments
: prStageSnapshot() sends only repository, number, title,
body, and linked Issues; the Issue stage sends Issue snapshots; the code stage
sends the diff chunk and Issue context. It was write-only data. The code-stage
prompt already declared "every ... discussion comment" untrusted input, so the
wiring was intended and never completed.

run.mjs now writes a code-discussion stage input beside the existing
code-linked-issues one. The first code chunk turn reads it after the Issue
context, and an aggregation turn that runs without a preceding code turn reads
it directly. Per-comment clipping rises from 600 to 2,000 characters and the
triggering comment gets 8,000 — the two real #1084 comments are 1,180 and 899
characters, so the old limit truncated both mid-sentence and would have cut off
the 验证:… validation summary entirely. Comments now carry id,
is_trigger, author_is_bot, and body_truncated.

Three constraints hold:

  • Untrusted. Any commenter can write it. The prompt frames it as usable
    only for author-stated intent, validation claims, disclosures, and request
    focus; it cannot relax, override, or extend the trusted caller review
    profile; every claim must be checked against the diff, and a claim the diff
    contradicts is reported.
  • Not instructions. The trigger itself remains a pure eligibility gate.
  • Out of every stage identity. Discussion is deliberately excluded from
    prStageSnapshot() and the code stage identity. Including it would let any
    passing comment invalidate cached evidence and force a paid re-review, which
    would contradict the documented "unchanged head → reused, zero model tokens"
    behavior. The consequence, documented in the README: a comment on an
    unchanged head still reuses evidence and its text is not read until a code
    turn runs for another reason. That is the real case here — both #1084
    comments followed a push, so the code stage runs incrementally and sees them.

Validation

node .github/scripts/review-request/test.mjs   # review-request tests passed
node .github/scripts/issue-review/test.mjs     # issue-review tests passed
node .github/scripts/pr-readiness/test.mjs     # pr-readiness tests passed
node .github/scripts/pr-review/test.mjs        # pr-review scripts: ok
actionlint .github/workflows/*.yml             # 8 findings, all job.workflow_* (see below)

review-request/test.mjs is a 22-case table over comment bodies: bare
@codex, @codex review, @codex review <focus>, mixed case, surrounding
whitespace, explanation text followed by the command on its own line, the
command followed by trailing prose, a three-space indent, and a fence that
closes before a real command — all accepted. Mid-sentence mention,
@codex when you can, @codex reviewing, backtick and tilde fences, an
unterminated fence, single- and double-backtick inline spans, a four-space
indented block, a block quote, unrelated text, an empty body, a Bot author, a
[bot] login, and missing or non-string bodies — all rejected.

pr-review/test.mjs gains assertions that the discussion input file is written
and loaded by both code turns, that the trigger comment keeps the larger
budget, that the untrusted framing is present, and — the load-bearing one —
that neither prStageSnapshot() nor the code stage identity mentions comments.
Every new assertion was checked against the pre-change sources and fails there,
so none is vacuous.

Both real #1084 comment bodies were replayed through the new trigger predicate
and the new collection mapping: accepted, marked, and carried untruncated.

actionlint reports job.workflow_repository / job.workflow_sha as undefined;
that is a stale context type in actionlint. Six of the eight already exist on
main for the identical checkout steps in review and finalize; the other
two are the same finding for the new resolve checkout.

Not verified: no end-to-end run against a live PR. The changes are covered by
the tests above and by replaying real payloads, not by an executed review.

Reaching consumers

latest is a moving lightweight tag in this repository, not a release and
not a branch — refs/tags/latest currently points at 7794db9, the tip of
main. Consumers such as GizClaw/gizclaw pin @latest, so merging this PR is
not enough on its own: after the squash-merge, latest must be force-moved to
the new main tip and force-pushed:

git tag -f latest main && git push -f origin latest

Once that lands, GizClaw/gizclaw#1084 picks the fix up with no change on the
consumer side. v1 is an annotated tag at an older commit and is untouched by
this PR.

Review round 1 (a24205a)

Three blockers from the automated review, all real:

  • Multi-line inline code spans (P1). CommonMark lets a code span cross line
    breaks, but the stripper ran line by line, so `example\n@codex review\nexample`
    left the middle line intact and started a review. stripInlineCode now runs
    over the whole comment, blanking every character of a span while preserving
    newlines. The closing-delimiter search stops at the paragraph break — where
    CommonMark ends a span — so one stray unmatched backtick cannot blank a later
    request. New cases: multi-line single- and double-backtick spans (rejected), a
    span closed on a later line followed by a real request (accepted), unmatched
    backticks in the same and an earlier paragraph (accepted).
  • Trigger comment crowded out (P2). The 8,000-character budget was applied
    after comments.slice(-20), so 20 newer comments dropped the request itself.
    The trigger is now looked up across all comments and prepended when missing,
    deduplicated by id.
  • Issue scope wording (pr-format). Issue bug: Start and inform a review from the @codex review comment #29's non-goals said "do not use
    the comment body as review instructions", which read as contradicting the
    goal of passing request focus to the code turn. Reworded: focus reaches the
    code turn only as untrusted background checked against the diff, never as
    trusted instructions; trigger matching itself still parses no parameters.

Testing got stronger rather than wider. pr-review/test.mjs now compiles all 8
inline github-script blocks (a structural break in one is invisible to
actionlint and only fails mid-review at run time — this caught a duplicated
line while writing the fix) and executes the real discussion script against
synthetic comments, asserting selection, retention, deduplication, clipping
budgets, truncation flags, and bot marking. The weaker source-pattern
assertions those replaced were deleted. Every new assertion was checked against
the pre-fix sources and fails there.

Closes #29

🤖 Generated with Claude Code

The resolve gate anchored the trigger to the start of the whole comment, so a
comment that explained the push and ended with `@codex review` on its own line
was silently ineligible: no error, no reaction, and every downstream job
skipped.

- Match the request on its own line anywhere in the comment body
- Ignore mentions quoted in fenced, inline, indented, or block-quoted text
- Reject comments authored by an app
- Move the predicate into a tested review-request module the resolve job pins

Closes #29

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ OpenAI PR Review: FAIL

Conclusion: Not ready. 2 readiness blockers and 1 actionable finding must be resolved before merge.

Review checks

Check Result
PR format ✅ PASS
Issue design ✅ PASS
Code & plan conformance ❌ FAIL

Scope: 55aaac03c6 · 7794db9a80..55aaac03c6 · full · 1 diff chunk

Usage: 1m 7s · 295,829 tokens · 75.7% cache hit · 7.377 credits

Blockers

  • plan-conformance The linked Issue requires mentions inside inline code spans to remain ineligible. The implementation only strips inline code up to a newline, so a multi-line inline code span containing @codex review is accepted.
  • code-review .github/scripts/review-request/common.mjs:10: Reject commands inside multi-line inline code spans

Findings

1 inline finding published.

Summary

The new trigger module leaves multi-line inline code spans unstripped, allowing a quoted command to start a review.

Review metadata
  • Commit: 55aaac03c6
  • Range: 7794db9a80..55aaac03c6
  • Mode: full
  • Diff chunks: 1
  • Model: gpt-5.6-terra
  • Reasoning effort: medium
  • Session: repo-1309321116-pr-30-v2
  • Generation: 13a4614cfdcf0458f01b4243afccb54ab979d1e5048c4fe2131d6b2fc030d000
  • Evidence: a4498a7d3d659ed24cb3d7d8001998ef8388fdcef2fa06d7a1eed5267515d321

Totals

  • Input: 291,667
  • Cached input: 220,669
  • Cache write: 70,962
  • Output: 4,162
  • Reasoning: 1,753
  • Total: 295,829
  • Estimated credits: 7.377
  • Credit rate per 1M tokens: 62.5 input / 6.25 cached / 375 output
Token and cache details
Stage Mode Target Time Input Cached Hit Output Total Credits
pr deterministic format rules 0s 0 0 0.0% 0 0 0.000
pr full pr 10s 30,184 18,888 62.6% 502 30,686 1.012
issue full Issue #29 20s 78,274 59,635 76.2% 1,336 79,610 2.039
code full chunk 1/1 30s 103,350 66,103 64.0% 1,871 105,221 3.443
code full aggregate 7s 79,859 76,043 95.2% 453 80,312 0.884

Comment thread .github/scripts/review-request/common.mjs
The discussion step already collected the last 20 PR comments and the trigger
comment id into the context file, but nothing ever read context.comments: the
PR stage sends only title, body, and linked Issues, and the code stage sends
only the diff and Issue context. The code-stage prompt already declared
discussion comments untrusted input, so the wiring was intended and never
finished. Codex could not see what the author said the push did, what they had
already validated, what they disclosed, or any focus after `@codex review`.

- Write a code-discussion stage input and load it in both code turns
- Keep the triggering comment intact and mark bot authors and truncation
- Frame discussion as untrusted background that cannot change review policy
- Keep discussion out of every stage identity so reuse stays zero-token

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@idy idy changed the title workflows: accept an @codex review mention anywhere in a comment workflows: start and inform a review from the @codex review comment Sep 3, 2026
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

🤖 OpenAI PR review completed with an error

Reason: PR or linked-Issue readiness identity changed before publication.

Open the Actions run.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ OpenAI PR Review: FAIL

Conclusion: Not ready. 5 readiness blockers and 2 actionable findings must be resolved before merge.

Review checks

Check Result
PR format ❌ FAIL
Issue design ✅ PASS
Code & plan conformance ❌ FAIL

Scope: 5a11f45307 · 55aaac03c6..5a11f45307 · incremental · 1 diff chunk

Usage: 1m 0s · 447,167 tokens · 75.7% cache hit · 10.350 credits

Blockers

  • pr-format The PR now delivers PR-comment content to the code-review turn and lets it provide review focus, validation claims, and disclosures. Issue #29 explicitly says the comment body must not be used as review instructions and limits the work to a pure eligibility gate. This is a material expansion beyond the reviewed plan; update the Issue’s scope, design, acceptance criteria, and validation plan (or split the work) before treating the PR as plan-conformant.
  • plan-conformance The linked Issue requires mentions inside inline code spans to remain ineligible. The implementation only strips inline code up to a newline, so a multi-line inline code span containing @codex review is accepted.
  • plan-conformance The linked Issue requires discussion context to identify and retain the triggering comment. Limiting comments to the last 20 before selecting the trigger can omit it when newer comments arrive.
  • code-review .github/scripts/review-request/common.mjs:10: Reject commands inside multi-line inline code spans
  • code-review .github/workflows/codex-openai-review.yml:534: Always include the triggering comment in discussion context

Findings

2 inline findings published.

Summary

Two trigger-related requirements remain unmet: multi-line inline code can invoke a review, and the triggering comment can be dropped from discussion context after later comments arrive.

Review metadata
  • Commit: 5a11f45307
  • Range: 55aaac03c6..5a11f45307
  • Mode: incremental
  • Diff chunks: 1
  • Model: gpt-5.6-terra
  • Reasoning effort: medium
  • Session: repo-1309321116-pr-30-v2
  • Generation: 91f031f53bb26cdda5c38b59538a167a4103d3abc2946af230328b4fd3721eee
  • Evidence: 3cad19b46480a5225388bc5cded064cc14ab8045def8ed3b1ed17bce181b566c

Totals

  • Input: 443,076
  • Cached input: 335,573
  • Cache write: 107,479
  • Output: 4,091
  • Reasoning: 2,085
  • Total: 447,167
  • Estimated credits: 10.350
  • Credit rate per 1M tokens: 62.5 input / 6.25 cached / 375 output
Token and cache details
Stage Mode Target Time Input Cached Hit Output Total Credits
pr deterministic format rules 0s 0 0 0.0% 0 0 0.000
pr incremental pr 16s 86,518 50,351 58.2% 1,009 87,527 2.954
issue incremental Issue #29 11s 102,208 91,480 89.5% 652 102,860 1.487
code incremental chunk 1/1 25s 119,902 65,824 54.9% 1,795 121,697 4.464
code incremental aggregate 8s 134,448 127,918 95.1% 635 135,083 1.446

Comment thread .github/scripts/review-request/common.mjs
Comment thread .github/workflows/codex-openai-review.yml Outdated
CommonMark inline code spans may cross line breaks, but the stripper ran line
by line, so a quoted `example\n@codex review\nexample` left the middle line
intact and started a review. The discussion step also applied the trigger's
larger budget after slicing the window, so 20 newer comments dropped the
request itself.

- Strip inline code spans across line breaks, bounded at the paragraph break
- Restore the triggering comment when newer comments crowd it out
- Execute the real discussion script and every inline block in the tests
- Say in the Issue that focus is untrusted background, not instructions

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

🤖 OpenAI PR review completed with an error

Reason: Fetched pull-request head a24205a does not match requested head 5a11f45.

Open the Actions run.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ OpenAI PR Review: PASS

Conclusion: Ready from the OpenAI review perspective. PR format, linked Issue design, and code/plan conformance passed with no actionable findings.

Review checks

Check Result
PR format ✅ PASS
Issue design ✅ PASS
Code & plan conformance ✅ PASS

Scope: a24205a3b2 · 5a11f45307..a24205a3b2 · incremental · 1 diff chunk

Usage: 46s · 628,235 tokens · 76.7% cache hit · 13.151 credits

Summary

The incremental changes address both prior findings: inline-code stripping now preserves multiline span boundaries, and discussion collection retains an out-of-window triggering comment. No actionable findings.

Review metadata
  • Commit: a24205a3b2
  • Range: 5a11f45307..a24205a3b2
  • Mode: incremental
  • Diff chunks: 1
  • Model: gpt-5.6-terra
  • Reasoning effort: medium
  • Session: repo-1309321116-pr-30-v2
  • Generation: b376e17c51f739f883b0c297e049d7bdda0540e04cd980bac83f198c488c9b6e
  • Evidence: 990ba0d18bf59ba634b649f4401054743f26e1ab35043921d0af29e845cde201

Totals

  • Input: 625,406
  • Cached input: 479,952
  • Cache write: 145,430
  • Output: 2,829
  • Reasoning: 1,690
  • Total: 628,235
  • Estimated credits: 13.151
  • Credit rate per 1M tokens: 62.5 input / 6.25 cached / 375 output
Token and cache details
Stage Mode Target Time Input Cached Hit Output Total Credits
pr deterministic format rules 0s 0 0 0.0% 0 0 0.000
pr incremental pr 13s 141,579 79,114 55.9% 808 142,387 4.702
issue incremental Issue #29 6s 148,649 143,885 96.8% 320 148,969 1.317
code incremental chunk 1/1 22s 161,724 85,894 53.1% 1,420 163,144 5.809
code incremental aggregate 5s 173,454 171,059 98.6% 281 173,735 1.324

@idy
idy merged commit b6904f1 into main Sep 3, 2026
17 of 19 checks passed
@idy
idy deleted the claude/issue-29-comment-trigger branch September 3, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Start and inform a review from the @codex review comment

1 participant