Skip to content

fix: the test identity is the pack's login, not its token's owner - #743

Merged
soydiloreto merged 9 commits into
mainfrom
fix/739-the-test-identity-is-the
Nov 14, 2026
Merged

soydiloreto merged 9 commits into
mainfrom
fix/739-the-test-identity-is-the

Conversation

@dilux-bot

@dilux-bot dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

📝 What changes

The login stops being data: kinds/engine-test/pack.yml keeps identity.secret only, and org.py gets one function, test_identity(env=os.environ), the only place any script learns the login. It returns DX_TEST_LOGIN from the environment, lowercased, or "" when it is unset ("" means no test identity: nobody is trusted or refused as it). org.py test-identity prints it. org.py test-identity --from-token asks GitHub who owns the token in GH_TOKEN (gh api user --jq .login) and prints that.

  • policy.py: in the test repository only (org.test_repo), the test identity is added to the trusted authors (C2). In every other repository it is never trusted, even when a list names it.
  • deciders.py: test_identity_refused keeps its rule and gets the login from org.test_identity() instead of the pack (C3).
  • sync-repos.py: the drift check takes the login from the same place. With no login, it reports that the test identity is unknown and does not skip the check without saying so.
  • install.py: it reads the login from the same value instead of the pack, so it can give the identity write on the test repository.

What a maintainer has to choose is how the login read in the lane, where DX_TEST_TOKEN exists (the release environment of .github only), reaches the workflows that decide. Those are the test repository's issues.yml and org-pull-request.yml, the other repositories' deciders, and org-drift.yml. None of them can read the secret. Either way, each of those workflows passes DX_TEST_LOGIN: ${{ vars.DX_TEST_LOGIN }} in its env:, as DX_BOT_LOGIN is passed today.

  • Option A (recommended): the lane keeps an organisation variable DX_TEST_LOGIN set. Before play, a step in the release environment runs org.py test-identity --from-token with DX_TEST_TOKEN. When the variable differs from the result, an App token minted in that job sets it (PATCH orgs/{org}/actions/variables/DX_TEST_LOGIN, or POST the first time). The lane's own scripts then use the login read from GitHub. C1 holds for any owner with nothing for a person to write. It widens a permission: the App's organisation-level "Variables: write", used only in that one job of release-lane.yml on main. I recommend it because the problem is a login written by hand that drifts from the token's real owner, and this removes the hand-written login altogether.
  • Option B: a person sets the organisation variable DX_TEST_LOGIN when installing, as with DX_BOT_LOGIN. install.py and docs/installing.md say how. The lane still reads the token's owner from GitHub and uses it for its own work. When the variable is unset or differs, the lane stops before play and names the value to set and the URL. Nothing is widened, but a token whose owner changes stops the lane until a person updates the variable. That is the same failure as today, only stated clearly.

💡 Why

Closes #739. The autopilot works it here in the open: one commit per criterion, each box ticked once its commit is pushed.

🎯 Acceptance criteria

  • C1 Given a DX_TEST_TOKEN whose owner is any login, when the lane works out the test identity, then its login is that owner's, read from GitHub, and no login is written in the pack.
    Test: scripts/org.py (--test: test_identity returns DX_TEST_LOGIN for an owner such as someone-else, "" when it is unset; --from-token prints what gh api user returns through an injected runner; the real pack has no login)
  • C2 Given a scenario pull request in the test repository authored by the token's owner, when the auto-merge decision runs, then it is a trusted author there and nowhere else.
    Test: scripts/policy.py (--test: a pull request by someone-else with DX_TEST_LOGIN=someone-else is trusted in the test repository and not trusted in another repository)
  • C3 Given an accepted added by the token's owner in the test repository, when the deciders are worked out, then it counts there and is refused in any other repository.
    Test: scripts/deciders.py (--test: accepted by someone-else counts in a repository of kind engine-test and is refused in any other repository; with no login, nobody is refused)
  • C4 Given the self-tests of org.py, policy.py, deciders.py and sync-repos.py, when they run, then they cover a token owner that is not {org}-test-bot.
    Test: scripts/org.py, scripts/policy.py, scripts/deciders.py, scripts/sync-repos.py (--test: each covers a token owner, someone-else, that is not {org}-test-bot; sync-repos's drift check names that login when it has write outside the test repository)

🧭 Decisions

  • 2026-11-14: policy.py compares a pull request's author with the trusted authors whatever its case (is_trusted): the test identity's login is lowercased, as the Spec says, while GitHub gives PR_AUTHOR in the login's own case, and a login on GitHub is the same whatever its case.
  • 2026-11-14: Option B (maintainer's answer): the check runs as the first step of the release lane's move job after the groups are listed, so the callers never move and play is skipped, and act still reads a failed move as a failing row; the message goes to the step's error annotation and the run summary.

🙌 Credits

🧪 How I tested it

Ran the self-test (--test) of every script in scripts/ at HEAD. All pass except three, and none of those three is about this change: design-lint.py needs stylelint and plugin-i18n.sh needs gettext, neither installed here, and review-brief.sh fails only because the run's PR variable is set (it passes with that variable unset). The repository has no make pre-pr.

The ones that cover this change: org.py (test_identity for an owner someone-else; --from-token through an injected gh api user; identity_problem: the lane plays when DX_TEST_LOGIN is the owner whatever its case, and stops naming the value and the URL when it is unset or differs); policy.py (a pull request by someone-else is trusted in the test repository only); deciders.py (accepted by someone-else counts in engine-test and is refused elsewhere, through the CLI); sync-repos.py (the drift check names someone-else, and says when DX_TEST_LOGIN is unset); release-lane.py (the move job checks DX_TEST_LOGIN against DX_TEST_TOKEN's owner before the callers move); install.py. Also ran tests/repro/test_issue_727.py with a small stand-in for pytest, since pytest isn't installed here. org.py test-identity --check with a token GitHub refuses exits 1 and says so. local-review.sh: conventions OK, floor high.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

The draft pull request opens before any code: the Spec as its checklist.
@dilux-bot dilux-bot Bot added autopilot Opened by the autopilot from an accepted issue autopilot:run-1 The autopilot's runs on this pull request labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

The pull request's board

State
Waits on nobody

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot mentioned this pull request Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot ended without a result. The agent finished without marking the pull request ready or saying why it stopped. Nothing was marked ready and the agent left no stop comment, so a person takes it from here (needs-human): a maintainer's answer here, a hint in words, brings it back, or they take it over. What it did is in the run: its summary, and the full transcript as an artifact.

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added the needs-human The autopilot handed this over to a person label Nov 14, 2026
@soydiloreto

Copy link
Copy Markdown
Member

The run finished its work, but nothing was pushed: while it worked, the branch was brought up to date with main after #732 merged, so its commits no longer built on the branch ("The agent's commits do not build on fix/739-the-test-identity-is-the as GitHub has it"). Please redo the same work on the branch as it is now.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: a person takes it from here (needs-human): read why and decide #743

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added autopilot:run-2 The autopilot's runs on this pull request and removed needs-human The autopilot handed this over to a person labels Nov 14, 2026
dilux-bot Bot added 5 commits November 14, 2026 03:41
kinds/engine-test/pack.yml keeps only the token's secret. scripts/org.py test_identity reads the login from DX_TEST_LOGIN, lowercased, "" when it is unset, and `org.py test-identity --from-token` asks GitHub who owns the token in GH_TOKEN. install.py takes the login from the same value, and the settings schema and docs/configuration.md say so.

Assisted-by: Claude Opus 5.5 (Anthropic)
trusted_authors_of takes the test identity from DX_TEST_LOGIN (org.py test_identity) and adds it in the test repository alone, the repository being GITHUB_REPOSITORY's name against org.yml `test-repo`; anywhere else it is never trusted, even when a list names it. The author is compared whatever its case, as GitHub's logins are. The self-test and the #727 reproduction cover a token owner, someone-else, that is not {org}-test-bot.

Assisted-by: Claude Opus 5.5 (Anthropic)
…(C3)

test_identity_refused keeps its rule, the test identity decides only in a repository of kind engine-test, and deciders.py takes the login from org.py test_identity instead of the pack. The self-test plays `accepted` by someone-else with DX_TEST_LOGIN=someone-else: it counts in the test repository, is refused and left out of the list in any other, and with no login nobody is refused.

Assisted-by: Claude Opus 5.5 (Anthropic)
The drift check takes the test identity's login from org.py test_identity, DX_TEST_LOGIN, instead of the pack. With no login and a test repository it says the test identity is unknown rather than skipping the check in silence. Its self-test covers a token owner, someone-else, that is not {org}-test-bot, named when it has write outside the test repository, as the self-tests of org.py, policy.py and deciders.py now do.

Assisted-by: Claude Opus 5.5 (Anthropic)
The workflows that run deciders.py or policy.py, and org-drift.yml's drift check, read the test identity's login from the organisation variable DX_TEST_LOGIN in their env, as they read DX_BOT_LOGIN, since none of them can read DX_TEST_TOKEN. docs/workflows.md lists the variable.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

The autopilot has a question before it goes on. C1 to C4 are done and pushed: the scripts read the login from DX_TEST_LOGIN (org.py test_identity), org.py test-identity --from-token reads the token's owner from GitHub, the pack keeps only secret, and the workflows that decide (and org-drift.yml) pass DX_TEST_LOGIN: ${{ vars.DX_TEST_LOGIN }}. What remains is the choice the Spec leaves to a maintainer: how the login read in release-lane.yml gets into that organisation variable. 1) Option A: before play, a step in the release environment runs org.py test-identity --from-token with DX_TEST_TOKEN and, when the variable differs, sets DX_TEST_LOGIN with an App token minted in that job. Nobody has to write anything, but it widens a permission: the App gets organisation-level Variables: write (scripts/install.py APP_PERMISSIONS has none today), used only in that job of release-lane.yml on main. 2) Option B: a person sets DX_TEST_LOGIN when installing, as with DX_BOT_LOGIN (install.py steps and docs/installing.md say how). The lane reads the token's owner and, when the variable is unset or differs, stops before play, naming the value to set and the URL. Nothing is widened, but a token whose owner changes stops the lane until a person updates the variable. I recommend 1 (A), as the Spec does: it removes the hand-written login, which is the cause of #739. Choose B if the App must not gain an organisation-level write permission.

Answer here in your own words; the autopilot goes on with your answer (autopilot:waiting).

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@soydiloreto

Copy link
Copy Markdown
Member
  1. Option B: the App gains no organisation-level write. The variable is set now (DX_TEST_LOGIN, organisation variable), and the lane stops before play, naming the value and the URL, when the token's owner differs.

🤖 AI-assisted · Claude Opus 5.5 (Anthropic)

@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@soydiloreto: answer the autopilot's question #743

🤖 AI-generated · Claude Opus 5.5 (Anthropic)

@dilux-bot dilux-bot Bot added the autopilot:free-3 The autopilot's runs on this pull request that do not count towards its cap label Nov 14, 2026
The maintainer chose option B: a person sets the organisation variable DX_TEST_LOGIN, and the App gains no organisation-level write. The release lane's move job now asks GitHub who owns DX_TEST_TOKEN (scripts/org.py test-identity --check) and, when DX_TEST_LOGIN is unset or names someone else, stops before the callers move and anything plays, naming the value to set and the URL. org.py's identity_problem says it, with its self-test; release-lane.py's workflow check holds the step before the move; installing, workflows and versioning docs say how to set it.

Assisted-by: Claude Opus 5.5 (Anthropic)
@dilux-bot
dilux-bot Bot marked this pull request as ready for review November 14, 2026 03:49
@dilux-bot
dilux-bot Bot requested a review from soydiloreto as a code owner November 14, 2026 03:49
@dilux-bot dilux-bot Bot added risk:high Set by the Claude review complexity:medium Set by the Claude review type:fix The kind of change, read from the diff by the Claude review labels Nov 14, 2026
@dilux-bot

dilux-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Claude review · risk high · complexity medium · type fix

Nothing that must be fixed · 2 minor

First review, so no new commits to credit. The pack's login is replaced by DX_TEST_LOGIN (org.test_identity()), trusted in the test repository only (policy.py, deciders.py), also read by the drift check and install.py, plus the release lane's move guard (Option B). I checked that every workflow running policy.py, deciders.py or sync-repos.py check now passes DX_TEST_LOGIN, and that move declares environment: release, so DX_TEST_TOKEN is readable and the guard runs before callers move; a refused token fails closed. The description is stale: "What changes" still presents Option A (App gains Variables: write) as recommended and the choice as open, while the diff implements only Option B, mentioned only under Decisions. It also does not say that is_trusted now compares every trusted author case-insensitively, which widens the old exact match. Minor: taken_over_by (scripts/policy.py:191) still compares exactly, while the test identity is now lowercased and GitHub gives HEAD_AUTHOR, HEAD_COMMITTER and NEEDS_HUMAN_BY in the login's own case, so a mixed-case token owner's takeover is not recognised. Reuse is_trusted and add a fixture. Minor: the trust check compares only the repository name with test_repo(conf), not the owner, looser than the old kind check. Credits block is empty, which is fine.

✍️ The description does not match the code (see the summary): no auto-merge until it does.

  • minor scripts/policy.py:191: taken_over_by still matches logins case-sensitively
  • minor scripts/policy.py:225: Test repository is matched by name only, not owner

Policy floor: high (touches high-risk paths: .github/workflows/autopilot.yml, .github/workflows/claude-review.yml, .github/workflows/conventions.yml, .github/workflows/issue-accepted.yml, .github/workflows/issue-repro.yml …). Reviewed 848bdb5 (whole pull request; review 1 of 5 automatic). Author trusted for auto-merge: false.

🤖 AI review · claude-sonnet-5-5 (Anthropic) · $0.42, 13 turns

@soydiloreto
soydiloreto merged commit 24dea17 into main Nov 14, 2026
33 of 34 checks passed
@soydiloreto
soydiloreto deleted the fix/739-the-test-identity-is-the branch November 14, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autopilot:free-3 The autopilot's runs on this pull request that do not count towards its cap autopilot:run-1 The autopilot's runs on this pull request autopilot:run-2 The autopilot's runs on this pull request autopilot Opened by the autopilot from an accepted issue complexity:medium Set by the Claude review risk:high Set by the Claude review type:fix The kind of change, read from the diff by the Claude review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: the test identity is the pack's login, not its token's owner

1 participant