Skip to content

feat(policies): read all pages in the chainloop.discover builtin - #3781

Merged
migmartri merged 4 commits into
mainfrom
pfm-8141-discover-builtin-read-all-pages
Nov 4, 2026
Merged

migmartri merged 4 commits into
mainfrom
pfm-8141-discover-builtin-read-all-pages

Conversation

@migmartri

@migmartri migmartri commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

The chainloop.discover policy builtin (Rego and WASM) returned only the first page of the paginated DiscoverPrivate API, which holds the 20 newest references by default. A policy that looked for an older reference of a digest, such as a CI run behind several newer deploy or scan runs, could not find it and got no error.

The builtin now reads all the pages:

  • It requests pages of 100 references (the maximum the API accepts) and follows next_cursor until there is no next page.
  • It merges the references of all pages into one result with the same shape as before, so existing policies need no change.
  • A cap of 10 pages (1000 references) bounds the requests of one call. When the read stops before the last page, the result has "truncated": true; otherwise "truncated": false.
  • A control plane without pagination support returns all references in one response with no next cursor, so the read ends after one request with nothing dropped. A control plane that ignores the cursor returns the same next cursor again; the read stops there, does not merge the repeated page, and sets truncated.
  • With no control plane connection, the Rego builtin returns null instead of panicking.

The builtin does not send the run's project_name. That would filter out references from other projects, which policies such as provenance checks can depend on, so it would change what the builtin returns.

This change is limited to the page-reading loop in pkg/policies/engine/builtins and does not overlap the organization scoping in #3777.

Note: pkg/policies/engine/rego/builtins/discover.go holds an older copy of the discover builtin that nothing in the repo registers. It still reads one page. This PR leaves it as is; it can be removed in a follow-up.

Testing: table-driven unit tests with a fake referrer client cover a single page, several pages, the last page allowed by the cap, the cap that stops the read (truncated), an error on a later page, a control plane without pagination, and a control plane that ignores the cursor. They also cover the JSON shape of the result and the output of the Rego builtin.

AI disclosure: this contribution was made with the help of Claude Code.

Closes #3780

🤖 Posted by Maximus bot (Claude Code) on behalf of @migmartri using the eng-autopilot skill

The chainloop.discover builtin (Rego and WASM) returned only the first
page of references (20 by default), so a policy could not see older
references of a digest. It now requests pages of 100 references, follows
the pagination cursor, and merges the references of all pages into the
same result shape. A cap of 10 pages bounds the requests of one call;
when the cap stops the read, the result has "truncated": true.

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ec58f4bc-6b1c-47bc-aa15-a90ad6cfff3a
@migmartri
migmartri requested a review from a team November 4, 2026 13:53
@chainloop-platform

chainloop-platform Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ✅ 3 passing

Status Policy Material Messages
✅ Passed pr-min-approvals pr-info -
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — 🟢 94% · ✅ 0 failing

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟢 94% 1 ✅ 0 98% AI / 2% Human 4 +306 / -20 53m4s

🟢 94% — 98% AI — ✅ All policies passing

Nov 4, 2026 13:48 UTC · 53m4s · $12.32 · 344 in / 157.8k out · claude-code 2.1.318 (claude-opus-5-5)

View session details ↗

Change Summary

Adds shared cursor pagination and ordered reference merging with a ten-page cap. Exposes truncation metadata through both policy adapters and handles older servers and repeated cursors. Adds pagination, serialization, error, and no-connection tests.

AI Session Overall Score

🟢 94% — Autonomous pagination change is aligned, narrowly scoped, and thoroughly tested; direct human trust feedback is unavailable.

AI Session Analysis Breakdown

🟢 94% · alignment

No notes.

🟢 94% · scope-discipline

No notes.

🟢 94% · solution-quality

🟢 Pagination lives in one shared service rather than duplicated adapter loops or a larger first-page workaround. · High Impact

🟢 93% · verification

🟢 Observed runs cover pagination boundaries, errors, old servers, adapter output, and a passing full policy suite. · High Impact

🟢 Mutation checks fail for removed truncation and adapter-output fixes, then restored code passes. · High Impact

🟡 New tests check shared serialization but do not directly invoke the changed host callback. · Low Severity

🟡 Autonomous implementation with automated review relays has no direct user confirmation by design. · Low Severity

🟢 92% · context-and-planning

No notes.

abstained · user-trust-signal

🟡 user-trust-signal not assessed — reviewer should confirm coverage · Low Severity


File Attribution

███████████████████░ 98% AI / 2% Human

Status Attribution File Lines
created ai pkg/policies/engine/builtins/discover_test.go +217 / -0
modified ai pkg/policies/engine/builtins/discover.go +73 / -11
modified ai pkg/policies/engine/builtins/rego.go +14 / -7
modified ai pkg/policies/engine/builtins/wasm.go +2 / -2

Policies (4)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-ec58f4 -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-ec58f4 -
✅ Passed ai-config-no-secrets ai-coding-session-ec58f4 -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-ec58f4 -

Security Checks — ✅ 5 passing

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed sast -
✅ Passed cwe-top25 -
✅ Passed cwe-top26-40-cusp -
Scans not applied (3)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed
iac-scan no IaC files changed

View attestation ↗

Security context

✅ Nothing this change touches has a recorded security-fix history.

View in Chainloop ↗ · How this works ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread pkg/policies/engine/builtins/rego.go Resolved
@migmartri
migmartri marked this pull request as ready for review November 4, 2026 14:04
The discover builtin now stops when a control plane ignores the cursor
and returns the same next cursor again, so it does not merge a page
twice. A control plane without pagination support already returns all
references in one response with no next cursor, and a test now covers
that case. The Rego builtin returns null instead of panicking when there
is no control plane connection, and has its own tests.

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ec58f4bc-6b1c-47bc-aa15-a90ad6cfff3a

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

When a control plane repeats its next cursor, the discover builtin stops
with only the pages it could read. It now sets truncated in that case, so
a policy does not take incomplete references for complete ones.

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ec58f4bc-6b1c-47bc-aa15-a90ad6cfff3a

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

The truncated field is set when the page cap is reached and when the
control plane repeats its next cursor. The field and builtin comments now
say so.

Assisted-by: Claude Code
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ec58f4bc-6b1c-47bc-aa15-a90ad6cfff3a
@migmartri
migmartri merged commit 8c52cb8 into main Nov 4, 2026
17 checks passed
@migmartri
migmartri deleted the pfm-8141-discover-builtin-read-all-pages branch November 4, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chainloop.discover policy builtin: read all pages of the paginated API

2 participants