UN-3770 [MISC] Follow DRF pagination on all clone list endpoints

[chandrasekharan-zipstack]

What & why

Part of the UN-3770 list-pagination workstream. This is step 1 of 4 and is the prerequisite for flipping the OSS list endpoints to unconditional pagination — it must be merged and released before that lands.

Every list_* helper in PlatformClient already unwrapped a paginated envelope:

return result if isinstance(result, list) else result.get("results", [])

…but none of them ever sent ?page or followed next. They read page one and stopped.

That is already a live bug, not a future one. tags/, pipeline/ and api/deployment/ use CustomPagination on the backend today, so cloning an org with more than 50 tags / pipelines / API deployments silently drops the rest — no error, no warning. The stale comment on list_adapters ("no pagination on this endpoint") is the assumption that made this easy to miss.

Once unstract#2187's follow-up makes adapters, connectors, workflows and prompt-studio unconditionally paginated, the same truncation hits those too. Silent partial data loss is a far worse failure mode than a hard break, so the client has to be fixed first.

Approach

PlatformClient._paginate(path, params):

• Bare list → returned unchanged. No server-version negotiation, no feature flag. The same client works against an old on-prem deployment that doesn't paginate and a new one that does, so this ships and releases entirely independently of the backend work.
• Envelope → walks next to exhaustion.
• Refuses to return a short read. Collected rows are checked against the reported count; a mismatch raises PlatformAPIError. This is the durable protection — any future truncation becomes loud instead of silent.
• Cyclic next raises rather than hanging a migration forever.

_request() is split into a URL-building wrapper over _send(), because DRF's next links are absolute and can't go through org-relative path composition. _request()'s signature is unchanged, so every existing caller is untouched.

All 23 list helpers now route through _paginate. list_lookup_versions deliberately keeps its bespoke unwrap — that endpoint returns {"versions": [...], "next_version_number"}, not a DRF envelope.

Testing

• tests/clone/test_client.py: 3 new tests — follows next across pages and hits the absolute URL verbatim; raises on a short read; raises on a cyclic next. The two pre-existing tests pinning bare-list and single-page-envelope behaviour still pass unchanged, which is the backward-compatibility guarantee.
• Full suite: 250 passed.
• ruff check + ruff format clean.
• ⚠️ Live validation still pending — hence draft. tags/ already paginates in staging, so list_tags exercises the real envelope → next → count path end-to-end with no backend change required.

Residual risk

A customer pinned to an older client version still truncates silently after the backend flip; nothing here helps them retroactively. Tracked in the rollout plan, along with the option of instrumenting bare-array responses for one release to measure who is still affected before anything breaks.

🤖 Generated with Claude Code

https://claude.ai/code/session_018HXYFVCZGi7YN9qjDGyJCQ

[greptile-apps]

Greptile Summary

Adds comprehensive DRF pagination support to clone list operations.

• Introduces shared page traversal, response-shape validation, count verification, cycle detection, and same-origin URL pinning.
• Routes clone list helpers through the new paginator while retaining compatibility with bare-list responses.
• Adds pagination-focused tests and a dedicated, commit-pinned clone-test workflow.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/unstract/clone/client.py	Adds shared pagination and hardens subsequent-page payload and URL handling; the previously reported defects are addressed.
tests/clone/test_client.py	Adds coverage for multi-page traversal, malformed payloads and links, count mismatches, cycles, and origin pinning.
.github/workflows/clone-tests.yml	Adds focused Python 3.11/3.12 clone tests using immutable action commit references.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[List helper] --> B[_paginate]
    B --> C[Request first page]
    C --> D{Payload shape}
    D -->|Bare list| E[Return rows]
    D -->|DRF envelope| F[Validate results]
    F --> G[Append rows]
    G --> H{next present?}
    H -->|No| I[Verify reported count]
    H -->|Yes| J[Validate and pin URL to configured origin]
    J --> K[Request next page]
    K --> F
    I --> E

Loading

_{Reviews (9): Last reviewed commit: "Wrap urlunparse args to satisfy ruff E50..." | Re-trigger Greptile}

[chandrasekharan-zipstack]

Addressed both Greptile P1s in c38d1ef:

• Equivalent origins rejected — _assert_same_origin now compares a normalised (scheme, host, port): host lower-cased, explicit default ports (80/443) folded to the scheme default. A next link that differs only by case or an explicit :443 is followed, not rejected.
• Malformed results bypasses validation — new _results_or_raise validates results is a list on every page (first and subsequent), so a non-list value raises PlatformAPIError instead of corrupting rows via extend.

Tests: tests/clone/test_client.py 21 passed (added test_paginate_follows_equivalent_origin_next and test_paginate_raises_on_nonlist_results), ruff clean.

The third finding (SHA-pin CI actions) is left as a repo-wide policy call — every workflow here uses mutable major-version tags, so pinning one file would be inconsistent. Happy to pin all workflows in a separate change if we want that policy.

[chandrasekharan-zipstack]

Addressed the P1 Malformed ports escape error handling (cc0786d): _assert_same_origin now catches the ValueError urlparse raises on .port access and re-raises PlatformAPIError (test test_paginate_raises_on_malformed_port_in_next). 22/22 clone tests green.

The other four code findings were already resolved on c38d1ef:

• Validate every pagination response — _results_or_raise runs at the top of every loop iteration, not just the first page.
• Constrain pagination link origins — _assert_same_origin gates each next before it receives the bearer key.
• Equivalent origins are rejected — _origin lowercases host + fills default ports, so case/port-equivalent links follow.
• Malformed results bypass validation — the guard is isinstance(result.get("results"), list), not a bare in check.

Leaving the Mutable CI action references (SHA-pin) finding: test.yml and main.yml also reference actions by major-version tag, so pinning clone-tests.yml alone would be inconsistent. SHA-pinning is worth doing repo-wide as a separate hardening pass, not a one-workflow patch here.

[chandrasekharan-zipstack]

Pinned the three clone-tests.yml actions to full commit SHAs (c9f6357), reversing my earlier "leave it" call so this PR clears the last open finding:

• actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
• actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
• astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6.8.0

Trailing comment tracks the version for readability/dependabot. test.yml and main.yml still tag-ref their actions — worth a follow-up repo-wide pin, but out of scope for this clone-pagination PR.

[pk-zipstack]

Premise and sequencing look right — the silent truncation on tags/, pipeline/ and api/deployment/ is a live bug, and fixing the client ahead of the backend flip is the correct order. The sweep is complete (29 helpers converted; list_users and list_lookup_versions correctly excluded), and the test coverage on _paginate is thorough.

Two things I'd want resolved before this merges — both are new hard-failure modes introduced by the change, on paths that previously succeeded (albeit truncated). Details inline.