[imit]

What this adds

tests/live: scute-python against a real Scute API (the v2 deployment), with no mocks. It's off in the normal run and in CI (addopts = -m "not live"), so pytest and the CI job are unchanged.

One file per area:

1. App: its public data, and the SDK finding the public id from the app's UUID.
2. Sign-in: email OTP and SMS OTP with test identities, then the user, refresh, sign out, and listing and revoking sessions.
3. Tokens: local JWKS verification (tampered, expired, other audience and malformed tokens refused) and remote=True.
4. MFA: TOTP enrollment (RFC 6238 codes), a sign-in that needs MFA finished with TOTP, backup codes, removing the method.
5. Users: create, get by id and by identifier, list, update, deactivate, activate, delete, and meta.
6. Signing in as a user: start (the act claim, authz_context), list, stop, and "not while impersonating" refused inside the session.
7. Authorization:
   • policy import, role assign and remove
   • check, check-batch, permissions, authorized users, filter
   • the signed snapshot with decide_locally matching the server over a 10-case matrix
   • access requests
   • the FastAPI and Flask glue on real sessions
8. Agents: a task, checks through it (allow, outside_task), a step-up through the human steps, properties (read a secret; an ES256 signature checked against the JWKS), a budget of 2 pausing the agent on the 3rd action, suspend and resume.
9. Auth MCP: initialize, tools/list, scute_identify, scute_submit_code, scute_check, then the backend's conversation lookup and check.
10. Decision log: rows read back for the checks above.

The SDK has no API for some of this: the sign-in itself, MFA, policy and roles, filter, the snapshot, access requests, agents, properties, the auth MCP and the decision log. Those tests call the HTTP API with httpx and say so. There's no Python agent harness yet.

A few more details:

• Isolation: everything a run makes is named live-<runid> and deleted at the end, also when tests fail.
• Sign-ins: five per run.
• Secrets: tokens, keys and codes are wrapped so a failing assert prints them masked.
• Helpers: tests/live/test_support.py checks the suite's own helpers offline (RFC 6238 vectors, masking, the env file, cleanup order). It runs in CI.

How to run

heroku run -a scute-api-v2 rake "sdk_live:setup[python]"   # once; prints three lines
# put them in .sdk-live/python.env next to the checkout (outside the repo), or export them
pip install -e ".[dev]"
pytest -m live

Without credentials every live test is skipped with one line and the run exits 0.

Result against scute-api-v2 (v21)

48 passed, 4 xfailed. ruff and mypy --strict tests/live are clean. After the run the app has no users, roles, resources, permissions, agents or properties, and its authz and MFA settings are back as they were.

Findings

These are confirmed against the live API and marked xfail(strict=True) with the finding as the reason. A fix shows up as an unexpected pass.

1. Refreshed tokens fail local verification.
   • A sign-in access token carries aid = the app's public id (api token_session.rb:57). A token from POST /v1/auth/:app_id/tokens/refresh carries aid = the app's internal UUID (TokenSession#jwt_session, token_session.rb:300).
   • Scute itself accepts the refreshed token (GET current_user: 200).
   • scute.tokens.verify refuses it with InvalidToken, reason wrong_app, so every refreshed session fails local verification.
   • Test: test_a_refreshed_token_verifies_locally.
2. scute.sessions.list and scute.sessions.revoke don't work with the secret key.
   • GET /v1/:app_id/users/:id/sessions answers 401 {"error": "Not authorized"}, with no error_code. DELETE /v1/:app_id/users/:id/sessions/:session_id answers the same.
   • The same key gets 200 on GET /v1/:app_id/users/:id.
   • The controller also runs authorize_access_request!, which wants a user session in X-Authorization (api sessions_controller.rb:6-8).
   • Tests: test_lists_sessions, test_revokes_a_session.
3. scute.users.find_by_identifier creates users.
   • The docstring says it returns None when nobody uses the identifier. In fact GET /v1/auth/:app_id/users?identifier=<unused> answers 200 and creates an active user (the count for that email went from 0 to 1).
   • The endpoint does find-or-create (api api_base_controller.rb:282).
   • Test: test_find_by_identifier_is_none_for_nobody.

Checked, not bugs (behaviour pinned in tests or noted)

• Meta on create: users.create(meta=...) with a key the app hasn't declared as a user meta field answers 200, with user_meta_errors: [{key, message: "Invalid value for <key>"}], and the meta isn't saved. users.update(user_meta=...) answers 422 {"errors": [...]}. The SDK doesn't raise on the create. So the README's meta= examples need the field declared in the dashboard first (the app secret can't declare it). Pinned by test_meta_needs_declared_fields.
• Unknown app id: tokens.verify with an app id whose JWKS can't be fetched raises APIError (404, app_not_found), not InvalidToken. A misconfiguration surfaces as an API error, which seems right. No test.

Not covered

• users.invite: it sends a real magic-link email, and test identities only cover OTP codes.
• The MFA "re-verify before removing a method" refusal: it only triggers for sessions older than mfa_reverify_minutes (minimum 5). The suite passes the verification anyway.