EBP-3107: Add Python verify.sh, checklist template, and smoke fixture

[aelsammak]

What is the purpose of this change?

Python Implement mode had no verify script, so every generated Python project ended in a compile-only fallback. This adds a Python verify.sh (the counterpart of the JCSMP script), a verification checklist template, and a CI check for the baked Python samples. It unblocks live_verify for the Python output evals (EBP-3108).

How is this accomplished?

1. Added references/python/scripts/verify.sh with the stages publisher, consumer, roundtrip, direct, direct-request-reply, and app, and the same VERIFY: markers and 0/1/2 exit contract as JCSMP. Each role runs in its project's own .venv. Separate projects use --subscriber-dir/--publisher-dir and --replier-dir/--requestor-dir.
2. Exit 2 (environment failure) maps the Python API error classes and SOLCLIENT_SUBCODE_* names for login, VPN, host, connection, trust store, certificate, and queue bind failures.
3. Moved the checklist into references/python/verification-checklist.md and rewrote implement-mode Steps 5 and 6 around the bundled script.
4. Added the python-sample-check CI job and its fixture. It compiles, imports, and builds every baked sample against the latest solace-pubsubplus without a broker, and parses verify.sh.
5. Renamed the JCSMP compile-check job to jcsmp-sample-check and moved its fixture, so every API uses <api>-sample-check and references/<api>/evals/sample-check/.
6. Closed false passes in both the Python and the JCSMP verify.sh: roundtrip ignores messages already on the queue, a failed or hung foreground role fails its stage, and the app stage ignores markers logged before the trigger.

Anything reviews should focus on/be aware of?

• Item 6 changes the shipped JCSMP verify.sh, so it reaches users only with the next release.
• Item 5 changes scope: the ticket said to leave compile-check alone. No branch protection requires the check by name. The agent-plugins-workflow skill in solly-marketplace still names compile-check.
• verify.sh refuses connection arguments when config.json exists, because the scripts would connect to the config.json broker.
• Verification: every stage passed live against a tcps:// broker on projects generated from this branch, and bad credentials, VPN, and trust store each exited 2. Item 6 was checked with stub roles, not a broker. All runs used macOS bash 3.2.
• Local evals ran on f4592a0, before item 6:
  • Trigger: Sonnet 5 62/62, Opus 5.5 62/62. Haiku not run.
  • Output: Sonnet 5 10/11, Opus 5.5 11/11. Live verify skipped (no broker). The Sonnet 5 failure, topics-no-files-negative, repeats and is not related to this PR: the model writes a file against the topics skill's chat-only rule.
  • The output corpus has no Python cases yet (EBP-3108).

[mtite-sol]

Tested verify.sh at ce6229e on Linux bash 5.3. I used stub roles for the stage logic, and projects built from the baked samples with solace-pubsubplus 1.11.0 for connection failures. No broker.

Findings

1. With Solace Suggested's HA values, an unreachable broker is classified as a code failure (exit 1, should be 2)
I applied the Solace Suggested HA values to the baked guaranteed subscriber (with_connection_retry_strategy(parametrized_retry(1, 3000)), with_reconnection_retry_strategy(parametrized_retry(20, 3000)), retries-per-host 5) and ran ./verify.sh consumer:

Target	Result
Refused port (tcp://127.0.0.1:55555)	exit 1 after 35 s
Unroutable host (tcp://10.255.255.1:55555)	exit 1 after 48 s; the subscriber also ignored SIGINT and was force-killed
Control: refused port, unmodified sample (no connection retries)	exit 2 after 1 s

On the refused port the API was still retrying at the TIMEOUT_S=30 deadline. SOLCLIENT_SUBCODE_COMMUNICATION_ERROR was logged only after the script's SIGINT, by which point the run was already classified. The unroutable host logged nothing at all. Either way the agent enters the fix loop over a wrong host or port. Two possible fixes: check the logs for environment signatures again after the long-running role stops, or treat "no VERIFY: CONNECTED before the deadline" as an environment failure. The JCSMP script has the same TIMEOUT_S=30 and likely the same problem with its HA values (not tested).

2. roundtrip can still pass on a message left on the queue (both APIs)
The leftover-message count is taken once, as soon as QUEUE_BOUND is seen (Python L404, JCSMP L462). watch_marker polls once a second, so a leftover message delivered just after that count is taken counts as the new receipt. Stub: the subscriber prints a leftover MESSAGE_RECEIVED 1.5 s after binding, and the publisher prints PUBLISH_ACKED but its message is never delivered. Result: PASS (roundtrip), exit 0. This is likely after a few failed runs leave messages on the queue, or with a slow handler. Possible fix: wait until the receipt count has stopped changing for a couple of seconds before starting the publisher.

3. The Python request-reply "no reply" message is unreachable
RR_NO_REPLY is set only when the requestor exits 0 (L490), but the Python blocking requestor exits 1 when no reply arrives (sample L232). Stub: the replier subscribes and the requestor exits 1 with no reply. Result: the generic CODE FAILURE text instead of the "request never arrived / reply lost" guidance. The verdict (exit 1) is right; only the diagnosis is lost. JCSMP isn't affected: its requestor exits 0 without a reply.

Passed

• Bash 5.3: the direct stage and a normal roundtrip pass on stubs.
• A project with only direct_requestor.py (the non-blocking variant) is picked up automatically, and direct-request-reply passes.
• Fan-out: the checklist and implement mode both say verify.sh drives guaranteed_subscriber.py only, and the other subscribers use their handed-back commands.
• The control run above: a fail-fast refused port exits 2 within a second.
• CI: jcsmp-sample-check and python-sample-check pass. No ruleset requires a check by name, so the rename is safe, and agent-plugins-workflow in solly-marketplace already uses the new names.

[mtite-sol]

Changes LGTM!