Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2c776b8
EBP-3103: Add best-practice Python pub/sub reference samples
aelsammak Sep 22, 2026
f0e0655
EBP-3103: Remove PEP 563 future import from Python samples
aelsammak Sep 22, 2026
86f190c
EBP-3103: Condense the HA failover comment in the Python samples
aelsammak Sep 22, 2026
7f99a82
EBP-3103: Condense the Python connection-config helper docstring
aelsammak Sep 22, 2026
6d26a27
EBP-3103: Disable the trigger-evals CI job
aelsammak Sep 22, 2026
2bc274c
EBP-3103: Ignore Python bytecode caches
aelsammak Sep 22, 2026
8d9c865
EBP-3103: Fix shutdown and exit-code handling in the Python samples
aelsammak Sep 22, 2026
064f2a2
EBP-3103: Explain the terminate() grace period in the Python samples
aelsammak Sep 23, 2026
9fed1c9
EBP-3103: Fix review findings in the Python samples and CI docs
aelsammak Sep 23, 2026
d751d33
EBP-3103: Stop naming specific model versions in the evals README
aelsammak Sep 23, 2026
0e25aa0
EBP-3103: Restructure the Python samples as functions and bound buffers
aelsammak Sep 24, 2026
6922b57
EBP-3103: Build Solace resources before connect in the Python samples
aelsammak Sep 25, 2026
250de38
EBP-3103: Publish direct messages builderless and template guaranteed…
aelsammak Sep 25, 2026
7b5ea83
EBP-3103: Remove the publish rate throttle from the samples
aelsammak Sep 25, 2026
d30d3ef
EBP-3103: Let configuration override the reconnect defaults in Python
aelsammak Sep 25, 2026
7a542a1
EBP-3103: Set the message handler before starting the Python receivers
aelsammak Sep 28, 2026
87fcee9
EBP-3103: Add an async suffix to the Python sample names
aelsammak Sep 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ name: ci
# + post-merge version backstop on pushes to main
# link-check: CI-02: the tools/check-links.sh sweep
# compile-check: EVAL-02 compile the baked JCSMP samples verbatim
# trigger-evals: lightweight skill-triggering check; self-skips green without a key
# trigger-evals: lightweight skill-triggering check; disabled until CI may hold an API key
on:
# Deliberately no `edited` type: running on title/body edits either re-runs
# the paid eval matrix for a no-code change or, if jobs are gated, lets the
Expand Down Expand Up @@ -133,7 +133,11 @@ jobs:
# run summary. The secret is legal in the job env but illegal in a job-level
# if, so the guard reads HAS_KEY at step level and the real key is scoped to
# the run step only.
# Disabled: no policy permits an API key on this public repo, so the job can
# only self-skip. Run the trigger evals locally instead; remove this `if` to
# re-enable the job once a key is allowed.
trigger-evals:
if: false
runs-on: ubuntu-latest
timeout-minutes: 120 # per model leg; 57 cases x 3 runs (~50 min observed)
strategy:
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
# GSD planning directory (local-only)
.planning/

# Python bytecode cache
__pycache__/
12 changes: 6 additions & 6 deletions plugins/solace-messaging-skills/evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ To run the evals manually, you need the `claude` CLI and `jq` on your PATH, plus

```shell
export ANTHROPIC_API_KEY=<your key> # or CLAUDE_CODE_OAUTH_TOKEN
./tools/run-trigger-evals.sh # defaults to claude-haiku-4-5
./tools/run-trigger-evals.sh --model claude-sonnet-5
./tools/run-trigger-evals.sh # defaults to a Haiku model
./tools/run-trigger-evals.sh --model sonnet # the latest Sonnet
```

Each prompt runs three times and the verdict is the majority result. Set `TRIGGER_EVAL_RUNS=1` for a quicker, cheaper local check, and `TRIGGER_EVAL_MODEL` to change the default model. A run passes when at least 90% of cases match their expectation, and any infrastructure failure (such as a missing credential, a malformed corpus, or zero discovered cases) fails the run regardless of the rate. A case may also set `"must_pass": true`, and any must-pass failure fails the run regardless of the pooled rate. Note that a negative case asserts only that the named skill stays silent, so another skill in the plugin may legitimately fire on the same prompt.

## Continuous integration

In GitHub Actions, the `trigger-evals` job in `.github/workflows/ci.yml` runs this same script on every pull request as a two model matrix (`claude-haiku-4-5` and `claude-sonnet-5`), each model an independent check, and self-skips green when the `ANTHROPIC_API_KEY` secret is absent.
The `trigger-evals` job in `.github/workflows/ci.yml` is disabled (`if: false`), because no policy permits an API key on this public repository. Run the trigger evals locally as shown above. To re-enable the job once a key is allowed, remove the `if`. It then runs this same script on every pull request as a two model matrix (Haiku and Sonnet), each model an independent check, and self-skips green when the `ANTHROPIC_API_KEY` secret is absent.

# Output evals

Expand Down Expand Up @@ -60,12 +60,12 @@ Run from the repository root:

```shell
export ANTHROPIC_API_KEY=<your key> # or CLAUDE_CODE_OAUTH_TOKEN
./tools/run-output-evals.sh # defaults to claude-sonnet-5
./tools/run-output-evals.sh --model claude-opus-5
./tools/run-output-evals.sh # defaults to a Sonnet model
./tools/run-output-evals.sh --model opus # the latest Opus
./tools/run-output-evals.sh --case appdev-quickstart-implement-full
```

Run the suite for both `claude-sonnet-5` and `claude-opus-5` before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. A permission denial of a tool the runner grants (Bash, Write, and the rest of its allowlist) is also INFRA: managed settings, hooks, or local command shims can deny a command, the model then improvises, and the result no longer measures the skill. The case line names the denied tool and command; adjust the policy for the run instead of reading the result as a skill failure. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default `claude-sonnet-5`) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails.
Run the suite for both the latest Sonnet and the latest Opus (`--model sonnet` and `--model opus`) before a PR that affects skill content, and record both results in the PR description, including whether live verify ran or skipped (the runner prints this in its summary). A full leg is a long run; keep the machine awake for it (on macOS, prefix the command with `caffeinate -i`), because a sleep mid-run surfaces as INFRA failures. A permission denial of a tool the runner grants (Bash, Write, and the rest of its allowlist) is also INFRA: managed settings, hooks, or local command shims can deny a command, the model then improvises, and the result no longer measures the skill. The case line names the denied tool and command; adjust the policy for the run instead of reading the result as a skill failure. Each case runs once by default (`OUTPUT_EVAL_RUNS=1`), because a full implement-flow case is expensive; raise it for a majority-vote stability study. `OUTPUT_EVAL_JUDGE_MODEL` (default a pinned Sonnet model) stays fixed across subject models so leg differences are attributable to the subject. `OUTPUT_EVAL_WORKDIR` receives transcripts and generated projects, and the work directory is kept and printed when the run fails.

The gate matches the trigger evals: a run passes when at least 90% of cases pass, any infrastructure failure fails the run, and any `must_pass` failure fails the run regardless of the pooled rate. The forbidden-behavior negatives and the compile case are `must_pass`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ Hold every invariant while adapting: keep the shared connection helper, the sess

**Preserve the marker contract.** A variant adapts the SHAPE, never the marker strings. It MUST still emit its leaf's `VERIFY:` markers, character for character, so the fixed `verify.sh` stages drive it unchanged (this is exactly why a durable-reply-queue requestor still prints `VERIFY: REPLY_RECEIVED`). Never rename or drop a marker to fit a variant.

**Demo harness is not application logic.** The samples carry demo-harness elements that exist only to make a standalone run observable: the ENTER-to-quit `System.in` loop, the once-per-second stats-printing thread, the pacing `Thread.sleep` between sends, and the rotating example payload. When the developer's request describes a real application domain, do NOT carry these into the generated classes: replace the example payload and send cadence with the application's real messages and triggers, and keep only the lifecycle pieces the leaf's steps call for (the SIGINT hook on the long-running role, the self-exit on the foreground role). When the request IS a demo or a try-it run, keeping the harness is fine. Two things are NOT demo harness: the `trace(...)` narration method (keep it; applications replace its single body to route narration to their logging or reporting system) and the `VERIFY:` markers, which are emitted through `trace(...)` like every other status line. Because `verify.sh` greps captured stdout for the markers and for the shutdown hook's `Shutdown signal received` proof line (both flow through `trace(...)`), the generated `trace(...)` body MUST stay `System.out.println` through the Step 5 verification stages; rerouting that body to a logger or reporting system is the developer's step AFTER the verify stages pass, never part of generation. Never treat the logging backend (Step 3) as demo harness: the backend and its config file ship in every generated app.
**Demo harness is not application logic.** The samples carry demo-harness elements that exist only to make a standalone run observable: the ENTER-to-quit `System.in` loop, the once-per-second stats-printing thread, the continuous publish loop, and the rotating example payload. When the developer's request describes a real application domain, do NOT carry these into the generated classes: replace the example payload and the publish loop with the application's real messages and triggers, and never add a delay or rate limit between sends that the application does not ask for, and keep only the lifecycle pieces the leaf's steps call for (the SIGINT hook on the long-running role, the self-exit on the foreground role). When the request IS a demo or a try-it run, keeping the harness is fine. Two things are NOT demo harness: the `trace(...)` narration method (keep it; applications replace its single body to route narration to their logging or reporting system) and the `VERIFY:` markers, which are emitted through `trace(...)` like every other status line. Because `verify.sh` greps captured stdout for the markers and for the shutdown hook's `Shutdown signal received` proof line (both flow through `trace(...)`), the generated `trace(...)` body MUST stay `System.out.println` through the Step 5 verification stages; rerouting that body to a logger or reporting system is the developer's step AFTER the verify stages pass, never part of generation. Never treat the logging backend (Step 3) as demo harness: the backend and its config file ship in every generated app.

**Comments follow their code.** The samples' comments are part of the reference, not decoration: they carry the documented best practices (the session-independent `createMessage`, the binary payload and the legacy `XML` naming, `getData()` versus `getBytes()`, registering a session event handler at session creation, the ACK/NACK handling options, the correlation-key rule) into the code a developer reads long after this session. Generation carries them under ONE contract, on every door and every app shape:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ public class DirectPublisher {
private static final String APP_NAME = DirectPublisher.class.getSimpleName();
static final String TOPIC_PREFIX = "solace/samples/"; // used as the topic "root"
private static final String API = "JCSMP";
private static final int APPROX_MSG_RATE_PER_SEC = 100;
private static final int PAYLOAD_SIZE = 512;

// remember to add log4j2.xml to your classpath
Expand Down Expand Up @@ -189,15 +188,10 @@ private static void runPublishLoop() throws IOException {
if (e instanceof JCSMPTransportException) { // all reconnect attempts failed
isShutdown = true; // let's quit; or, could initiate a new connection attempt
}
} finally { // add a delay between messages
try {
Thread.sleep(1000 / APPROX_MSG_RATE_PER_SEC); // do Thread.sleep(0) for max speed
// Note: STANDARD Edition Solace PubSub+ broker is limited to 10k msg/s max ingress
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // restore the interrupt status, do not swallow it
isShutdown = true;
}
}
// no delay between messages: the loop publishes as fast as send() returns, the way an
// application publishes each message as soon as it has one
// Note: STANDARD Edition Solace broker is limited to 10k msg/s max ingress
}
isShutdown = true;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,7 @@ public class DirectRequestor {
private static final String API = "JCSMP";
// POSITIVE blocking-request timeout in milliseconds: requestor.request(...) blocks up
// to this long for the correlated reply before throwing JCSMPRequestTimeoutException.
// A positive timeout is the direct reliability knob (NOT a burst); 3000 ms matches the
// upstream DirectRequestorBlocking sample.
// A positive timeout is the direct reliability knob (NOT a burst).
private static final int REQUEST_TIMEOUT_MS = 3000;

private static JCSMPSession session;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ public class GuaranteedPublisher {
static final String TOPIC_PREFIX = "solace/samples/"; // used as the topic "root"
private static final String API = "JCSMP";
private static final int PUBLISH_WINDOW_SIZE = 255; // max guaranteed-publish window (range 1-255)
private static final int APPROX_MSG_RATE_PER_SEC = 100;
private static final int PAYLOAD_SIZE = 512;

// remember to add log4j2.xml to your classpath
Expand Down Expand Up @@ -220,15 +219,10 @@ private static void runPublishLoop() throws IOException, InterruptedException {
if (e instanceof JCSMPTransportException) { // all reconnect attempts failed
isShutdown = true; // let's quit; or, could initiate a new connection attempt
}
} finally { // add a delay between messages
try {
Thread.sleep(1000 / APPROX_MSG_RATE_PER_SEC); // do Thread.sleep(0) for max speed
// Note: STANDARD Edition Solace PubSub+ broker is limited to 10k msg/s max ingress
} catch (InterruptedException e) {
Thread.currentThread().interrupt(); // restore the interrupt status, do not swallow it
isShutdown = true;
}
}
// no delay between messages: the loop publishes as fast as send() returns, the way an
// application publishes each message as soon as it has one
// Note: STANDARD Edition Solace broker is limited to 10k msg/s max ingress
}
isShutdown = true;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,7 @@ public void handleEvent(SessionEventArgs event) {
final Queue requestQueue = JCSMPFactory.onlyInstance().createQueue(REQUEST_QUEUE_NAME);

// ELEVATION: provision the durable request queue in-process and map the request
// topic onto it at startup, so a fresh broker works out of the box (the upstream
// sample provisions with flag 0, which throws "already exists" on a re-run).
// topic onto it at startup, so a fresh broker works out of the box.
// best practice: confirm the broker allows client-side endpoint management first
if (!session.isCapable(CapabilityType.ENDPOINT_MANAGEMENT)) {
System.err.println("This client/broker does not allow client-side endpoint management; "
Expand All @@ -168,8 +167,8 @@ public void handleEvent(SessionEventArgs event) {
EndpointProperties endpointProps = new EndpointProperties();
endpointProps.setAccessType(EndpointProperties.ACCESSTYPE_EXCLUSIVE); // single-replier sample
endpointProps.setPermission(EndpointProperties.PERMISSION_CONSUME);
// provision the durable request queue IDEMPOTENTLY so a re-run is safe (NOT the
// upstream flag 0, which throws on the second run)
// provision the durable request queue IDEMPOTENTLY so a re-run is safe (flag 0 would
// throw "already exists" on the second run)
session.provision(requestQueue, endpointProps, JCSMPSession.FLAG_IGNORE_ALREADY_EXISTS);
// map the request topic onto the queue with a subscription (PERSISTENT pub/sub onto
// a queue); WAIT_FOR_CONFIRM blocks until the broker confirms the subscription
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,7 @@ public void handleEvent(SessionEventArgs event) {
final Queue queue = JCSMPFactory.onlyInstance().createQueue(QUEUE_NAME);

// ELEVATION: provision the durable queue in-process and map the topic onto it
// at startup, so a fresh broker works out of the box (the upstream sample expects
// the queue to pre-exist and errors out otherwise).
// at startup, so a fresh broker works out of the box.
// best practice: confirm the broker allows client-side endpoint management first
if (!session.isCapable(CapabilityType.ENDPOINT_MANAGEMENT)) {
System.err.println("This client/broker does not allow client-side endpoint management; "
Expand Down
Loading
Loading