[jcshepherd]

(#403)

What

Plumb config.max_connections through to the driver config, so the driver can use more than the default single connection per node (typically Cassandra drivers are configured to use multiple connections per node). This has a side effect of improving driver stability due to an issue in the cdr-tokio driver where "stream ids" (like request ids") rollover at 2^16 and get re-used without checking to see if the id is already in use (e.g., by a slow-running query).

Configure containerized Cassandra for local testing and (hopefully) Github CI to use localhost as its broadcast address instead of the node's internal container-assigned address which is typically unreachable outside the container. Cassandra was defaulting to using the latter which prevented new connections from being established.

Test: connection_pool_size.rs asserts the pool size follows max_connections (red when it was ignored, green now). Recovery-after-drop verified manually via nodetool disablebinary/enablebinary.

Why

Two connection-resilience problems on the Cassandra backend:

1. create_session never applied config.max_connections, so the driver ran one connection per node regardless of config.

2. Containerized nodes advertise their internal bridge IP as broadcast_rpc_address. The driver reconnects to the advertised address, which was not reachable outside the container.

Mostly addresses #403: the underlying cdrs-tokio stream-id reuse is a driver defect, mitigated here (more connections divide the id space) and tracked upstream.

Closes #403

Testing done

Full pre-merge workflow (fmt, clippy, license, unit, pytest, rust-tests) + one new Cassandra direct-integration test, on Cassandra 4.11

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• [N/A] Breaking changes are noted below (if any)
• [N/A] If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

I ran the new test against a single Cassandra 4.1 node. The pool size follows max_connections: 1, 2 and 8 give 2, 3 and 9 clients on the node, and the test is red with engine.rs reverted to main. fmt and clippy are clean. The broadcast address half does not reproduce: with max_connections=8 every socket the process holds goes to the contact point, none to the advertised address, and after nodetool disablebinary then enablebinary the pool answers "No nodes available in query plan!" for 105 seconds with 127.0.0.1 advertised and without it. Please drop "Closes #403", correct the two documentation statements, and count only this engine's own connections in the test. The commit title also carries a stray bullet.

[LeeroyHannigan]

Major: I measured the opposite on a single node. With max_connections=8 against a node advertising its container address, all nine sockets the process holds go to the contact point (ss -tnp), which is what cdrs-tokio does for the control node's own row in system.local, using the address it connected to rather than the advertised one (cdrs-tokio 9.0.2 cluster/cluster_metadata_manager.rs:147-160, citing CASSANDRA-11181); after nodetool disablebinary then enablebinary the pool answered "No nodes available in query plan!" for 105 seconds both with 127.0.0.1 advertised and with the container address advertised. Please scope this paragraph to peer rows in a multi-node cluster, or re-measure the recovery claim and say what the measurement was.

[jcshepherd]

On a single node the advertised address has no effect and is for internode connections anyway. cdrs uses the connect address for the control node's own row (CASSANDRA-11181). Recovery on a single node took 2.1s (127.0.0.1 broadcast) vs 2.6s (240.x broadcast): not a meaningful difference. Removed the broadcast-address section from the deployment guide.

[LeeroyHannigan]

Major: there is no devtools/cassandra-node in the repository (git ls-tree origin/main devtools/) and this PR does not add one, so this names a script and a --broadcast-rpc-address flag a reader cannot use. devtools/run-cassandra-tests is what starts a local node and it is unchanged here, so a local run still advertises the container address. Please add the script, or describe what the repository has today.

[jcshepherd]

Edited out. We probably should have some discussion of whether we want to have test utilities in the project and where.

[LeeroyHannigan]

Major: both counts come from system_views.clients, which holds every client on the node, so an unrelated connection closing between them subtracts from the delta and fails the assertion with a message that blames the fix. I held a separate 8-connection client, started this test, and closed that client after the first count: "large pool (max_connections=8) added only 7 connections (small_count=5, both_count=12); expected at least 8". The CI job runs this suite without --test-threads=1 (integration-cassandra.yml:96) while the other 149 tests build and drop their own engines, so please take a baseline immediately around each engine's own construction, or isolate this test the way ttl_integration is isolated at line 89.

[jcshepherd]

This test was doing more than it needed to, in terms of (essentially) testing driver and node behavior, not just the basic configuration change.

[jcshepherd]

I ran the new test against a single Cassandra 4.1 node. The pool size follows max_connections: 1, 2 and 8 give 2, 3 and 9 clients on the node, and the test is red with engine.rs reverted to main. fmt and clippy are clean. The broadcast address half does not reproduce: with max_connections=8 every socket the process holds goes to the contact point, none to the advertised address, and after nodetool disablebinary then enablebinary the pool answers "No nodes available in query plan!" for 105 seconds with 127.0.0.1 advertised and without it. Please drop "Closes #403", correct the two documentation statements, and count only this engine's own connections in the test. The commit title also carries a stray bullet.

I don't think your test reproduced the problem, if I'm understanding your test correctly. The problem is not with initial connection, it is with reconnection. That's where the broadcast address comes into play.

I ran the new test against a single Cassandra 4.1 node. The pool size follows max_connections: 1, 2 and 8 give 2, 3 and 9 clients on the node, and the test is red with engine.rs reverted to main. fmt and clippy are clean. The broadcast address half does not reproduce: with max_connections=8 every socket the process holds goes to the contact point, none to the advertised address, and after nodetool disablebinary then enablebinary the pool answers "No nodes available in query plan!" for 105 seconds with 127.0.0.1 advertised and without it. Please drop "Closes #403", correct the two documentation statements, and count only this engine's own connections in the test. The commit title also carries a stray bullet.

I'm not able to replicate the "no nodes available in query plan" error, but the basic point is correct. The broadcast address was a red herring: the issues were more basic connection management. See below.