CASSANDRA-21506: Avoid Python dtest failure cascade by waiting for node socket release

[nivykani]

Ensure that CCM fully stops the node and releases addresses when wait=True.

Additionally, during start_cluster(), we can directly wait on ports to be available by retrying every 1 second up to the default timeout of 15 seconds per address.

Note: Fails fast; if one address is still not released by the timeout (default=15 seconds), it returns UnavailableSocketError rather than waiting for the others.

Each interface could use up almost the full 15s timeout in the worst case, which makes tests slower but more likely to succeed.

Cassandra JIRA

[aweisberg]

I don't think this really addresses the underlying issue which is the socket not being bindable. What are we waiting on? How often was this error occurring and did you run it enough to be reasonably confident this fixed the problem?

The socket should be immediately bindable (since SO_REUSEADDRESS should be set) once the C* process exits and we are supposed to be waiting and making sure it has exited before starting the next test.

This also doesn't help make the problem more debuggable when it does occur. I think assert socket available is a fine check for now and the only enhancement would be log more debug info after failing to bind the socket.

I think the JIRA is wrong about what to do here. The thing to do here is add debugging to CCM (both permanently and as part of working on the JIRA) in the assert socket is bindable function. Then run it repeatedly until the problem reproduces and we can then see the debug output and understand the nature of the problem.

I tasked Claude with writing some debug code that works on both Mac and linux and is as maximally thorough as it can be so you only have to go through the exercise of reproducing it once (hopefully). Will leave it as an attached comment once it finishes.

[aweisberg]

On behalf of Ariel Weisberg (posted by Claude): proposed diagnose_port() addition to ccmlib/common.py that fires from assert_socket_available() on bind failure. Tested on macOS (native) and Linux (debian:bookworm-slim container), all 4 diagnostic sections produce correct output with proper PID identification.

Diff: https://gist.github.com/aweisberg/68a18b46b81c0b1f2e22323a21511c19