Add multi-waiter support for TCP send - #116
Open
nding0405 wants to merge 6 commits into
Open
Conversation
Allow a single thread to wait for incoming connections across multiple listening sockets at once using multiwaiter_wait(), instead of having to block in network_socket_accept_tcp() on one socket at a time. Each socket now exposes an array of futex called eventFutexState, that a caller can register them with a multiwaiter. For this PR, only the added the accept functionality, others remains commented out for now. A thread can watch the accept futexes of several listening sockets simultaneously, block in multiwaiter_wait() until one of them signals a ready connection, and then accept that connection. The blocking wait therefore moves out of accept() and into the multiwaiter: accept() is called only once a connection is known to be ready, so it no longer has to block, which is what makes it possible to wait on several listening sockets at the same time from a single thread. Key changes: Added Event futex array (tcpip-internal.h, NetAPI.h) for socket wrapper: Only SocketAcceptEvent is implemented today, room reserved for future receive/send events. The value of the futex counts the ready events currently pending on the socket. For SocketAcceptEvent, it represents the number of connections that are ready to be accepted. The reserved value SocketNotAvailable (0xFFFFFFFF) means the socket will be freed soon. Read-only getter: network_socket_get_event_source() returns a capability to a socket's event futex stripped to read-only. Producing events: on_tcp_connect increments the SocketAcceptEvent futex and calls notify_all() when a new connection becomes ready. To let the callback can find the wrapper from the raw socket, network_socket_create_and_bind() now applys a bidirectional link between the wrapper and raw socket. Consuming events: network_socket_accept_tcp() decrements the futex after it successfully accepts a connection. This only keeps the futex value align with the semantic desbribed above. It does not call notify_all(), since dequeuing an existing connection is not a new, so no need to wake the threads up. Avoid waiting forever when socket is not available: the futex lives in the socket wrapper's heap memory, so a thread must never be left blocked on a socket that is being freed. Both network_socket_close() and the reset handler (reset_network_stack_state) set every event futex to SocketNotAvailable and call notify_all() before that memory goes away. The value change wakes any thread blocked in multiwaiter_wait(), which then observes the sentinel and returns instead of sleeping on a dead socket. A subsequent accept() will see the sentinel and reports the socket as unavailable.
Allow a single thread to wait for incoming connections across multiple listening sockets at once using multiwaiter_wait(), instead of having to block in network_socket_accept_tcp() on one socket at a time. Each socket now exposes an array of futex called eventFutexState, that a caller can register them with a multiwaiter. For this PR, only the added the accept functionality, others remains commented out for now. A thread can watch the accept futexes of several listening sockets simultaneously, block in multiwaiter_wait() until one of them signals a ready connection, and then accept that connection. The blocking wait therefore moves out of accept() and into the multiwaiter: accept() is called only once a connection is known to be ready, so it no longer has to block, which is what makes it possible to wait on several listening sockets at the same time from a single thread. Key changes: Added Event futex array (tcpip-internal.h, NetAPI.h) for socket wrapper: Only SocketAcceptEvent is implemented today, room reserved for future receive/send events. The value of the futex counts the ready events currently pending on the socket. For SocketAcceptEvent, it represents the number of connections that are ready to be accepted. The reserved value SocketNotAvailable (0xFFFFFFFF) means the socket will be freed soon. Read-only getter: network_socket_get_event_source() returns a capability to a socket's event futex stripped to read-only. Producing events: on_tcp_connect increments the SocketAcceptEvent futex and calls notify_all() when a new connection becomes ready. To let the callback can find the wrapper from the raw socket, network_socket_create_and_bind() now applys a bidirectional link between the wrapper and raw socket. Consuming events: network_socket_accept_tcp() decrements the futex after it successfully accepts a connection. This only keeps the futex value align with the semantic desbribed above. It does not call notify_all(), since dequeuing an existing connection is not a new, so no need to wake the threads up. Avoid waiting forever when socket is not available: the futex lives in the socket wrapper's heap memory, so a thread must never be left blocked on a socket that is being freed. Both network_socket_close() and the reset handler (reset_network_stack_state) set every event futex to SocketNotAvailable and call notify_all() before that memory goes away. The value change wakes any thread blocked in multiwaiter_wait(), which then observes the sentinel and returns instead of sleeping on a dead socket. A subsequent accept() will see the sentinel and reports the socket as unavailable.
- Claim `signal_event_futex` and `consume_event_futex` are owned by
SealedSocket
- Use `TimeoutWaitForever` instead of timeout{UnlimitedTimeout}
A TCP child socket can become visible and accept-ready to `FreeRTOS_accept()` before `on_tcp_connect()` increments the accept futex. If a waiter accepts the child while the counter is zero, the old futex consume helper does not decrement it to -1, because -1 represents an invalid futex. The delayed `on_tcp_connect` callback then increments the counter to one, though no connection is pending. This leaves pending multi-waiters with a misleading futex. They will repeatedly treat the socket as ready, call `accept()`, and never block, defeating the purpose of multi-waiter waiting. Use a signed counter and always decrement after a successful accept. An early accept records a temporary negative debt that the callback later repays. Reserve INT32_MIN for an invalid futex so that normal negative debt cannot conflict with the sentinel value.
Add a TCP send event for multiwaiter, so one thread can wait for free send space on several connected TCP sockets at once. Workflow: The caller first calls `network_socket_send()` with a zero timeout. The first non-empty send creates `txStream` and sets the send event to its exact free space. A send queues as many bytes as it can and subtracts that count from the send event futex. When the peer ACKs bytes, `on_tcp_sent()` adds that count to the futex and wakes all waiters. If more bytes remain, the caller waits with multiwaiter and then tries the send again. If the connection closes, the waiter wakes and its next call to any network API will returns `-ENOTCONN`. Key changes: - Add `SocketTCPSendEvent`, whose corresponding futex represents the free byte count in `txStream`. - Keep `txStream` setup lazy just like FreeRTOS_send(), and use `FreeRTOS_tx_space()` for its usable size. - Register the sent callback for TCP clients and listeners. Because accepted sockets inherit it from their listener, although the listening socket itself cannot send application bytes. - Add `SocketConnectionClosed` for a disconnected TCP state. Keep `SocketNotAvailable` for a socket wrapper that is being removed. - On disconnect, set the send event futex to `SocketConnectionClosed` and wake its waiters.
nding0405
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a TCP send event for multiwaiter, so one thread can wait for free send
space on several connected TCP sockets at once.
Workflow:
The caller first calls
network_socket_send()with a zero timeout. The firstnon-empty send creates
txStreamand sets the send event futex value toits exact free space. A send queues as many bytes as it can and subtracts
that count from the send event futex. When the peer ACKs bytes,
on_tcp_sent()adds that count to the futex and wakes all waiters. If more bytes remain, the caller
waits with multiwaiter again. If the connection closes, the waiter wakes and its next
call to any network API will returns
-ENOTCONN.Key changes:
SocketTCPSendEvent, whose corresponding futex represents the free bytescount in
txStream.txStreaminitialization lazy just like FreeRTOS_send(), and useFreeRTOS_tx_space()for its usable size.sockets inherit it from their listener, although the listening socket itself
cannot send application bytes.
SocketConnectionClosedfor a disconnected TCP state. KeepSocketNotAvailablefor a socket wrapper that is being removed.SocketConnectionClosedand wakeits waiters.