Bound blocking I/O on data plane transport sockets - #837
Open
sanjeevrg89 wants to merge 1 commit into
Open
Conversation
The H2H data plane sets TCP_NODELAY and socket buffer sizes and nothing else, so every send(), recv(), and connect() on a data plane socket blocks with no deadline. A peer that holds its connection open but stops reading parks the sender inside writev() for as long as that peer lives. BlockTransport runs those calls inline on a fixed pool of socket worker threads, so one unresponsive peer also stalls transfers to every healthy peer. Set SO_SNDTIMEO, SO_RCVTIMEO, TCP_USER_TIMEOUT and TCP keepalive on both the connect and the accept side, and bound connect() with a non-blocking connect plus poll(). Two flags configure it: --raiden_transport_io_timeout (default 5m) --raiden_transport_connect_timeout (default 30s) The defaults are ceilings on a stalled peer, not latency targets, so a healthy transfer behaves exactly as before. Setting raiden_transport_io_timeout to zero restores the previous unbounded behavior in full. The two mechanisms cover different failures. SO_SNDTIMEO and SO_RCVTIMEO bound one blocking call, which catches a peer that stops reading. Keepalive catches the opposite case, an idle pooled connection whose peer vanished without sending FIN, where no data is in flight for a send deadline to expire on. TCP_USER_TIMEOUT replaces the tcp_retries2 default of roughly fifteen minutes for data already in flight. SO_SNDTIMEO and SO_RCVTIMEO expire as EAGAIN or EWOULDBLOCK even on a blocking socket, a case TcpSocketUtil previously treated as unreachable via DCHECK(!WouldBlock(...)). Those loops now report a deadline as DEADLINE_EXCEEDED and a kernel connection timeout as UNAVAILABLE, so a caller can tell a stalled or dead peer from a protocol error. A connect that runs out of time reports DEADLINE_EXCEEDED; every other connect failure keeps the historical UNAVAILABLE that the transfer failure metric labels on. This bounds the failure, it does not retry it. A failed transfer still fails its whole plan and recovery stays with the caller. The control plane already does this in kv_cache/reshard/framed_rpc.cc. This brings the data plane in line with it. Tests: a peer that accepts and never reads now fails the send with DEADLINE_EXCEEDED instead of blocking; ConnectToPeer returns a socket that is bounded and back in blocking mode, which send and recv assert on; a refused peer still reports UNAVAILABLE; and a zero io_timeout leaves the socket untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The H2H data plane sets TCP_NODELAY and socket buffer sizes and nothing
else, so every send(), recv(), and connect() on a data plane socket
blocks with no deadline. A peer that holds its connection open but stops
reading parks the sender inside writev() for as long as that peer lives.
BlockTransport runs those calls inline on a fixed pool of socket worker
threads, so one unresponsive peer also stalls transfers to every healthy
peer.
Set SO_SNDTIMEO, SO_RCVTIMEO, TCP_USER_TIMEOUT and TCP keepalive on both
the connect and the accept side, and bound connect() with a non-blocking
connect plus poll(). Two flags configure it:
--raiden_transport_io_timeout (default 5m)
--raiden_transport_connect_timeout (default 30s)
The defaults are ceilings on a stalled peer, not latency targets, so a
healthy transfer behaves exactly as before. Setting
raiden_transport_io_timeout to zero restores the previous unbounded
behavior in full.
The two mechanisms cover different failures. SO_SNDTIMEO and SO_RCVTIMEO
bound one blocking call, which catches a peer that stops reading.
Keepalive catches the opposite case, an idle pooled connection whose peer
vanished without sending FIN, where no data is in flight for a send
deadline to expire on. TCP_USER_TIMEOUT replaces the tcp_retries2 default
of roughly fifteen minutes for data already in flight.
SO_SNDTIMEO and SO_RCVTIMEO expire as EAGAIN or EWOULDBLOCK even on a
blocking socket, a case TcpSocketUtil previously treated as unreachable
via DCHECK(!WouldBlock(...)). Those loops now report a deadline as
DEADLINE_EXCEEDED and a kernel connection timeout as UNAVAILABLE, so a
caller can tell a stalled or dead peer from a protocol error. A connect
that runs out of time reports DEADLINE_EXCEEDED; every other connect
failure keeps the historical UNAVAILABLE that the transfer failure metric
labels on.
This bounds the failure, it does not retry it. A failed transfer still
fails its whole plan and recovery stays with the caller.
The control plane already does this in kv_cache/reshard/framed_rpc.cc.
This brings the data plane in line with it.
Validation
DEADLINE_EXCEEDEDinstead of blocking.ConnectToPeerreturns a socket that is bounded and back in blocking mode, which send and recv assert on.UNAVAILABLE.io_timeoutleaves the socket untouched.//tpu_sync/transport/lib/socket:util_test(PASSED in 7.2s)//tpu_sync/transport/peregrine/src/internal/socket:tcp_socket_util_test(PASSED in 18.2s, verifiedSendReportsDeadlineExceededWhenPeerStopsReadingnon-hanging)//tpu_sync/transport/lib:raw_buffer_transport_test(PASSED in 16.9s)//tpu_sync/transport:block_transport_test(PASSED in 18.9s, verified telemetry assertions expectingerror_code="UNAVAILABLE")//tpu_sync/transport/peregrine/src/internal/socket:all(PASSED 6/6 tests)