netring stage 3: a TCP stream is one region and one connection (design round)

[Japabu]

What this round is

The design round of stage 3 of issues/netstack-moves-every-byte-through-two-kernel-pipes.md (the track #863 landed). One empty commit; nothing is built until the orchestrator accepts this design in a comment here. The track's design, its amendments A to E and its rulings 1 to 14 (#863's body and thread) are taken as accepted; this body is how stage 3 lands them in the code, the three places where landing them reaches past what the track says, and the roast of that.

Stages 1 (#867, 867ec23ed: fstat's DeviceMemory, SharedMemory::lent) and 2 (#878, 4ca63a698: toyos-transport's stream.rs) are on main at 52e32385d, the base of this branch. Nothing stage 3 needs is unbuilt.

No kernel addition. The data path stays userland's, the owner's constraint on every stage: the kernel makes and maps the region and carries doorbells and hang-ups on pipes, as it does today. Under ruling R1 below, the kernel loses SYS_CONNECTION_JOIN here rather than in stage 4.

Three rulings this design asks for

R1. Accepted streams move onto the region in stage 3 (recommended), or stay piped until stage 4. The track keeps [tcp]'s heap storage "for stage 4's passive children" and puts the accept's copy into the region, and its two listener cases, in stage 4's exit. Read literally, stage 3 makes only a connected stream a region, so every client layer carries two kinds of stream for one stage: std's TcpStream, libc's socket entry, mio's and socket2's arms and netstack's serve.rs keep the piped stream beside the region one; TcpShutdown, TcpSetOption, TcpClose for streams and make_socket_fd stay for accepted streams alone; and stage 3's handle rules cover only half the streams. TcpStream::from_raw_fd of an accepted stream's handle would panic with "no end in this process", since toyos::net enters only region ends.

• Option B (recommended): stage 3 is every stream. MsgType::TcpAccept { listener } carries a fresh region, netstack copies what the passive child holds in [tcp]'s heap storage (in-order text, out-of-order ranges, the FIN) into the region's receive ring at its offsets before it answers, as amendment C designed, and answers TcpAcceptResponse { remote_addr, remote_port, local_port, nodelay } with the bell (ruling 6). The accept still takes the listener's wake first, as std, libc and mio do today, so no region is made before a child is there (ruling 5 holds as it holds today). Stage 4 keeps the listener's connection in place of its wake pipe, the datagram socket and the *Piped bind messages. Stage 3's exit then takes stage 4's two listener cases (a_listener_answers_a_syn_and_wakes_its_owner_when_the_handshake_ends over the region, and ruling 1's text-past-a-hole-and-FIN case with its three controls). make_socket_fd loses its last caller, so SYS_CONNECTION_JOIN, ConnectionEnd::joined and their abuse tests go here, as dead code goes.
• Option A: as the track reads. The two stream kinds above for one stage, and none of B's moves.

The rest of this body assumes B. Under A, everything below applies to connected streams only, and the piped path stays as it is.

R2. mio and socket2 reach a stream through std, not through their own toyos::net. The track says std, libc, mio and socket2 "read and write the rings through toyos::net", and amendment E keeps one table of the process's ends there. But a Rust program links two instances of toyos: std's, built with rustc-dep-of-std, and the one a crate that names toyos links (issues/a-rust-program-holds-two-copies-of-its-stream-state.md). mio and socket2 name toyos directly. A stream mio connects would be entered in mio's instance, and std's from_raw_fd, OwnedFd's drop, File::from_raw_fd and the spawn checks, all in std's instance, would not see it. So two instances would hold two tables, and the refusals would answer wrongly in both directions. The design: the one end table is std's instance's, which toyos::net holds. mio's and socket2's ToyOS arms hold a std::net::TcpStream and read, write, connect and shut down through it, as their unix arms do. std::os::toyos::net (in sdk/std/os/) gives them what std has no word for: a stream's two doorbell handles to watch (the connection for READABLE, the bell for WRITABLE) and register(interest), which runs amendment D's arm and ring. libc is one instance in a C program and uses toyos::net directly. A Rust program that calls toyos::net's stream API itself reaches its own instance's table; the module header says so, and std is the only Rust caller in the tree.

R3. toyos-transport and toyos-untrusted join std's dependency graph. #863's unsure item. The client end is toyos-transport's code, which std reaches through toyos. Both crates become published SDK crates: they join build/sdkversion.rs's PUBLISHED before toyos, gain the rustc-dep-of-std feature toyos and toyos-abi have, and join build/sysroot.rs's std_patch, SYSROOT_SOURCES and SYSROOT_MANIFESTS. The rust/ fork's library/Cargo.lock gains both. The sysroot's key then moves when either changes, which is right, since std links them. The alternative, a second copy of the ring's decisions inside toyos, would be a sibling of stream.rs, and is refused.

The design, layer by layer

toyos-transport (lib/blockring/transport): the session's decisions, generic and loom-modelled

• stream.rs gains amendment B's ring-back, as netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 says it would: Consumer::ring_back(page, ended) -> bool and Producer::ring_back(page, ended) -> bool exchange the peer's want for 0 and answer whether to ring, but only when the direction is ready: bytes past the head, or ended, for a consumer's peer; room, or a fatal ended, for a producer's peer.

• A new session.rs: a duplex byte stream of two rings and two end words, over N words and two doorbells. Nothing in it is network-specific. An end word is 0 while open; ORDERLY is the producer's last byte published; any other value is a failure that also makes the other direction ready to every look (ruling 2). The protocol says what the values mean.

  • The client end has two directions. Each is a ring end and a waiter count behind a lock the adapter supplies, and the read, write, poll-look and register sequences are written once here, each generic over Word, a Bells trait (drain this direction's doorbell, wait on it until a deadline, ring the server) and a Lock trait. The sequences are amendment B's: a direction's end is held to drain, arm, look and copy, and never across a wait. A waiter counts itself in before it drains and out when it returns. A thread that took the doorbell, by a wait or a drain, and leaves the end with another waiter counted arms again, looks, and rings the server if the direction is ready. WouldBlock is answered only after the drain, the arm and the second look. write loads the receive end (Acquire) before it copies, and a failure is its answer with nothing copied (ruling 3). register arms each named want, looks, and rings the server if a named direction is ready (amendment D); it drains nothing and counts nothing.
  • The server end (netstack's): rung exchanges each client want whose direction is ready for 0 and answers which doorbells to ring (amendment B); fail(code) stores the receive end (Release) after the last tail, then exchanges both client wants for 0 and answers each that was set (amendment A, ruling 2); publishing, releasing and the orderly end answer their ring as stream.rs does.
  • The adapter copies bytes through the Spans it is handed, and blocks only inside its Bells::wait. The crate still maps, copies and blocks nothing itself.

• The three exit models are tests/loom.rs cases that drive the sequences above. Their bells are loom models of a pipe (a byte count under a loom mutex, read level-triggered, a wait that takes a byte), and their lock is loom's mutex. The kernel's deliver answers a watch by the pipe's level at its look and arms it again if empty; the poll's watch models that.

  1. Two sleepers: a reader asleep on rx_data and a writer asleep on tx_room, while the server releases room and publishes data in either order, end with neither waiting. Control: the one-doorbell feature rings both wants on the connection's doorbell.
  2. Same direction: a blocking call and a poll of one direction, the blocking call leaving bytes, the receive end or room behind, end with the poll answered. Control: no-pass-on.
  3. Registered while ready: in both directions, with the receive end, and with a fatal end while the send ring is full, each made ready before the registration or racing it, the source gets its first event (eight cases). Controls: register-no-arm, register-no-ring, both together, stage 2's want-relaxed, and head-only, under which a look reads only the send ring's head and not the fatal end.

  Each control is a declared feature that no shipping build enables. Each joins build/ci.rs's CONTROLS under its case's name, and build/clippy.rs gets a shape for each. Bound: the five preemptions netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's models used. The unbounded run of each green case is posted as evidence, with its verdict or its time-out.

toyos::net (sdk/toyos/src/net.rs): the protocol, the region's layout and the process's ends

• The region, one declaration that netstack and every client read. It is 8 MiB, four 2 MiB pages from SYS_SHM_CREATE. The header is the first 4 KiB: 1,024 words, each one the protocol uses on a 64-byte line of its own. The receive ring is the next 4 MiB (C = 4,194,304), and the send ring the last 4 MiB less 4 KiB (C = 4,190,208). The header's words are the two rings' heads and tails, the four wants, the receive end (netstack's: ORDERLY is the peer's FIN, then RESET, TIMED_OUT, UNREACHABLE), the send end (the client's: shut down for writing), and nodelay (the client's). They are two stream::Places and three words.
• Messages. TcpConnect { addr, port, timeout_ms, nodelay } carries one handle, the region. Its answer is TcpConnectResponse { local_port } with one handle, the bell's read end, or an error. TcpAccept { listener } with a region is answered TcpAcceptResponse with the bell (R1). Both go on a connection that is then the stream's. TcpConnectPiped, TcpAcceptPiped, TcpShutdown and TcpSetOption go, and TcpClose stays only for a listener. The client writes the region's nodelay before it sends the connect, so libc's "nodelay on from the first segment" needs no second request.
• The process's ends. A fixed table in the instance std links (R2) maps each handle the process holds of a stream to its end. An end holds the region's mapping, the bell, the two directions' locks and counts, its handle count, and the count of calls inside it, so an end is freed only once no handle and no call is left. The table's own lock and each direction's lock are a three-state futex lock over SYS_FUTEX_WAIT and SYS_FUTEX_WAKE, the published design of Drepper's "Futexes Are Tricky" (2011, §"Mutex, Take 2"). toyos has no lock and allocates nothing, and keeps it that way: the table is two static arrays, MAX_STREAMS = 128 ends and 256 handles. That is an estimate, the per-process bound libc's socket table already has (MAX_SOCKETS). Past it a connect or an accept is refused ResourceExhausted before anything is made.
• Its API is what std and libc call: Stream::connect, accept, read, write, shutdown_write, set_nodelay, nodelay, register, look, adopt(handle) (a handle the process holds an end for, sharing it; otherwise Refused::NotThisProcess, named, with nothing written on the connection), duplicate, let_go(handle), and is_stream(handle). adopt and is_stream are lookups and write nothing.
• netstack gone reads as a drain that answers 0 on the connection or the bell. The stream is then ended without a word from netstack, and every call answers ConnectionAborted.

[tcp] (system/netstack/shard/tcp): its buffers are the region's rings

• Ring keeps its heap storage and gains a lent one. It holds a Storage, the heap Vec it has today or a Box<dyn Lent>, which copies bytes in and out at ring offsets and never hands out a slice of memory a client can write. [tcp] stays forbid(unsafe_code): the copy through the mapping is netstack's adapter's.
• Receive: place writes an out-of-order segment's payload at its offset in the lent ring, and commit advances its tail. The window is the ring's room as [tcp] counts it, so the client's head is [tcp]'s consume. netstack reads the head and hands [tcp] after_read(n), as the user's read does today, which keeps the capacity rule of rx's module: it grows by what the client reads, from RECEIVE_BUFFER_INITIAL up to the configured buffer, never past the 4 MiB ring.
• Send: the bytes the client published are what [tcp] segments from. queued(n) counts them in with no copy, and an acknowledgment's consume is netstack's head store. Exit::send gets a payload it copies out of the ring into the segment once, before the checksum. A client that rewrites a byte after publishing it corrupts only its own stream (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's hostile-client section).
• The accept (R1): Tcp::lend(conn, rx, tx) moves a passive child onto lent storage, copying what its heap holds to the same offsets, the out-of-order ranges included, and the FIN stays [tcp]'s state. The heap storage is then dropped.

The node (system/netstack/node) and netstack (system/netstack/src)

• A stream is a [tcp] connection, its lent rings and its session server end. A pass moves what the frames and the client's doorbells changed:

  • publish the receive tail and ring rx_data;
  • read the client's head into after_read and release room, ringing tx_room;
  • take newly published send bytes into [tcp];
  • read nodelay and the send end, which is the FIN once [tcp] has segmented to that tail;
  • and on a failure, fail(code).

  The node never blocks on a client. Everything it reads from the region is a word it judges, as netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's hostile-client section says: a cursor past its bound is a Violation that resets that stream alone.

• netstack's own wants. tx_data is armed when netstack has taken every published byte and [tcp] has room. rx_room is armed only while the window last offered is below [tcp]'s update threshold (RFC 9293 §3.8.6.2.2), the one state in which a read owes the peer a segment. Otherwise netstack reads the head on its next pass.

• The client leaving is the connection's hang-up (OTHER_END_GONE). The node closes the connection to [tcp] with what the client published, which [tcp] sends before its FIN (RFC 9293 §3.10.4), or resets it when unread text is left (§3.6.1), as close does today. [tcp] owns the orphan from then (ORPHAN_IDLE). OWNERLESS_LIFE, OWNERLESS_PER_PEER and the cut's clock go: they exist because a pipe cannot say its reader left.

• netstack's shell: serve.rs takes a region through SharedMemory::lent, refusing by name, before anything is mapped, a handle that is no region, one that is not memory the kernel allocated, and one that is not exactly 8 MiB. It makes the bell and keeps the stream on its connection, and watches each stream once, the connection for READABLE and OTHER_END_GONE, where today it watches two pipes. It drains doorbells non-blocking. WATCHES_PER_PLACE falls from 2 to 1. PLACE_BYTES becomes the region and its three doorbell pipes' pages, 14 MiB, until issues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md closes. The region is the client's allocation, and netstack's mapping and handle keep it alive, so it is counted as a place, as the pipes are counted today.

std, the rust/ fork, libc, mio and socket2

• std's TcpStream (sdk/std/sys/net/connection.rs) is a handle entered in toyos::net, with the timeouts and the nonblocking flag it holds today. read, write, shutdown, set_nodelay, nodelay and duplicate call toyos::net. take_error answers the receive end's failure. make_socket_fd, ended, RESET and stream_refused go, and so do toyos::net's read_end, write_refused and StreamRefusal.
• Raw handles (sdk/std/os/io.rs, ToyOS's own file, where AsRawFd for TcpStream already is):
  • FromRawFd, IntoRawFd, From<OwnedFd> and From<TcpStream> for OwnedFd for TcpStream. from_raw_fd panics with adopt's refusal. into_raw_fd leaves the entry for the number it gives out.
  • File::from_raw_fd asks is_stream and panics with the refusal if the handle is a stream's.
  • A spawn asks is_stream of every handle it would put in a child's slot by Stdio::inherit(), a File or inherit_handle into any slot, and of every handle endow or provide would move. It answers the refusal before anything moves, with no child started.
  • OwnedHandle::from_raw's # Safety excludes a handle toyos::net answers for, so send_handles refuses one by its type (ruling 12).
• The rust/ fork: a commit on its branch changes the bodies of the two existing ToyOS arms in library/std/src/os/fd/owned.rs. OwnedFd's drop calls toyos::net::let_go, and BorrowedFd::try_clone_to_owned calls toyos::net::duplicate. Each is a lookup and then the raw syscall for a handle that is no stream's. The commit adds toyos-transport and toyos-untrusted to library/Cargo.lock (R3). The gitlink moves in this pull request.
• libc (sdk/libc/src/socket.rs): a stream's entry holds its handle in place of rx_fd and tx_fd. connect, accept, send, recv, shutdown, setsockopt(TCP_NODELAY) and close call toyos::net. recv answers each end in its POSIX word: ECONNRESET, ETIMEDOUT, EHOSTUNREACH. libc's poll of a socket asks the rings first, then counts in, drains, arms and looks again under each direction's end (amendment B), and watches the connection for POLLIN and the bell for POLLOUT.
• mio and socket2 (R2): one commit each on its toyos branch, and a lockfile bump. mio's TcpStream wraps std::net::TcpStream; it registers the connection for READABLE and the bell for WRITABLE, and runs register on each register and reregister. Its listener's accept takes the wake and calls std's accept. socket2's ToyOS Socket holds a std TcpStream once connected, or accepted, in place of its own table's joined handle.

What it deletes

netstack's src/pipes.rs ToClient/FromClient half, the node's ToClient, FromClient, ReadRefusal, Pipes, PipeEnd, pipe_gone, pipe_broken, the end ordering, the CHUNK read, drained, and the ownerless clock. serve.rs loses data_pipes, two watches a stream, TOKEN_FROM_CLIENT and TOKEN_TO_CLIENT. toyos::net loses TcpConnectPiped, TcpAcceptPiped, TcpShutdown, TcpSetOption, read_end, write_refused and StreamRefusal. std loses make_socket_fd and the empty-write probe. Under R1-B the kernel loses SYS_CONNECTION_JOIN and ConnectionEnd::joined. The listener's wake pipe and every datagram pipe stay for stage 4.

Issues this reaches: netstack-spends-two-poller-slots-per-piped-connection.md and netstack-watches-both-pipes-of-every-connection-on-every-pass.md (a stream is one watch, and a datagram socket is still a pipe until stage 4), and a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md (its exit names a pipe's word, which this does not build). Each is touched only if its own exit is met. The track records stage 3 done.

Checks: each exit item, its tier and its control

exit item	where	tier, and why	red under
two sleepers	toyos-transport tests/loom.rs	loom over the code std and netstack run	one-doorbell
poll beside a blocking call: bytes, end, room	same	same	no-pass-on
registered while ready, eight cases	same	same	register-no-arm, register-no-ring, both, want-relaxed, head-only
write with room after a reset: ConnectionReset, nothing copied	toyos-transport unit test of the client end	host: the decision is the end's, over heap words	the write copying before it loads end
a foreign handle refused by name, nothing written; into_raw_fd, From<TcpStream> for OwnedFd, try_clone_to_owned taken back share the one end and its cursors	tests/guest/rust-tests	guest: std's ToyOS arms build only for toyos and stand on the kernel's handle table; a fake netstack, a port of the test's own served to a child as netstack, as netstack_gone_mid_bind.rs does, plays netstack's side with toyos-transport's server end, so no NIC or peer is needed, and reads its end of the connection to see nothing written	giving the handle out lets its entry go
File::from_raw_fd of a stream's handle panics with the refusal	same	same	from_raw_fd taking it silently
stdio slot, inherit_handle past the stdio slots, endow, provide: refusal, no child, the stream reads and writes through the handle after	same, one case a path	same	the path taking it silently; for endow and provide, the handle moving and its entry staying
R1-B: a_listener_answers_a_syn_... over the region; text in order, past a hole and a FIN before the accept, the hole filled after	system/netstack/node/tests/listeners.rs	host: the node is pure	dropping at the accept the in-order bytes, the out-of-order range, the FIN
pipe_copy_bytes < 0.01 per byte downloaded, C0, hold and spin per MB beside the scout's rows	T14, internet_download with the scout's instrument (git diff 777c96b2b 2f9e090b8) carried onto stage 3's head, less the netstack stamps on the pipe writes stage 3 deletes; never landed	metal	stage 3's base

The node's stream tests (node/tests/streams.rs, 1,482 lines by wc -l, most of them asserting pipe behaviour) are rewritten over a heap-backed region with the real toyos-transport client end as the client. A pipe-behaviour case with no region meaning is cut, and the body names each one. Stage 3's negative control for the whole change is the guest suite and the host suite run on main with the new tests applied, where the new tests must red. The independent oracles are RFC 9293, POSIX.1-2024 (send, recv, select, accept) and virtio 1.2 §2.9, the track's, and the host's TCP in the node's existing differential tests.

Gates the build round runs: cargo run -- --ci host; cargo run -- --build-only; cargo test for the guest suite (every boot that starts netstack reaches this: sshserver's tokio listener, the desktop's, netcase); each control by CONTROLS; the T14 image by --metal-readback.

The roast

• Size. By the files it rewrites, not measured: [tcp]'s ring.rs, rx.rs and tx.rs at their edges; the node's streams.rs and listeners.rs, with their tests; serve.rs; toyos::net; std's connection.rs and os/io.rs; libc's socket.rs and poll; two forks and rust/; the build's publish and sysroot lists. New: session.rs and its models, the end table and its futex lock. Production should come out near even. The piped stream code it deletes is about what the session's client end adds, and the tests grow by the exit's cases.
• R3 makes two crates published SDK crates, which main's release then publishes. Their API becomes a published API at a version that moves with every change, under the SDK's 0.N.0+key rule.
• The end table is a fixed bound, 128 streams a process, and a new limit. The alternative is to give toyos an allocator, which no part of the SDK has. A process past the bound is refused ResourceExhausted by name. The bound is an estimate.
• Every OwnedFd drop and try_clone_to_owned in every ToyOS Rust program takes the table's lock for a lookup, and so does each File::from_raw_fd and each handle a spawn moves (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's roast). Not measured.
• 8 MiB of contiguous memory is committed at every connect and accept, as netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 accepted, and a fragmented machine can refuse one a pipe would have taken.
• A doorbell is a pipe byte through the global PIPES lock. On a download, a client read arms rx_data at most once per empty ring, and netstack rings rx_room only below the window threshold. The T14 boot reads what is left of the lock.
• R2 narrows the track's sentence: mio and socket2 go through std, not through their own toyos::net. A Rust program that drives toyos::net's streams itself has a second table that std cannot see. Nothing in the tree does that, and the module header says so.
• A received region's header is the client's to scribble: each word is a bit or a clamped cursor, and the worst costs that stream alone (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's hostile-client section, unchanged).

What I am unsure of

• ConnectionAborted for a stream whose netstack went away: it is neither the peer's reset nor its FIN, and std has no closer word.
• Whether MAX_STREAMS = 128 is the right fixed bound, or whether toyos should take alloc for this one table.
• Whether rx_room armed only below the update threshold costs the receive autotuning anything on the T14. It reads the client's reads at the next pass, not at the read. The boot shows the throughput beside the scout's 785.95 Mb/s.
• Whether the five preemptions bound netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's models used reaches every interleaving of models that now run the real sequences, which are longer than netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's sketches.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

[Japabu]

Rulings on R1-R3, and acceptance.

• R1: accepted streams move onto the region in stage 3, with amendment C's copy, stage 4's two listener cases, and SYS_CONNECTION_JOIN's deletion. One kind of stream at every stage.
• R2: mio and socket2 reach streams through std, plus the small std::os::toyos::net extension. One copy of the stream state.
• R3: toyos-transport and toyos-untrusted become published SDK crates in std's graph.

The track's text moves with these in the build's diff. Accepted: build stage 3.