netstack hands the node a pass's received frames as one batch, about 15% fewer pipe calls on the T14, and a reset keeps the text received before it, as Linux and the BSDs do

issues/a-stream-its-peer-reset-refuses-the-option-requests-a-host-answers.md

@@ -9,7 +9,8 @@ opened: 2026-10-08
 The node lets a stream go once it holds neither of its pipe ends
 (`userland/netstack/node/src/streams.rs`), and netstack's id for it goes with
 it (`userland/netstack/src/serve.rs`, `Sockets::settle`). A peer's reset ends
-both pipes at once, so a client that still holds the stream names an id
+both pipes once the client's pipe has taken the text received before it,
+so a client that still holds the stream names an id
 netstack no longer has, and `Sockets::set_option` answers it
 `ERR_NOT_CONNECTED`.
 

issues/netstacks-virtio-receive-has-no-per-pass-budget.md

@@ -0,0 +1,31 @@
+---
+status: open
+kind: defect
+opened: 2026-10-10
+---
+
+# netstack's virtio receive has no per-pass budget
+
+`VirtioNet::poll_rx` (`userland/netstack/src/virtio_net.rs`) answers a frame for
+as long as the used ring holds one, and `Node::receive`
+(`userland/netstack/node/src/lib.rs`) takes frames until the card says none is
+left. A device that refills the ring as fast as it is read keeps one pass in
+receive: the transmit opportunity, the deadlines and every stream's pass wait
+for the ring to empty, and the whole batch is read at one `now`. The I219 is
+bounded at `toyos_i219::RX_BUDGET` frames a pass.
+
+The I219's bound is not one to copy. A pass that stops at it leaves frames whose
+interrupts the pass's `begin_pass` already took, and nothing in netstack's loop
+(`userland/netstack/src/main.rs`) makes the next wait return at once, so they
+wait for the next interrupt or deadline. Read from the code, not measured.
+
+The fix is one bound for every driver where the pass takes its frames, at
+`Node::receive`, answering that it stopped at the bound so the loop's next wait
+is zero; the I219's own budget is then deleted. That touches `toyos-i219`, which
+the branch that found this (#828) was not briefed to change.
+
+**Exit condition**: a host test feeds `Node::receive` a source that never empties
+and sees it stop at the bound and answer that a pass is owed; netstack's loop
+waits zero after such a pass; `toyos_i219::RX_BUDGET` is gone.
+
+**Owner**: whoever holds `issues/toyos-has-its-own-network-stack.md`.

issues/toyos-has-its-own-network-stack.md

@@ -30,9 +30,9 @@ What the node does not yet meet:
 - A listener's handshakes in progress are [tcp]'s `limits::LISTEN_PENDING`, each held until `limits::SYNACK_GIVE_UP`, and a SYN past them is dropped: 256 SYNs a minute from addresses that never answer shut a port to every other peer for as long as they keep coming. [tcp] has no SYN cookie and lets no older handshake go for a newer one. Exit: a test in `toyos-net-shard/tcp/tests/` in which a peer's handshake completes while LS-12's flood runs.
 - The node counts a wake unspent until an accept arrives, and cannot tell an accept on its way from one that will never be sent: an owner that reads a wake and sends no accept is owed one wake fewer from then on, and the last connection waiting is not announced. That is the node's half of `issues/an-accept-that-never-reaches-netstack-strands-its-listener.md`. Exit: that issue's, by a client that cannot spend a wake without its accept arriving, or by an accept that waits in netd and needs no wake.
 - A stream its client can see no more, its reader gone or the peer's FIN read through and its client writing no more, is reset once its pipe has given up no byte for 100 s, and a peer address restarts that clock for at most 16 such streams at once (`OWNERLESS_PER_PEER`, `userland/netstack/node/src/streams.rs`); one past them has 100 s from the pass that found it so, until one of the 16 is done and it takes its place. What holds in this stage, from the code: per address, 16, an estimate no measurement set. There is no floor on the rate: each of the 16 is kept for as long as its pipe gives up one byte per 100 s, which for a full 2 MiB pipe is 2,097,152 x 100 s, over six years, and then for its tail in [tcp] (the line below on a client gone with nothing left in its pipe). The 16 is no bound across addresses: addresses are not counted, and where a connect's address is its client's choice, an accepted stream's is the peer's, which on the link answers for as many addresses as it likes. The bound across them is the node's places (`userland/netstack/node/src/places.rs`), of which every stream holds one until the node lets it go, and there is no second one: in `peers_at_more_addresses_than_there_are_places_hold_no_stream_past_them` (`userland/netstack/node/tests/listeners.rs`) peers at four addresses bring sixteen connections each to a node with three places, which holds two such streams at a time, each kept alive past its 100 s, and refuses every accept and connect past them. Exit: the T14 or a guest shows a peer holding one at a byte per 100 s, and the cut takes a floor on the rate, or the owner rules it needs none.
-- A stream its peer reset is gone from the node at once, so for a client that still holds it `Node::set_nodelay` answers `false`: `issues/a-stream-its-peer-reset-refuses-the-option-requests-a-host-answers.md`, carried by the node as built. std's `nodelay()` asks netd nothing: it answers from the value its `TcpStream` holds (`sdk/std/sys/net/connection.rs`), and libc's `getsockopt` of `TCP_NODELAY` from its socket's entry (`userland/libc/src/socket.rs`). What the node carries is `set_nodelay` on a reset stream, refused where macOS refuses it under another kind and Linux answers. The node cannot hold the stream for the client: its two pipe ends are all it has of the client, letting both go is how the client learns of the reset, and with both gone nothing tells it when the client leaves, so a kept stream would be kept for ever. Exit, outside the node: that issue's guest test is green, and a `getsockopt` of `TCP_NODELAY` on a stream its peer reset is answered, by libc from a value its socket holds as std's does or by a pipe ABI that gives netd a handle whose other end's leaving the kernel reports.
+- A stream its peer reset is gone from the node once its client's pipe has taken the text received before the reset, so for a client that still holds it `Node::set_nodelay` answers `false`: `issues/a-stream-its-peer-reset-refuses-the-option-requests-a-host-answers.md`, carried by the node as built. std's `nodelay()` asks netd nothing: it answers from the value its `TcpStream` holds (`sdk/std/sys/net/connection.rs`), and libc's `getsockopt` of `TCP_NODELAY` from its socket's entry (`userland/libc/src/socket.rs`). What the node carries is `set_nodelay` on a reset stream, refused where macOS refuses it under another kind and Linux answers. The node cannot hold the stream for the client: its two pipe ends are all it has of the client, letting both go is how the client learns of the reset, and with both gone nothing tells it when the client leaves, so a kept stream would be kept for ever. Exit, outside the node: that issue's guest test is green, and a `getsockopt` of `TCP_NODELAY` on a stream its peer reset is answered, by libc from a value its socket holds as std's does or by a pipe ABI that gives netd a handle whose other end's leaving the kernel reports.
 - How a stream ended reaches its client as the order its two pipes end in, which is one bit: the peer's FIN, or a failure. The node keeps a send pipe whose FIN is queued until its writer leaves and lets a failed stream's send pipe go first (`userland/netstack/node/src/streams.rs`), so std and libc read a FIN as a FIN and a failure as `ConnectionReset` or `ECONNRESET`, whichever failure it was: `issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md`. Exit: that issue's.
-- Every frame and every deadline ends in a pass over every stream, every transmit opportunity walks every stream twice (once for the count each peer address keeps alive, once to pass those whose connect is not answered), and `Node::next_deadline` reads every stream: linear in streams per frame, as netstack's bridge on smoltcp was. For an idle stream a pass is one `recv_with`, two `status` and one read of its client's pipe, which in netstack is a system call: 1,000 system calls a frame with 1,000 idle streams. A pass's `recv_with` also re-files its connection's deadline and offers it to the transmit round, so one frame makes every idle stream eligible and the next opportunity serves each to learn it has nothing. [tcp] cannot say which connections a segment, a timer, an ICMP error or a failed next hop moved: it reports `drain_eligible` and `drain_gone`, which are the round's. The change: [tcp] marks a held connection in `for_conn`, `tick`, `icmp` and `end` and hands the marks out once (`drain_moved`); the node maps a connection to its stream, keeps its deadlines ordered, and passes only the streams [tcp] moved, the one a pipe's wake names, and those due. A pipe's end-gone answer (`Node::pipe_gone`) runs a pass of its own as well, beside the one `Sockets::bridge` runs for a wake's readiness answers, so k clients leaving in one wake cost k + 1 passes. Not measured at the move. The count is what a guest can read and a time is not: netstack's places end at 103, what one poller watches (`MAX_PLACES`, `userland/netstack/src/main.rs`), so 1,000 idle streams cannot exist, the T14's bench opens no peer that holds streams, and QEMU gives no verdict on time. Exit: a guest test counts netstack's reads of its clients' pipes per frame received beside one bulk stream, with 1 idle stream and with 100, and the line closes when the two counts are equal.
+- Every batch of received frames and every deadline ends in a pass over every stream, every transmit opportunity walks every stream twice (once for the count each peer address keeps alive, once to pass those whose connect is not answered), and `Node::next_deadline` reads every stream: linear in streams per batch, as netstack's bridge on smoltcp was per frame. For an idle stream a pass is one `recv_with`, two `status` and one read of its client's pipe, which in netstack is a system call: 1,000 system calls a batch with 1,000 idle streams. A pass's `recv_with` also re-files its connection's deadline and offers it to the transmit round, so one batch makes every idle stream eligible and the next opportunity serves each to learn it has nothing. [tcp] cannot say which connections a segment, a timer, an ICMP error or a failed next hop moved: it reports `drain_eligible` and `drain_gone`, which are the round's. The change: [tcp] marks a held connection in `for_conn`, `tick`, `icmp` and `end` and hands the marks out once (`drain_moved`); the node maps a connection to its stream, keeps its deadlines ordered, and passes only the streams [tcp] moved, the one a pipe's wake names, and those due. A pipe's end-gone answer (`Node::pipe_gone`) runs a pass of its own as well, beside the one `Sockets::bridge` runs for a wake's readiness answers, so k clients leaving in one wake cost k + 1 passes. Not measured at the move. The count is what a guest can read and a time is not: netstack's places end at 103, what one poller watches (`MAX_PLACES`, `userland/netstack/src/main.rs`), so 1,000 idle streams cannot exist, the T14's bench opens no peer that holds streams, and QEMU gives no verdict on time. Exit: a guest test counts netstack's reads of its clients' pipes per batch received beside one bulk stream, with 1 idle stream and with 100, and the line closes when the two counts are equal.
 - A client that is gone with nothing left in its pipe is finished by [tcp]'s rules for a user who let go: a reset once it has been idle 60 s, where netd on smoltcp reset it 100 s after the client left whatever the peer said. A peer that acknowledges a byte a minute holds such a connection for as long as it has bytes to acknowledge. Exit: the T14 or a guest shows such a peer holding one, and [tcp]'s rule takes a bound from the client's leaving; or the owner rules the idle bound is the one.
 - `Tcp::ready`, `Tcp::orphans` and `Tcp::options` are additions to [tcp] the specifications do not hold, made for the node's wakes, its places and the copy a stream keeps of its connection's options; their tests (`toyos-net-shard/tcp/tests/held.rs`) name no scenario id. `Shard::listen` refuses an address [ip] does not hold usable, by the rule [udp]'s bind refuses by (`udp.bind-address-not-local`), and counts nothing for it; the node's test of it (`a_listener_is_at_the_address_it_named_and_only_one_the_machine_holds`) names no scenario either. A listener whose address the machine loses stands as a datagram socket bound to it does, with its port and its place and its owner told nothing, and answers again once the address is back (`a_listener_stands_while_its_address_is_lost_and_answers_when_it_is_back`, `userland/netstack/node/tests/lease.rs`); no host was read for it. Exit: the specifications hold the three calls and the refusal, and the tests name their scenarios.
 - The word each of [udp]'s refusals is answered in on the pipe ABI is `Refused` in `userland/netstack/node/src/datagram.rs`, tested by `each_refusal_is_answered_in_the_pipes_word_for_it` against no reader's scenario; `udp.no-ephemeral-port` has a word and no test that reaches it. Exit: the scenario owed below exists, and the test names its id.
@@ -73,6 +73,7 @@ What stage 4 departs from its specifications:
 - The readers' stage 4 design has a flow waiting for its next hop keep its place in the round. [shard]'s round lets it go, deficit and all, and [tcp] offers it again when woken, at the round's tail, so a waiting flow costs the round nothing. Exit: a `toyos-net-shard/tests/drr.rs` test finds a woken flow served in the place it held when it began to wait.
 - The readers' stage 4 design has a flow leave the round once it has nothing eligible, its deficit reset. [udp] says so as it hands out a sender's last datagram; [tcp] learns it only when it serves the connection, since only its `next_segment` decides what is due, so a connection whose turn ended on its last segment stays in the round with its deficit until its next turn, and what it queues before then is sent against that deficit, debt included, as RFC 8290 §4.2 keeps an emptied queue until it is next selected. Exit: [tcp] reports at hand-off that nothing is left, and a `toyos-net-shard/tests/drr.rs` test finds such a connection out of the round, its deficit reset.
 - `Tcp::recv_with`, the read of a reader that may take less than it is shown, is not in the specifications: the node's bridge to a client's pipe needs it, since a byte `recv` copied out has opened the window and cannot be put back. Its tests (`toyos-net-shard/tcp/tests/take.rs`) name no scenario. Exit: the specification names the call and its scenarios, and the tests take their ids.
+- RFC 9293 §3.10.7.4 says a reset in a synchronized state SHOULD flush every queue. [tcp] keeps the in-order text received before it, which `recv` and `recv_with` give ahead of `Err(Failed(Reset))` (`Ended`, `toyos-net-shard/tcp/src/stack.rs`), as Linux and the BSDs read buffered data ahead of `ECONNRESET`: the orchestrator ruled to follow them, since ToyOS's clients are written against those stacks, and since text a pass has already written to a client's pipe cannot be flushed, so only keeping it makes what a client reads independent of how the card batched frames. `s_rx_023_a_reset_comes_after_the_text_before_it` holds it. Exit: the specification names the departure, or the owner rules for the flush.
 - The readers' TCP specification disagrees with itself, and [tcp] follows its user timeout: one replaces R2 and runs only while sent data is unacknowledged or a zero window holds data back, so with one set a connection whose data never left, for want of credit or of a next hop, has no give-up, where its rule that give-up clocks run on wall time has a device that never offers credit end its connections. Exit: RFC 9293 §3.10.8 ends a connection in any state once its user timeout expires: a test that sets one and withholds credit, as `s_pl_009_give_up_without_credit` does without one, finds the connection ended.
 
 Open for the owner. On 2026-10-03 he chose "Fair scheduler now": "Build the byte-fair scheduler as the network spec describes, about 280 lines, and decide later on the laptop whether new connections should jump the queue." It is built as one round, in the order flows became eligible, and nobody jumps it. The track reads a queue that new connections jump as RFC 8290's new-flows list, and offers him this evidence: a sparse flow such as a DNS query waits one turn of every flow ahead of it in the round, up to a quantum of 1,514 bytes each, where [udp]'s datagrams and [tcp]'s segments alternated frame by frame before, so a query waited at most one TCP frame: in DRR-02 a 100-byte segment that becomes eligible leaves behind a full frame of each of the three bulk flows ahead of it (`s_shard_drr_002_a_bulk_flow_starves_no_other_beyond_its_quantum`); a T14 measurement at stage 5 of a sparse flow's wait behind bulk flows would add to it. Exit: the owner rules on whether new connections jump the queue.

toyos-net-shard/tcp/src/stack.rs

@@ -100,7 +100,8 @@ enum User {
 
 struct Ended {
     failure: Option<Failure>,
-    /// After both FINs: what the user has still to read.
+    /// After both FINs or the peer's reset: what the user has still to read, ahead of the failure
+    /// (Linux and the BSDs, not RFC 9293 §3.10.7.4's SHOULD-flush).
     rx: Option<Box<Rx>>,
 }
 
@@ -474,7 +475,7 @@ impl Tcp {
     }
 
     /// A connection's end: the user who holds it keeps the socket to learn why (and to read what
-    /// both FINs left); anyone else's is freed.
+    /// both FINs or a reset left); anyone else's is freed.
     fn end(&mut self, index: u32, failure: Option<Failure>, rx: Option<Box<Rx>>) {
         let Some(conn) = value(&mut self.conns, index) else { return };
         match conn.user {
@@ -887,7 +888,7 @@ impl Tcp {
                 self.settle(index, now);
             }
             Verdict::Closed => self.end(index, None, Some(Box::new(sync.rx))),
-            Verdict::Reset => self.end(index, Some(Failure::Reset), None),
+            Verdict::Reset => self.end(index, Some(Failure::Reset), Some(Box::new(sync.rx))),
             Verdict::TimeWait => {
                 let tw = TimeWait::from_sync(&sync, now);
                 let owed = sync.rx.ack_now || sync.rx.dup_owed > 0 || sync.rx.delayed.is_some();
@@ -1010,10 +1011,9 @@ impl Tcp {
         let result = match &mut conn.state {
             Tcb::SynSent(_) | Tcb::SynRcvd(_) => Err(Error::WouldBlock),
             Tcb::Sync(sync) => sync.recv(out, now),
-            Tcb::Ended(Ended { failure: Some(failure), .. }) => Err(Error::Failed(*failure)),
-            Tcb::Ended(Ended { failure: None, rx }) => match rx.as_mut().map(|rx| rx.read(out)) {
+            Tcb::Ended(Ended { failure, rx }) => match rx.as_mut().map(|rx| rx.read(out)) {
                 Some(n) if n > 0 => Ok(Received::Data(n)),
-                _ => Ok(Received::End),
+                _ => failure.map_or(Ok(Received::End), |f| Err(Error::Failed(f))),
             },
         };
         self.settle(id.index, now);
@@ -1029,10 +1029,9 @@ impl Tcp {
         let result = match &mut conn.state {
             Tcb::SynSent(_) | Tcb::SynRcvd(_) => Err(Error::WouldBlock),
             Tcb::Sync(sync) => sync.recv_with(take, now),
-            Tcb::Ended(Ended { failure: Some(failure), .. }) => Err(Error::Failed(*failure)),
-            Tcb::Ended(Ended { failure: None, rx }) => match rx.as_mut() {
+            Tcb::Ended(Ended { failure, rx }) => match rx.as_mut() {
                 Some(rx) if rx.unread() > 0 => Ok(Received::Data(rx.read_with(take))),
-                _ => Ok(Received::End),
+                _ => failure.map_or(Ok(Received::End), |f| Err(Error::Failed(f))),
             },
         };
         self.settle(id.index, now);

toyos-net-shard/tcp/tests/acceptability.rs

@@ -75,7 +75,8 @@ fn s_ac_004_zero_window_pure_ack() {
 fn s_ac_005_zero_window_rst() {
     let mut h = zero_window();
     nothing(&h.input(10, seg(70_536).rst()));
-    reset(&mut h);
+    let status = h.status();
+    assert_eq!((status.state, status.failure), (State::Closed, Some(Failure::Reset)));
 }
 
 #[test]

toyos-net-shard/tcp/tests/receive.rs

@@ -230,10 +230,11 @@ fn s_rx_022_no_sack_unnegotiated() {
 }
 
 #[test]
-fn s_rx_023_a_reset_is_never_end_of_stream() {
+fn s_rx_023_a_reset_comes_after_the_text_before_it() {
     let mut h = fixture_e();
     h.input(1, seg(5001).ack(1001).len(100));
     h.input(2, seg(5101).rst());
+    assert_eq!(recv(&mut h, 1000), Ok(Received::Data(100)));
     assert_eq!(recv(&mut h, 1000), Err(Error::Failed(Failure::Reset)));
 }