diff --git a/issues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md b/issues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md new file mode 100644 index 0000000000..3412e6ad96 --- /dev/null +++ b/issues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md @@ -0,0 +1,28 @@ +--- +status: open +kind: defect +opened: 2026-10-11 +--- + +# A pipe is a 2 MiB page for a mapping nothing makes + +A pipe's ring is one `pmm::alloc_page` page, `PIPE_SIZE = PAGE_2M` +(`kernel/src/pipe.rs`), allocated and zeroed on its first write, because +`SYS_PIPE_MAP` maps that page into a process and a 2 MiB page is the only one +this kernel maps. Nothing outside the guest tests calls `SYS_PIPE_MAP`: its +last caller, netd, mapped its clients' rings only to read the closed flags, +and stopped in a23734684 when they were found forgeable. Since then the map +costs every pipe and every IPC connection, which is two pipes, 2 MiB a +direction once a byte moves, whatever the traffic: a wake bell holds 2 MiB to +carry one byte, and so does each direction of a connection that carries one +request. It also keeps `Pipe::publish_ends` writing reader and writer flags +into the page for a reader that no longer exists, and the ring's header +contract over a page a process may write (`sdk/abi/src/ring.rs`). + +Owner: the orchestrator, who stages +`issues/netstack-moves-every-byte-through-two-kernel-pipes.md`, whose streams +ring their doorbells through pipes and pay a page for each until this closes. + +Exit: `rg -n -i 'pipe_?map|publish_ends|RING_(WRITER|READER)_CLOSED' kernel +sdk tests` is empty, and a pipe's ring is no `pmm::alloc_page` page: +`rg -n 'alloc_page|PAGE_2M' kernel/src/pipe.rs` is empty. diff --git a/issues/a-stream-cannot-move-between-processes.md b/issues/a-stream-cannot-move-between-processes.md new file mode 100644 index 0000000000..ecce1a6e0a --- /dev/null +++ b/issues/a-stream-cannot-move-between-processes.md @@ -0,0 +1,49 @@ +--- +status: open +kind: defect +opened: 2026-10-12 +--- + +# A stream cannot move between processes + +Today a TCP stream's handle reads and writes the stream in any process that +holds it. The handle is the stream's two data pipes joined into one +connection (`make_socket_fd`, `sdk/std/sys/net/connection.rs`), which +`TcpStream` reads and writes with `syscall::read` and `write`. +`File::from_raw_fd` takes any handle (`sdk/std/os/io.rs`) and reads and writes +it with the same calls (`sdk/std/sys/fs.rs`), and `Stdio::from(File)` hands +it to a child, whose slot the kernel fills with a duplicate +(`sdk/std/sys/process.rs`; `build_child_handles`, +`kernel/src/loader/start.rs`). So an inetd-style child serves the stream on +its stdin and stdout for as long as the process that connected it holds its +`TcpStream`. + +The session design takes that away +(`issues/netstack-moves-every-byte-through-two-kernel-pipes.md`, stage 3; PR +#863, amendment E). After its answer, a stream's connection carries only +doorbells, and the region that holds the stream's bytes is mapped only by +the process that connected or accepted it. Read through a `File` or a +child's stdio, the handle would wait for ever or read doorbells as data, and +netstack would drain what the child writes as doorbells, with no error on +either side. Stage 3 makes std refuse both loudly instead. +`File::from_raw_fd` of a stream's handle panics with `toyos::net`'s refusal. +Every path std offers to move a raw handle into another process answers that +refusal before anything moves: a spawn's stdio slots, +`CommandExt::inherit_handle` into any slot, `CommandExt::endow` and +`Command::provide`, each starting no child. `toyos::ipc`'s `send_handles` +takes only `OwnedHandle`s, and a stream's handle becomes one only through +`OwnedHandle::from_raw`, whose contract stage 3 says excludes a handle +`toyos::net` answers for: the type refuses it. And `toyos::net` refuses by +name a handle the process holds no end for, std's `TcpStream::from_raw_fd` +included. A program that hands a stream to another process by any of these, +as an inetd-style server does through its child's stdio, then fails loudly +on ToyOS where today it works, and one that takes a moved stream back as a +`TcpStream`, as socket activation does and as no ToyOS std builds today, is +refused, though root `CLAUDE.md` says existing Rust just works. The bare +`dup`, `SYS_HANDLE_SEND` and `SYS_SPAWN` syscalls are not checked. + +Owner: the orchestrator, who stages +`issues/netstack-moves-every-byte-through-two-kernel-pipes.md`. + +Exit: a stream moved by its handle reads and writes in the receiving process, +and the process that moved it holds no end. diff --git a/issues/netstack-moves-every-byte-through-two-kernel-pipes.md b/issues/netstack-moves-every-byte-through-two-kernel-pipes.md new file mode 100644 index 0000000000..8714da2b72 --- /dev/null +++ b/issues/netstack-moves-every-byte-through-two-kernel-pipes.md @@ -0,0 +1,203 @@ +--- +status: open +kind: track +opened: 2026-10-11 +--- + +# netstack moves every byte through two kernel pipes + +A received byte is copied three times between the card's buffer and the +application's: netstack writes it into its connection's heap ring +(`system/netstack/shard/tcp/src/ring.rs`), then into the client's pipe with a +`SYS_WRITE` whose copy runs under the one global `PIPES` lock with preemption +off (`kernel/src/pipe.rs`, `issues/the-global-pipe-lock-spans-a-user-copy.md`), +and the client's `SYS_READ` copies it out under the same lock. A sent byte is +copied into the pipe, out of it into netstack's stack chunk, into the +connection's send ring, and into the frame. Both buffers hold the same bytes: a +stream holds its pipes' 2 MiB pages beside its own 4 MiB rings +(`PLACE_BYTES`, `system/netstack/src/main.rs`). + +Evidence, PR #861's round 3 on the T14, stage B at K = 32, medians of six runs +per MB of a 170 MB HTTPS download: the pipe table's lock held 0.281 ms and +spun for 0.047 ms (`pipe_lock_hold_cycles`, `pipe_lock_spin_cycles`, TSC +time), netstack's ring write took 0.225 ms (`cy_ring_write`), against 5.447 ms +of C0. The kernel copied 2.00 bytes of pipe per byte downloaded +(`pipe_copy_bytes`). Stage 0's scout split the hold on one T14 boot: the two +copies are 73% of it, and of the lock the session design removes at most +0.192 ms per MB on the receive path, beside the pipe syscalls' work outside +the hold, which no row reads apart +(https://github.com/ToyOSOrg/ToyOS/pull/863#issuecomment-6114364774). + +What a pipe also cannot say is carried by workarounds this track deletes: how +a stream ended is the order its two pipes close in, one bit +(`issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md`); +a stream's client leaving is learned only through those pipes +(`OWNERLESS_LIFE`, `system/netstack/node/src/streams.rs`); a datagram costs a +pipe write and a request on a fresh connection. + +The owner ruled the replacement a go, in his words "pipes aren't meant for a +lot of data transfer, especially not network", and ruled zero copy from the +NIC into an application's memory out unless the design showed a useful +benefit. The design dropped it under that condition, on its own reading: the +one copy it would remove is 4% of C0, against a filter and a queue per flow, +header split, an IOMMU window per client and a reassembly that copies. He +accepted the design on the scout's reading with one constraint on every +stage: the data path is userland's, and the kernel gains only what it alone +can answer (https://github.com/ToyOSOrg/ToyOS/pull/863#issuecomment-6114483274). +The design and its roast are PR #863's body. + +- Stage 0, done: the scout boot on the T14 that splits the lock's hold into + its two copies (PR #863). +- Stage 1, done (PR #867): the kernel's one new answer is `SYS_FSTAT`'s + `DeviceMemory`, and a receiver judges a lent region through `fstat` before + it maps it. +- Stage 2, done (PR #878): a byte ring with untrusted cursors and want words + in `toyos-transport` (`lib/blockring/transport`), host-tested and modelled + in loom. +- Stage 3: a TCP stream is one region and one connection: netstack's receive + and send buffers are the region's two rings, beside [tcp]'s heap storage, + which stays for stage 4's passive children, and the connection carries the + connect and its answer, then only doorbells and the hang-up. A stream + belongs to the process that connected or accepted it: `toyos::net` refuses + by name a handle it holds no end for. netstack rings a reader on the + connection and a writer waiting for send room on a pipe of the stream's + own, so neither drains a wake rung for the other. Within a direction no + thread holds the end across its wait, and a thread that took the doorbell + while another waits on it passes the take on through netstack. A + registration arms the want of each direction it names and rings netstack if + that direction is already ready, and netstack rings back. A fatal `end`, + any but a FIN, makes the send direction ready to every look, as POSIX's + `select` counts a descriptor ready for writing when a write "would not + block, whether or not the function would transfer data successfully", and a + `write` reads it before it copies, answering its error and copying nothing, + as RFC 9293 flushes every segment queue at a reset or a user timeout + (§3.10.7.4, §3.10.8) and answers a SEND that finds no connection an error + (§3.10.2). std, libc and the mio and socket2 forks read and write the rings + through `toyos::net`. std gains `FromRawFd`, `IntoRawFd` and the `OwnedFd` + conversions for `TcpStream`, which take only the process's own ends, and + the `rust/` fork's ToyOS arms of `OwnedFd`'s drop and + `BorrowedFd::try_clone_to_owned` let go of and enter a stream's handle + through `toyos::net`. Today a stream's handle reads + and writes the stream through a `File` or a child's stdio; over the region + it would read doorbells, so `File::from_raw_fd` of a stream's handle panics + with `toyos::net`'s refusal, and every path std offers to move a raw + handle into another process asks `toyos::net` of each handle before + anything moves and answers its refusal: a spawn's stdio slots, + `CommandExt::inherit_handle` into any slot, `CommandExt::endow` and + `Command::provide`, each with no child started and the handle still the + caller's, so no refused move leaves an entry that outlives its handle's + number. `toyos::ipc`'s `send_handles` is refused by its type: it takes + `OwnedHandle`s, and a stream's handle becomes one only through + `OwnedHandle::from_raw`, whose `# Safety` stage 3 extends to exclude a + handle `toyos::net` answers for. The bare `dup`, `SYS_HANDLE_SEND` and + `SYS_SPAWN` syscalls stay unchecked. A regression, + `issues/a-stream-cannot-move-between-processes.md`. Exit: three loom models + that drive the client end and netstack's ringing code as stage 3 lands + them, each red under a control that reverts its decision. In one, a + stream's reader and writer, each asleep on its own want while netstack's + thread releases room and publishes data in either order, end with neither + waiting, and the control rings both wants on one doorbell. In another, a + blocking call and a `poll` of one direction, the poll's watch answered by + the doorbell's level at its look as the kernel's `deliver` answers it and + the blocking call leaving bytes, the receive ring's `end` or room behind, + end with the poll answered, and the control does not pass the take on. In + the third, a source registered while its direction is ready, in both + directions, with the receive ring's `end`, and with a fatal `end` while the + send ring is full, made ready before the registration or racing it, gets + its first event; it is red without the registration's arm and ring, red + with either of the two removed, red under stage 2's `want-relaxed`, and red + with the fatal `end` where a look reads only the send ring's head. A test: + `toyos::net`, handed a stream handle the process holds no end for, refuses + it by name and writes nothing on its connection, and a stream's own handle, + given out by `into_raw_fd` or `From for OwnedFd` or made by + `try_clone_to_owned`, and taken back, shares the one end and its cursors, + red when giving the handle out lets its entry go. A test: + `File::from_raw_fd` of a stream's handle panics with that refusal, red when + it takes the handle silently. A test per moving path, each handed a + stream's handle: a spawn's stdio slot, `inherit_handle` into a slot past + the stdio slots, `endow` and `provide` each answer the refusal and start no + child, and the stream then reads and writes through that handle; each is + red when its path takes the handle silently, and `endow`'s and `provide`'s + are red when the handle moves and its entry stays. A test: a `write` with + room after a reset answers `ConnectionReset` and copies nothing, red when + `write` copies before it reads the `end`. And a T14 boot of round 3's + `internet_download` on stage 3's head reads, in every run, + `pipe_copy_bytes` under 0.01 per byte downloaded, its C0, hold and spin + per MB posted beside the scout's rows. No tree that lands has that row or + that counter: the boot carries the scout's instrument, `git diff + 777c96b2b 2f9e090b8` (`wt/toyos-netring-scout` over its base), onto stage + 3's head, less the netstack stamps on the pipe writes stage 3 deletes. +- Stage 4: a listener and a datagram socket the same way, and the pipe ABI's + messages, `SYS_CONNECTION_JOIN` and `toyos::net`'s pipe handles go. A + passive child's bytes wait in [tcp]'s heap storage, at most + `limits::RECEIVE_BUFFER_INITIAL` a child and `limits::LISTEN_READY` children + a listener, and its accept copies them into the region before it answers. An + accept takes one of the wakes netstack writes on the listener's connection + for each child that waits before it makes a region, so a nonblocking accept + with none answers `WouldBlock` having made nothing, and the accept's answer + carries the peer's address and the `nodelay` the child inherited from its + listener. netstack copies each datagram record's `len`, `port` and `addr` + out of the region once and judges the copy: a `len` past the bytes the + record holds ends that socket alone. `toyos::net`'s `send_to` refuses + what a sender can judge and writes no record: a payload past + `toyos_abi::net::UDP_MAX_PAYLOAD`, as POSIX's `sendto` fails + `[EMSGSIZE]`; port 0; an unspecified address. That bound is the one + declaration: [udp]'s `limits::MAX_PAYLOAD` goes, and [udp], the node's + `name.rs` and `resolve.rs`, netstack's `serve.rs` and [udp]'s + `tests/send.rs` read it. [udp] still refuses each by its counter at the + trust boundary. What only netstack can judge, a connected socket's other + destination, a route, broadcast permission, a source address and a pending + ICMP error, netstack stores in the region's `pending` word if it is 0, + then exchanges both client wants for 0 and rings each that was set, as it + does for a stream's fatal `end`; a call's look after it arms reads + `pending` beside its ring, and the socket's next call answers it and + clears it, as POSIX's pending error is "returned in response to a + subsequent send(), recv(), or getsockopt() operation" (XSH 2.10.10). + netstack takes a record off the send ring only while [udp]'s queue for + the socket has room, and looks again when a datagram leaves it or is + dropped, so a full queue holds the ring and refuses nothing. A + `send_to` over a full send ring waits as a stream's writer does, on a + bell of the socket's own that netstack makes at the bind and alone + writes, its want `tx_room`, as POSIX's `sendto` "shall block until space + is available or a timeout occurs"; nonblocking, which std's + `UdpSocket::set_nonblocking` then sets, it answers `WouldBlock`. Exit: + over the region, + `system/netstack/node/tests/listeners.rs`'s + `a_listener_answers_a_syn_and_wakes_its_owner_when_the_handshake_ends` + passes, and so does a case in which a child holds, before its accept, text + in order, text past a hole and its FIN behind that, the hole is filled after + the accept, and the region reads all of it in order, then its `end`. That + case is red under each of three controls that drop at the accept one thing + the child held: its in-order bytes, which also reds the first, its + out-of-order range, its FIN. And a datagram record whose `len` is past the + bytes it holds is refused while netstack serves on, red when netstack takes + that `len` as written. A case per refusal: an oversized payload, port 0 + and an unspecified address each answer their error at `send_to` with no + record written, and each refusal only netstack judges answers on the + socket's next call; each is red when that call answers `Ok`. A case: a + nonblocking `send_to` over a full send ring answers `WouldBlock` and + writes no record, red when `set_nonblocking` sets nothing. A case: the + socket's queue is full to a next hop that never answers and a `send_to` + sleeps over a full send ring; when [ip] gives the hop up and [udp]'s + `Udp::fail` drops the queue, netstack takes the held record and the sender + wakes, red when netstack looks again only after a datagram leaves. Two loom + models drive the client end and netstack's code as stage 4 lands them. In + one, a `recv_from` asleep with nothing to receive, and a `send_to` asleep + over a full send ring, each answers a refusal only netstack judges that + lands meanwhile; each is red when the look reads `pending` before it arms, + red when netstack exchanges the wants before it stores `pending`, and red + under `want-relaxed`, and the sender's is red when netstack rings only + `rx_data`. In the other, [udp]'s queue starts full and a datagram leaves + it as its own step, interleaved with netstack's looks, so netstack can find + a published record while the queue is full and hold it; a sender that + fills the send ring, beside a `recv_from` asleep on the connection, has + every record sent, in order, none refused. It is red when netstack refuses + a record at the full queue, red when it rings `tx_room` on the connection, + red when after the leave it waits for the client's next ring without + looking at the send ring again, and red under `want-relaxed`. + +Owner: the orchestrator, who stages it. + +Exit: stage 3's and stage 4's exits, and the pipe ABI gone with stage 4: +`rg -n 'Piped|DATA_HANDLES|NOTIFY_HANDLES|connection_join|CONNECTION_JOIN' sdk +system kernel` and `rg -n 'pipe_pair' sdk/toyos/src/net.rs` are empty.