Repository navigation
Conversation
The session-region design that replaces the pipes between netstack and its clients, and its roast, are this pull request's body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
…e for a mapping nothing makes The track records the copies and the lock as a present-state weakness, with round 3's T14 rows as evidence and the stages the design in PR #863's body names. The defect is the pipe-section finding of that design: SYS_PIPE_MAP has had no caller outside the guest tests since a237346, and it is why every pipe and every connection direction costs a 2 MiB page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
The netring scout patch: applies with diff --git a/kernel/src/counters.rs b/kernel/src/counters.rs
index df5f87d3b..9d2144048 100644
--- a/kernel/src/counters.rs
+++ b/kernel/src/counters.rs
@@ -133,7 +133,10 @@ fn sample(me: usize) -> [u64; WORDS] {
| Counter::InboxRearms
| Counter::PipeLockSpinCycles
| Counter::PipeLockHoldCycles
- | Counter::PipeCopyBytes => Some(soft::get(me, counter)),
+ | Counter::PipeCopyBytes
+ | Counter::PipeLockTakes
+ | Counter::PipeReadCopyCycles
+ | Counter::PipeWriteCopyCycles => Some(soft::get(me, counter)),
};
if let Some(value) = value {
words[1] |= 1 << counter as usize;
diff --git a/kernel/src/pipe.rs b/kernel/src/pipe.rs
index 6bd53207e..e46d1e9af 100644
--- a/kernel/src/pipe.rs
+++ b/kernel/src/pipe.rs
@@ -184,6 +184,7 @@ fn with_pipes<R>(f: impl FnOnce(&IdMap<PipeId, Pipe>) -> R) -> R {
let guard = PIPES.lock();
let t1 = soft::now();
soft::add(Counter::PipeLockSpinCycles, t1.wrapping_sub(t0));
+ soft::add(Counter::PipeLockTakes, 1);
let r = f(guard.as_ref().expect("pipes not initialized"));
soft::since(Counter::PipeLockHoldCycles, t1);
r
@@ -196,6 +197,7 @@ fn with_pipes_mut<R>(f: impl FnOnce(&mut IdMap<PipeId, Pipe>) -> R) -> R {
let mut guard = PIPES.lock();
let t1 = soft::now();
soft::add(Counter::PipeLockSpinCycles, t1.wrapping_sub(t0));
+ soft::add(Counter::PipeLockTakes, 1);
let r = f(guard.as_mut().expect("pipes not initialized"));
soft::since(Counter::PipeLockHoldCycles, t1);
r
@@ -235,7 +237,12 @@ pub fn try_read(pipe_id: PipeId, buf: &mut UserBytesMut) -> Option<usize> {
.as_mut()
.expect("available() > 0 implies a ring")
.ring
- .read(len, |off, src| buf.write_run(off, &src));
+ .read(len, |off, src| {
+ let t = crate::counters::soft::now();
+ let r = buf.write_run(off, &src);
+ crate::counters::soft::since(toyos_abi::counters::Counter::PipeReadCopyCycles, t);
+ r
+ });
crate::counters::soft::add(toyos_abi::counters::Counter::PipeCopyBytes, n as u64);
let boost = pipe.rt_boost_pending;
pipe.rt_boost_pending = false;
@@ -270,7 +277,12 @@ pub fn try_write(pipe_id: PipeId, buf: &UserBytes) -> Option<PipeWrite> {
return Some(PipeWrite::NoMemory);
};
if backing.ring.space() > 0 {
- let n = backing.ring.write(buf.len(), |off, mut dst| buf.read_run(off, &mut dst));
+ let n = backing.ring.write(buf.len(), |off, mut dst| {
+ let t = crate::counters::soft::now();
+ let r = buf.read_run(off, &mut dst);
+ crate::counters::soft::since(toyos_abi::counters::Counter::PipeWriteCopyCycles, t);
+ r
+ });
crate::counters::soft::add(toyos_abi::counters::Counter::PipeCopyBytes, n as u64);
Some(PipeWrite::Wrote(n))
} else {
diff --git a/sdk/abi/src/counters.rs b/sdk/abi/src/counters.rs
index 090907631..686d189bf 100644
--- a/sdk/abi/src/counters.rs
+++ b/sdk/abi/src/counters.rs
@@ -73,10 +73,16 @@ pub enum Counter {
PipeLockHoldCycles,
/// Bytes pipe reads and writes copied.
PipeCopyBytes,
+ /// Takes of the pipe table's lock.
+ PipeLockTakes,
+ /// Stamp ticks of a pipe read's copy out of the ring, under the lock.
+ PipeReadCopyCycles,
+ /// Stamp ticks of a pipe write's copy into the ring, under the lock.
+ PipeWriteCopyCycles,
}
impl Counter {
- pub const COUNT: usize = 21;
+ pub const COUNT: usize = 24;
pub const ALL: [Counter; Self::COUNT] = [
Self::Stamp,
Self::Smi,
@@ -99,6 +105,9 @@ impl Counter {
Self::PipeLockSpinCycles,
Self::PipeLockHoldCycles,
Self::PipeCopyBytes,
+ Self::PipeLockTakes,
+ Self::PipeReadCopyCycles,
+ Self::PipeWriteCopyCycles,
];
pub const fn name(self) -> &'static str {
@@ -124,6 +133,9 @@ impl Counter {
Self::PipeLockSpinCycles => "pipe_lock_spin_cycles",
Self::PipeLockHoldCycles => "pipe_lock_hold_cycles",
Self::PipeCopyBytes => "pipe_copy_bytes",
+ Self::PipeLockTakes => "pipe_lock_takes",
+ Self::PipeReadCopyCycles => "pipe_read_copy_cycles",
+ Self::PipeWriteCopyCycles => "pipe_write_copy_cycles",
}
}
@@ -147,7 +159,10 @@ impl Counter {
| Self::InboxRearms
| Self::PipeLockSpinCycles
| Self::PipeLockHoldCycles
- | Self::PipeCopyBytes => Rights::COUNTERS.union(Rights::TRACE),
+ | Self::PipeCopyBytes
+ | Self::PipeLockTakes
+ | Self::PipeReadCopyCycles
+ | Self::PipeWriteCopyCycles => Rights::COUNTERS.union(Rights::TRACE),
}
}
}
diff --git a/tests/guest/rust-tests/src/bin/https_download.rs b/tests/guest/rust-tests/src/bin/https_download.rs
index b1885732a..1dd8f7d61 100644
--- a/tests/guest/rust-tests/src/bin/https_download.rs
+++ b/tests/guest/rust-tests/src/bin/https_download.rs
@@ -58,7 +58,7 @@ const PROBES: usize = 5;
/// cadence's `K` netstack is told before it: a palindrome, so drift across the
/// boot weighs every `K` alike. A netstack with no cadence refuses the word, and
/// the label is then the run's place in the order.
-const K_ORDER: [u32; 6] = [8, 16, 32, 32, 16, 8];
+const K_ORDER: [u32; 6] = [32; 6];
/// MEASUREMENT ONLY: the word that sets netstack's `K`: `K=` and the value.
fn set_k(k: u32) { |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Scout image build, |
The scout boot, judged: S = 0.192 ms per MB, so the rule says the build may claim a CPU savingThe boot. One T14 boot of The judge. The instrument is sound
The rows, per MB downloaded (MB = 10⁶ bytes), TSC time at the counter's 2,419,361,856 HzM is read from a counter. D is derived from M rows, and the derivation is named.
Arithmetic. Each M cell is counter ÷ 2,419,361,856 × 10³ ÷ 170.439044. For example, run 1's R is 48,835,466 ÷ 2,419,361,856 × 10³ ÷ 170.439044 = 0.1184 ms. D is computed per run first, and then the median is taken. From the medians:
The decision, by the rule stated before the boot
What the rule does not seeThese points do not change the rule's verdict. They bound what the build's pull request can claim.
The script every number above came from (python3 -I scout.py judge.log, EXIT=0)import re, sys, statistics as st
log = open(sys.argv[1]).read().splitlines()
TSC = None
runs = []
dl = {}
for l in log:
if '[download] perf idle' in l:
TSC = int(re.search(r'tsc_hz=(\d+)', l).group(1))
m = re.search(r'\[download\] perf run=(\d+) k=32 (.*)', l)
if m:
kv = dict(w.split('=',1) for w in m.group(2).split())
runs.append((int(m.group(1)), kv))
m = re.search(r'\[download\] whole .* mbps=([\d.]+) .* run=(\d+)', l)
if m: dl[int(m.group(2))] = float(m.group(1))
c0 = {}
cur = None
for l in log:
m = re.match(r'netperf: run (\d+) per MB', l)
if m: cur = int(m.group(1))
m = re.search(r'M system C0 \(sum mperf / tsc_hz\)\s+([\d.]+) ms', l)
if m and cur: c0[cur] = float(m.group(1))
m = re.search(r'M netstack cy_ring_write \(tsc\)\s+([\d.]+) ms', l)
if m and cur: c0.setdefault(('rw',cur), float(m.group(1)))
rows = {k: [] for k in 'R W H P S X takes copyB C0 ringw mbps hold_per_take R+W<=H'.split()}
for n, kv in runs:
B = int(kv['bytes']); MB = B/1e6
ms = lambda c: int(kv[c])/TSC*1e3/MB
R=ms('k.pipe_read_copy_cycles'); W=ms('k.pipe_write_copy_cycles'); H=ms('k.pipe_lock_hold_cycles'); P=ms('k.pipe_lock_spin_cycles')
takes=int(kv['k.pipe_lock_takes'])/MB; cb=int(kv['k.pipe_copy_bytes'])/B
ok = int(kv['k.pipe_read_copy_cycles'])+int(kv['k.pipe_write_copy_cycles']) <= int(kv['k.pipe_lock_hold_cycles'])
hpt = int(kv['k.pipe_lock_hold_cycles'])/int(kv['k.pipe_lock_takes'])
vals = dict(R=R,W=W,H=H,P=P,S=H-R+P,X=H-R-W,takes=takes,copyB=cb,C0=c0[n],ringw=c0[('rw',n)],mbps=dl[n],hold_per_take=hpt)
for k,v in vals.items(): rows[k].append(v)
rows['R+W<=H'].append(ok)
for k,v in rows.items():
if k=='R+W<=H': print(k, 'all' if all(v) else 'NOT all'); continue
print(f"median {k}: {st.median(v):.4f} sorted {[round(x,4) for x in sorted(v)]}")The worktree and the readback are left as they were. Nothing in the PR's code was changed. |
|
The scout judge, |
|
The scout judge log, masked, part 2 of 2 (lines 389-722 of 722) |
|
The scout flash, |
|
Design acceptance, recorded by the orchestrator for the owner. The owner accepted the session-region design after the scout's result (#863 (comment)). His answer to "Build it?", verbatim: "i agree but explain why an abi change shouldnt that be completely userland? isnt this why we have userland drivers?" His question sets the constraint for every stage: the data path is userland's. A kernel change lands only where the kernel is the sole authority able to answer, as stage 1's two answers are: what a received handle is, and a lent region's length and whether it is kernel-allocated RAM. A stage that adds anything else to the kernel says why userland cannot do it. Stages build in the order the body gives, each its own pull request; stage 1 first. |
main moved by issues and tests/common/qemu.rs alone since 06788fc; nothing under kernel, sdk, system, lib or userland that the design cites changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…design as amended Stage 0 is done: the scout's reading on #863 (S = 0.192 ms per MB, the copies 73% of the hold) joins the evidence. The owner's acceptance and its constraint, that the kernel gains only what it alone can answer, is the design line every stage keeps (#863 (comment)). Stage 1 is #867, narrowed by the orchestrator's ruling there (#867 (comment)): the kernel's one new answer is fstat's DeviceMemory, SYS_HANDLE_RECV answers no kind and SYS_SHM_MAP no length, and a receiver judges a lent region through fstat before it maps it. It closes the lent-window issue alone; issues/a-received-handle-has-no-knowable-type.md stays open, narrowed by #867. Stage 2 is #878. The exit read pipe_copy_bytes "per MB downloaded", a unit the row does not print: the row is per byte downloaded, 2.00 today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
|
Review of #863 at Net lines ( Checked
BLOCKER
NOTE
SEND BACK |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…for send room, and the pipe page has an owner Review #863 (comment). The owner ruled zero copy out unless the design showed a useful benefit; the track said he dropped it. It now says the design dropped it under his condition, on its own reading of a 4% gain (BLOCKER 1). Stage 3 has its own exit (BLOCKER 5): a loom model of one stream's reader and writer asleep at once, and the T14 boot of round 3's download. That boot reads pipe_copy_bytes, which no landing tree has: round 3's b245592 added it, and the boot carries the scout's measurement-only commits onto stage 3's head (BLOCKER 4). The track's own exit reads stage 3's and the pipe ABI gone, `pipe_pair` in toyos::net included, so a datagram socket cannot keep its pipes under another name (NOTE 3). Stage 3 rings a writer waiting for send room on a pipe of the stream's own, the amendment PR #863's body puts to the orchestrator: one netstack-to-client doorbell for both wants lets a reader drain the writer's wake, and a TcpStream split across two threads hangs (BLOCKER 6). A scratch loom model over #878's stream ends at 7ea0ff9 deadlocks with one doorbell and passes with two. The scout's S bounds the lock's share the design removes, not the receive path's saving: the pipe syscalls' work outside the hold goes too. The pipe-page defect has an owner, the orchestrator who stages the track that pays its pages (BLOCKER 2), and an exit two searches read (BLOCKER 3): the used-memory guest test it named reads a machine-wide figure every demand fault and heap growth moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0 Run 2026-10-12 10:08:49Z to 10:12:22Z, on wt/toyos-netring with main at 528b9a6c2 merged.
… run -- --ci host, EXIT=0
Triage of an external automated review (ChatGPT, supplied by the owner)Read against #863 at Counts: holds 3, already raised 2, does not hold 4, can't tell 0. Classification
BLOCKER
NOTE None. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0; amendment D's loom model and amendment C's pre-accept mutation, measurement only
Amendments C, D and E, proposed for the orchestrator's rulingAt e7e1179, Amendment C, proposed: a passive child's bytes wait in netstack until its acceptFor the orchestrator's ruling; not in the track until it. The defect (the triage's BLOCKER 1). Read again at e7e1179, it holds.
Published practice. Two shared-memory stacks answer this two ways.
The amendment: mTCP's answer.
Why this and not TAS's.
Why not the others.
What it costs.
What would have caught it: the node tests above. The control drops what a child held at its accept, which is what the design as written does to those bytes.
The runner and its log, whole: run.sh and pre-accept.log. What acceptance adds to the track.
Amendment D, proposed: a mio registration arms its wantsFor the orchestrator's ruling; not in the track until it. The defect (the triage's BLOCKER 2). Read again at e7e1179, it holds. The readiness bullet says "This is exactly mio's contract, since mio reads until
The amendment: a registration arms as a waiter does, and rings netstack if the direction is already ready.
This is epoll's Why this and not the others.
What it costs.
What a model reads, before anything is built. A scratch loom model, measurement only, never landing.
The model's source, manifest, lockfile and runner, and its log whole: registered_ready.rs, Cargo.toml, Cargo.lock, run.sh and registered-ready.log; evidence What acceptance adds to the track. Stage 3's exit gets a third loom model. It drives the client end's registration and netstack's ringing code, as stage 3 lands them. A source registered while its direction is ready gets its first event:
Its controls skip the registration's arm and, separately, its ring; each goes red. Amendment E, proposed: the connection carries one request, then only doorbellsFor the orchestrator's ruling; not in the track until it. The defect (the triage's BLOCKER 3). Read again at e7e1179, it holds.
Published practice. A doorbell carries no request, and control has a channel of its own.
The amendment:
What it takes away, and what it does not. Nothing in the tree moves a stream between processes now:
So E takes away nothing the tree has. It refuses the one path the design added. A program that hands a socket to another process (socket activation, an inetd-style child) cannot do so on ToyOS before E or after it. That gap is not filed today. If E is accepted, it is filed with E: a Why this and not the others.
What it costs. Nothing on any path the tree has. What would have caught it. A reading of three facts:
What acceptance adds to the track. Stage 3's exit gets a test: |
|
Amendments C, D and E accepted, by the orchestrator (#863 (comment)). Each follows published practice, per #885's rule.
Next, the record takes them in: the track's stage exits and the body's summary, with the full text in the linked comment, because of the body's size limit. Then it is reviewed again. |
…E's exits The orchestrator accepted amendments C, D and E to #863's design (#863 (comment)), as proposed in #863 (comment). - C: a passive child's bytes wait in [tcp]'s heap storage until its accept, which copies them into the region, as mTCP does. Stage 3 keeps that storage beside the region; stage 4's exit carries the two listeners.rs cases that pin those bytes, red when the accept drops them. - D: a registration arms the want of each direction it names and rings netstack if the direction is already ready, as epoll's ep_insert reports a ready source. Stage 3's exit carries a third loom model, red without D, red with the arm or the ring removed, and red under want-relaxed. - E: MsgType::Region goes; a connection carries the connect and its answer, then only doorbells, and a stream belongs to the process that connected or accepted it. Stage 3's exit carries a test of toyos::net's refusal of a handle it holds no end for. The track's exit now names stage 4's exit beside stage 3's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0 The log whole, the home directory masked as ~.
…ull request, and a reviewer triages what he returns The owner reviews design pull requests himself with ChatGPT's free chat, by hand: automating the chat platform is against its terms, so it is never scripted. He gave the orchestrator leave to ask him: "you can ask me if i can let chatgpt review a design pr". That is leave, not a duty: it names no pull request and no moment to ask. On #863 a single-design review found three design defects that three review rounds had missed, while a one-pass review of all open pull requests found almost nothing new. orchestrator.md's Review and land gains one sentence: the orchestrator may ask the owner whether he will have ChatGPT review a design pull request, and a reviewer triages whatever he returns against the code before anything acts on it. The leave to ask is the owner's ruling; the triage clause is the orchestrator's own design and not part of it. What backs it is #863's triage of his ChatGPT review there: of nine claims, three held, two had already been raised and four did not hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
|
Review of #863 at Net lines ( Earlier BLOCKERs (round 3 had none; these are the triage's, 6126387631)
Checked
BLOCKER
NOTE
SEND BACK |
|
The orchestrator, on round 4 (#863 (comment)):
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…a fatal end readies the send direction, and a stream cannot move between processes Round 4's review of #863 (issuecomment-6129829193) and the orchestrator's rulings on it (issuecomment-6129844256): - Stage 4's exit drops `an_accept_answers_the_option_of_a_stream_its_own_pass_let_go`, which reds only through `ToClient`'s `WriteRefusal::Broken`, both on stage 4's deletion list. In its place goes amendment C's own case: a child holds, before its accept, text in order, text past a hole and its FIN behind that; the hole is filled after the accept, and the region reads all of it in order, then its `end`. It is red under each of three controls dropping at the accept one thing the child held. Run against today's node tests, each control reds the case: the in-order bytes and the out-of-order range as zeros where text was, the FIN as an end that never comes. - A fatal `end`, any but a FIN, makes the send direction ready to every look, as Linux v6.12's `tcp_poll` answers a reset socket `EPOLLOUT` once `tcp_done` has set `SHUTDOWN_MASK`. Without it a writer or a `WRITABLE` registration that arms after a reset while the send ring is full looks only at the head and sleeps on a bell nobody rings again. Stage 3's third model carries the case and a control whose looks read only the head. - Stage 3 says std gains `FromRawFd`, `IntoRawFd` and the `OwnedFd` conversions for `TcpStream`, taking only the process's own ends, and that `OwnedFd`'s drop lets a stream's handle go through `toyos::net`, so no entry outlives its handle's number. - `issues/a-stream-cannot-move-between-processes.md` files the gap amendment E leaves, with its owner and E's exit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Triage of a second external automated review (ChatGPT, supplied by the owner)Read against #863 at Read: the body and the track at that head; amendments A and B in the body; C, D and E as proposed in 6127271821 and accepted in 6127307681; rounds 1 to 4 (6121859611, 6122446433, 6125978857, 6129829193); the first triage (6126387631); and the orchestrator's ruling on round 4 (6129844256). Also: the models at evidence Counts: holds 4, already raised 1, does not hold 2, stale 0. Classification
Three more parts of the review make no finding:
BLOCKER
NOTE
|
|
The orchestrator's rulings on the second triage's findings (#863 (comment)). Each follows Linux, per #885's rule.
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Amendments A and B in fullAs PR #863's body carried them from their acceptance through round 4, at 148d265, with round 4's ruling 2 added to amendment A's third bullet. The body now carries a summary of each and links here, as it does for C, D and E, to stay under GitHub's 65,536 characters. Amendment A, accepted: a doorbell per client sleeperAccepted by the orchestrator: #863 (comment). The defect (#863 (comment), BLOCKER 6). As accepted, netstack rang both of the client's wants, The amendment: each want a client sleeps on has a doorbell of its own.
No byte rung for one direction is then readable by the other direction's sleeper: a reader, a Why this and not the others.
What it costs. No kernel change: the bell is a pipe, as What a model reads, before anything is built. A scratch loom model, measurement only, never landing: #878's
The last two show the model reaches its sleeps: the handshake's own control reds it. The model's source, manifest and runner, and its log whole: two_sleepers.rs, Cargo.toml, Cargo.lock, run.sh and two-sleepers.log, evidence The property stage 3 must test. A loom model of one stream's reader and writer, each asleep on its own want while netstack's thread releases room and publishes data in either order, ends with neither waiting; its negative control rings both wants on one doorbell and goes red. The model above is its shape. The track's stage 3 exit names it, driving the client end and netstack's ringing code that stage 3 lands, not this scratch model. Amendment B, accepted: a take is passed onAccepted by the orchestrator: #863 (comment). The defect (#863 (comment), BLOCKER 1). Under the rule as written, within a direction the end's holder drains the doorbell before it looks, and its blocking wait takes the byte; a
A level-triggered The amendment: any number of threads wait on a direction, and a take is passed on.
Why it holds. The count is a plain field under the end's mutex. Every waiter takes that mutex to drain, arm and look, and every taker holds it when it reads the count. So either the taker finds the waiter counted and passes the take on, or the waiter's drain, arm and look come after the taker let the end go, and see whatever the taken byte announced. If the passer's look finds the direction ready, netstack rings the doorbell again, and a thread that takes that byte in turn passes it on while the waiter is counted. If the passer's look finds it empty, the waiter sleeps on the want the passer set, which the next publish clears and rings by #878's handshake. The amendment adds no memory order. Why this and not the others.
What it costs. No kernel change, and no pipe, handle or kernel watch past amendment A's.
What a model reads, before anything is built. A scratch loom model, measurement only, never landing.
The model's source, manifest, lockfile and runner, and its log whole: same_direction.rs, Cargo.toml, Cargo.lock, run.sh and same-direction.log; the unbounded run's unbounded.sh and unbounded.log; evidence The property stage 3 must test. A blocking call and a |
… region only for a child, and netstack judges each datagram record The orchestrator's rulings (issuecomment-6130141310) on the second triage of an external review of #863 (issuecomment-6130134066), each following Linux: - Stage 3: a `write` reads the receive ring's `end` before it copies, and a fatal one is its error with nothing copied, as v6.12's `tcp_sendmsg_locked` checks `sk_err` first. The record as it stood copied whenever the send ring had room, so after a reset every write answered `Ok` until 4 MiB less 4 KiB were queued for nobody. Stage 3's exit gains a test of it with its control. - Stage 4: an accept takes a wake from the listener's connection before it makes a region, so a nonblocking accept with nothing ready answers `WouldBlock` having made nothing, as `inet_csk_accept` answers `EAGAIN` from an empty queue; the answer carries the peer's address and the `nodelay` the child inherited from its listener. - Stage 4: netstack copies each datagram record's `len`, `port` and `addr` out of the region once and judges the copy, where today `udp_send_to` refuses a `len` naming bytes never written; its exit gains a datagram case with a control for each bound on `len`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0; amendment D's model with a reset over a full send ring and the head-only control (toyos-transport as at 386c34101); stage 4's pre-accept case and its three controls against today's node tests, measurement only
|
Review of #863 at Net lines ( Earlier BLOCKERs
Checked
BLOCKER
NOTE
SEND BACK |
|
Ruling on E, after round 5's BLOCKER 1 (#863 (comment)). The review shows a cost the record did not price. Today a stream's handle reaches a E stands, on these terms:
The fix round also answers BLOCKERs 2 and 3 as the review writes them, and makes NOTEs 1-3's corrections in the body. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…dly, the E test gains its boundary, and the track cites RFC 9293 and POSIX where it cited Linux Round 5's review (PR #863, 6130914125) and the orchestrator's ruling on E (6130929883): - The stream-move issue states E's cost as the regression it is. Today a stream's handle is its joined data pipes, and `File::from_raw_fd` and `Stdio::from(File)` read and write it with the same `syscall::read` and `write` as `TcpStream`; over the region that handle is the doorbell connection. Stage 3 makes `File::from_raw_fd` of a stream's handle panic with `toyos::net`'s refusal and a spawn that would put one in a child's stdio slot answer it, and its exit gains a test that reds when either takes the handle silently. The three claims the review quoted as false are gone. - Stage 3's E test gains its boundary (BLOCKER 3): a stream's own handle given out by `into_raw_fd` or `From<TcpStream> for OwnedFd`, or made by `try_clone_to_owned`, and taken back, shares the one end, red when giving it out lets the entry go. The track names the `rust/` fork's ToyOS arms of `OwnedFd`'s drop and `BorrowedFd::try_clone_to_owned`, which stage 3 routes through `toyos::net` (NOTE 1, NOTE 3). - Stage 4 judges a datagram record's `len` only against the bytes the record holds, and its `MAX_PAYLOAD` case goes (BLOCKER 2): [udp]'s `submit` already refuses that payload by `Counter::ExceedsMtu`, so the case could not go red under its control. The owner's direction, relayed by the orchestrator this round: ToyOS does not follow Linux and references it nowhere in the tree. The track's two Linux citations become POSIX's `select` ("would not block, whether or not the function would transfer data successfully") for a fatal end making the send direction ready, and RFC 9293 §3.10.7.4, §3.10.8 and §3.10.2 for a write after a fatal end copying nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
… run -- --ci host, EXIT=0
|
Rulings on round 6's two open calls.
|
What this lands
A design round, built on nothing: three issue files and this body, which is the design as accepted and amended, and its roast. No source, test or manifest changes.
issues/netstack-moves-every-byte-through-two-kernel-pipes.md(kind: track): the pipe copies and the lock as a present-state weakness, with round 3's T14 rows and the scout's reading as evidence, the owner's constraint, the owner, the stages with their pull requests, 0 to 2 done, the exits of stages 3 and 4, and the track's exit, which two searches and those exits read.issues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md(kind: defect): the one thing the pipe section below found to fix, with its owner and an exit two searches read. Not fixed here.issues/a-stream-cannot-move-between-processes.md(kind: defect): the regression amendment E brings, a stream's handle that reads and writes the stream through aFileor a child's stdio today and is refused loudly after stage 3, with its owner and E's exit.Amendments A to E, and the rulings on rounds 4 and 5 and on the second triage, were each accepted by the orchestrator, in the comments the next section links; the bullets each changed say "(amendment A)" to "(amendment E)", or "(ruling 1)" to "(ruling 7)".
Stages 1 and 2 landed before this branch: #878 as 4ca63a6 and #867 as 867ec23. This body describes both as they stand on
main. Stages 3 and 4 are not opened.The owner's rulings, as the orchestrator relayed them
He also asked whether the pipe design itself has a problem: its own section, from the code.
What was amended since, and where it was ruled
Amendment A: a writer waiting for send room has a doorbell of its own. Accepted by the orchestrator (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment)); summarized below, in full in netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment).
Amendment B: within a direction no end is held across a wait, and a take is passed on. Accepted by the orchestrator (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment)); summarized below, in full there.
Amendments C, D and E: a passive child's bytes wait in netstack until its accept; a registration arms its wants; a stream stays in the process that opened it. Accepted by the orchestrator (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment)); proposed in full in netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment), and summarized in their section below.
Rulings 1 to 7, on round 4's review (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment)), the second triage (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment)) and round 5's review (netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment)). Ruled by the orchestrator in netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment), netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment) and netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863 (comment). The first six were ruled following Linux; under the owner's direction 5 above, each now rests on the RFC or POSIX text it stands for, or on its own terms:
an_accept_answers_the_option_of_a_stream_its_own_pass_let_go;endmakes the send direction ready to every look;writeafter a fatalendanswers its error and copies nothing;len,portandaddr;nodelay;File::from_raw_fdand in a spawn's stdio, with a case in stage 3's exit that reds when either takes it silently.Round 4's notes also filed the stream-move gap, corrected the mio sentence, and had the record say what stage 3 adds to std for a stream's raw handle: the handle bullet. Round 5 let its panic on a foreign handle stand, gave the E test its boundary (BLOCKER 3), dropped netstack's own
MAX_PAYLOADjudgement (BLOCKER 2) and priced therust/fork (NOTE 3).Stage 1 (SYS_FSTAT answers whether a region is memory the kernel allocated, and a lent region is judged by it before it is mapped #867, landed as 867ec23): the kernel's only new answer is
fstat'sDeviceMemory. Ruled by the orchestrator under the owner's constraint (SYS_FSTAT answers whether a region is memory the kernel allocated, and a lent region is judged by it before it is mapped #867 (comment)). This design first hadSYS_HANDLE_RECVanswer each handle's kind andSYS_SHM_MAPa mapping's length and memory kind. Neither stands:fstatresolves a handle of any kind, answers a region's size, and answersSharedMemoryonly for RAM.Stage 2 (toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878, landed as 4ca63a6): cursors count modulo twice the ring, and the sleeper sets its want by exchange. Both accepted by toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878's review (toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878 (comment), "The two departures"). This design first said "the sleeper stores its want"; a store followed by a look at the cursor is the store-buffering shape and loses wakes, which toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878's mutation m1 reds in both loom wake models.
The starting facts, each read again at 06788fc
By 1acaa36, the
mainthis branch merges, stage 2 gavetoyos-transportits byte ring, and #861 (ab655a1) alone moved the other paths they cite:git diff --stat 06788fc53 1acaa362f -- system/netstack kernel/src/pipe.rs kernel/src/sync.rs kernel/src/inbox kernel/src/syscall/ipc.rs kernel/src/pcidev sdk/std sdk/libc sdk/toyos/src/net.rs sdk/toyos/src/wake.rs issues/toyos-has-its-own-network-stack.md issues/the-global-pipe-lock-spans-a-user-copy.mdnames netstack's watches, its receive budget, the I219's ring and a pipe end's leaving watch. Each fact below was read again at 97f07c3, which differs from 1acaa36 only in.claude/agents/, and holds.Ring::write_atcopies into aVec<u8>that grows by doubling (system/netstack/shard/tcp/src/ring.rs). Round 3 times it ascy_ring_write.PIPESlock with preemption off. True, and wider:Node::passhandstcp_recv's bytes toToClient::write, which isPipe::write_nonblock(system/netstack/src/pipes.rs);pipe::try_writeandtry_readrunRing::write/readwith the user copy insidewith_pipes_mut(kernel/src/pipe.rs), andLock::lockdisables preemption (kernel/src/sync.rs). The send direction is worse: the client's bytes go into a pipe, out into netstack'sCHUNKstack buffer (streams.rs), into the send ring bytcp_send, and into the frame, whichmain.rscopies again into the card's slot.issues/toyos-has-its-own-network-stack.md. The nearest isissues/the-global-pipe-lock-spans-a-user-copy.md, which records the lock and not the copies. The new track records them.lib/blockring/transporthas nothing block-specific in it. True:toyos-transportis entry queues with clamped peer cursors (queue.rs), an arena of slot runs (arena.rs) and a tag table (inflight.rs), all overWords an adapter hands in,forbid(unsafe_code), loom-modelled. It had no byte ring at 06788fc. A TCP stream needs one, the one addition below, and stage 2 (toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878) landed it asstream.rs.system/netstack/i219/): one receive ring, no queue, split or offload register is written. What the part itself has is not read here; the zero-copy section does not rest on it.The session-region design
A TCP stream is one shared region and one IPC connection. The client makes both: the region with
SYS_SHM_CREATE, the connection by connecting tonetstack. The connect request carries the region's handle. The connection carries that request and its answer, then nothing but one-byte doorbells each way and its own hang-up (amendment E), and netstack rings a writer waiting for send room on a bell of the stream's own (amendment A). A stream belongs to the process that connected or accepted it (amendment E).The region
8 MiB, four 2 MiB pages, physically contiguous as every
SYS_SHM_CREATEregion is today:The header's words: the receive ring's
tail(netstack) andhead(client); the send ring'stail(client) andhead(netstack: what the peer acknowledged); four want words, one per wait, each set by exchange by the side that will sleep and cleared by exchange by the side that rings (rx_dataandtx_roomset by the client,rx_roomandtx_dataset by netstack); the receive ring'send(netstack, written once, after the last tail:0open, then FIN, reset, timed out, unreachable, prohibited); the send ring'send(client: shut down for writing); andnodelay(client).The rings are the TCP buffers, not a copy of them, for a passive child from its accept on (amendment C). [tcp]'s
Ringtoday stores into aVec; it becomes a ring over the region's words through the same adaptertoyos-transportuses, sowrite_atplaces an out-of-order segment's payload at its offset in the receive ring andcommitis the published tail. The window is the receive ring's room. On the send sidetcp_send's copy goes: the bytes the client published are what [tcp] segments from, and the head moves only as the peer acknowledges. One copy of every received byte is gone (heap ring to pipe), and two of every sent byte (pipe to chunk, chunk to send ring).A passive child's bytes wait in netstack until its accept (amendment C). Its SYN-ACK offers a window before any region exists, so what arrives first stays in [tcp]'s heap storage: at most 65,535 bytes a child, its first window, and
LISTEN_READY, 128, children a listener. The accept copies them into the region's receive ring at their offsets, with any out-of-order range and a FIN, frees that storage, and only then answers, as mTCP keeps a child's payload untilmtcp_accept. An active open's region exists before its SYN.The addition to
toyos-transport(stage 2, #878, landed): a single-producer byte ring,stream.rs, besidequeue.rs.Release, load the peer's withAcquireonly when asked for room or what is ready, and hand the adapterSpans to copy through.u32under a mask would be right only for a power of two, and the send ring is not one. The peer's cursor is believed only as a position at most the ring away from this end's: a word at or past the lap names no position, and either isViolation::TailPastDepth/HeadPastTail, asqueue.rsnames them.Acquire), then reads the peer's cursor again, and sleeps only if that look finds nothing. The waker publishes its cursor, then exchanges the want for 0 (Release), and rings only if it was set. Either the sleeper's exchange reads the waker's clear and sees the cursor stored before it, or the clear follows and finds the want set. A sleeper drains its doorbell before it looks, never between arming and sleeping.Nothing in it is network-specific: a third byte stream between two processes would take it as it is.
Why the kernel changes at all
The owner's question. The data path is userland's: the stream's bytes, cursors, wants and end words live in a region two processes share, and the kernel neither copies nor reads one of them. The kernel's part is what it already does: make and map the region (
SYS_SHM_CREATE,SYS_SHM_MAP), and carry the doorbells and the hang-up on the connection and on the bell, both pipes.Most of what changes is not the kernel's ABI. The netstack message set below is a userland protocol between netstack and
toyos::net, as any server's is.The kernel's syscall ABI changes twice, and the first has landed:
SYS_FSTATanswersDeviceMemoryfor a region that is not memory the kernel allocated, where before it answered every regionSharedMemory(fstat'sSharedMemarm,kernel/src/object/ops.rs). Only the kernel knows what memory backs a region. Without the answer, a client holding a device's BAR or the GOP framebuffer could lend it to netstack as its stream, and netstack would write received bytes into device memory.SYS_CONNECTION_JOIN, whose only caller outside the abuse tests is std'smake_socket_fd.The protocol, and what the kernel answers about a lent region
MsgType::TcpConnect { addr, port, timeout_ms }with one handle, the region; answeredTcpConnectResponse { local_port }with one handle, the stream's bell (amendment A), or an error, on the same connection, which is then the stream's.MsgType::TcpListen { addr, port, nodelay }on a connection that is then the listener's, on which netstack writes one wake for each child that waits, aslisteners.rs'sWakeis written today. An accept takes a wake first (ruling 5): a nonblocking one with none answersWouldBlock, and a blocking one waits on the connection for one, so no region or connection is made until a child is there. POSIX'sacceptgives the same two answers to an empty queue: it "shall block until a connection is present", or, withO_NONBLOCKset, "shall fail and set errno to [EAGAIN] or [EWOULDBLOCK]". That nothing is made first is the design's own: a tokio accept loop callsacceptuntilWouldBlockon every event, and would make 8 MiB of contiguous memory per event to hear it, or, on a fragmented machine,ResourceExhaustedin its place. A process makes a connection to netstack only by connecting, so the accept is thenMsgType::TcpAccept { listener }on a new connection with a fresh region, answered once the child's bytes are in it (amendment C) with the stream's bell andTcpAcceptResponse { remote_addr, remote_port, local_port, nodelay }(ruling 6): the peer, and thenodelaythe child inherited from its listener, which netstack has also written into the region's header before it answers. POSIX leaves "implementation-defined which socket options, if any, on the accepted socket will have a default value determined by a value previously customized by setsockopt() on socket"; the child takesnodelay, as it does today throughTcpAcceptPipedResponse'soptions(sdk/toyos/src/net.rs), since a program that set it on its listener set it for the streams it accepts. An accept that took a wake and failed before netstack had its request rings the listener's connection, and netstack writes the wake again, which isissues/an-accept-that-never-reaches-netstack-strands-its-listener.md's exit. The listener id stays the numberissues/netstack-socket-ids-are-ambient.mdrecords.MsgType::UdpBind { addr, port }with a 2 MiB region whose two rings carry records, a record beinglen,port,addrand the payload padded to a word: a send is a record and at most a doorbell, where today it is a pipe write and a request on a new connection. netstack drops and counts a received datagram its client's ring has no room for, which is UDP's answer, where today a datagram that does not fit ends the socket. netstack judges each record's words (ruling 4) as the hostile-client section says.TcpShutdown,TcpSetOption,TcpClose,UdpSendTo,UdpRecvFrom,UdpCloseand the three*Pipedmessages go: shutdown andnodelayare header words, close is the hang-up, a datagram is a record.SYS_SHM_MAPof a handle that is no region isHandleError::WrongType, which ends the caller, andSharedMemory::adoptmaps what it is handed and believes the size it is told, so it is only for a region the kernel itself handed the process (sdk/toyos/src/shm.rs).SYS_FSTATresolves a handle of any kind and needs no right: only a region of memory the kernel allocated answersSharedMemory, and it answers the region's size. netstack takes a lent region throughSharedMemory::lent, which refuses by name a handle that is no region, a region that is not memory the kernel allocated, and one that is not exactly 8 MiB (2 MiB for a datagram socket), each before anything is mapped; netstack closes the connection on the refusal.SYS_HANDLE_RECVandSYS_SHM_MAPare unchanged. fileserver and diskserver take their lent regions throughSharedMemory::lentalready.issues/a-received-handle-has-no-knowable-type.mdstays open for the kindsfstatdoes not settle, none of which this design has netstack use.std,
poll, libc and mioThe session's client end is
toyos::net::Streaminsdk/toyos/src/net.rs, one implementation that std (sdk/std/sys/net/connection.rs), libc (sdk/libc/src/socket.rs) and the mio and socket2 forks call, as they all calltoyos::netnow.as_raw_fdanswers it, and the hang-up comes once no process holds a handle of it.toyos::netkeeps the process's ends by handle. std'stry_cloneand libc'sdupmake a handle of the same connection and enter it against the end they duplicate, asTcpStream::duplicateshares itsArcs today (sdk/std/sys/net/connection.rs); so doesBorrowedFd::try_clone_to_owned, whichOwnedFd::try_clonecalls, once stage 3 routes its ToyOS arm, the bareduptoday (therust/fork'sos/fd/owned.rs:159-165), throughtoyos::net. Stage 3 adds std'sFromRawFd,IntoRawFd,From<OwnedFd>andFrom<TcpStream> for OwnedFdforTcpStream, which therust/fork'sos/fdcompiles out fortoyostoday (mod.rs:16-17,owned.rs:378-418), so a program that turns a stream into its handle and back builds unchanged. Taken back, a handle the process holds an end for shares that end. A handle the process holds no end for, one moved in from another process or duplicated pasttoyos::netby the baredupsyscall, is no stream in it:toyos::netrefuses it by name and writes nothing on its connection, so two processes never share a region (amendment E), and std'sfrom_raw_fd, which cannot fail, panics with that refusal (issues/a-stream-cannot-move-between-processes.md). A handle given out stays entered against its end, andOwnedFd's ToyOSdrop, which closes by the raw syscall today (owned.rs:241-244), lets a handle go throughtoyos::netastry_clone_to_owned's arm enters one, so no entry outlives its handle's number. The end goes with the process's last handle of the stream.File::from_raw_fdtakes any handle (sdk/std/os/io.rs) and reads and writes it with the samesyscall::readandwriteasTcpStream(sdk/std/sys/fs.rs), andStdio::from(File)hands it to a child (sdk/std/sys/process.rs), so a stream's handle serves the stream through either. Over the region it is the doorbell connection. SoFile::from_raw_fd, the one way a raw handle becomes aFileon ToyOS (From<OwnedFd> for Fileis compiled out,owned.rs:358-359), askstoyos::netwhether the handle is entered against a stream end, and panics with the refusal if it is, asTcpStream::from_raw_fdpanics on a foreign handle. A spawn asks the same of each handle it would put in a child's stdio slot, byStdio::inherit()orCommandExt::inherit_handle, and answers the refusal with no child started. NoFilethen holds a stream's handle, andFile's drop, a raw close (sys/fs.rs), never lets one go pasttoyos::net.readcopies from the receive ring into the caller's buffer, a userland copy of the bytes the kernel copies today. Empty: it drains the connection's doorbell, setsrx_databy exchange, reads the tail again, and only then lets its end go and waits on the connection.writefirst reads the receive ring'sendwithAcquire: a fatal one, any but a FIN, is its error, and it copies nothing (ruling 3). RFC 9293 has a reset in a synchronized state flush "All segment queues" and delete the TCB (§3.10.7.4), as a user timeout does (§3.10.8), and answers a SEND that finds no TCB "error: connection does not exist" (§3.10.2); POSIX'ssendfails[ECONNRESET], "A connection was forcibly closed by a peer". Otherwise it copies into the send ring and rings iftx_datawas set; full, it drains the stream's bell, setstx_roomby exchange, reads the head again, and only then lets its end go and waits on the bell (amendment A). Each direction's end is behind its own mutex, held to drain, arm, look and copy and never across a wait, so any number of threads, through any of the process's handles of the stream, may wait on one direction, each counted on its one end, and a call under a timeout waits on nothing but its doorbell and its deadline (amendment B).WouldBlock, andWouldBlockis answered only after the want is set and the cursor read again, so the wake that follows reaches the doorbell that direction's sleeper waits on. mio also waits for an event after it registers, before any I/O, as tokio'sconnect_mioandpoll_iodo, and would wait for ever on a direction already ready whose registration reported nothing. Soregisterandreregistertake each named direction's end, arm its want, look, let the end go, and ring netstack if the direction is ready, and netstack rings back by amendment B's rule; a registration drains nothing and counts nothing (amendment D). The mio fork registers the connection for itsREADABLEinterest and the bell for itsWRITABLEinterest, each watched for reading. libc'spollof a socket asks the rings first, so a level-triggeredpollon a ring that holds bytes answers at once; for a direction that is not ready it takes that direction's end as any waiter does, counts itself in, drains, arms and looks again, lets the end go, and waits on the connection forPOLLINand on the bell forPOLLOUT. A doorbell is drained only under its direction's end, before the drainer arms and looks, and a thread that took a doorbell while another waits on its direction passes the take on (amendment B). The mio fork counts nothing: a source is read toWouldBlock, which leaves the want armed, so the next publish rings its watch. A program that polls the raw handle withtoyos::pollerand does not read toWouldBlocksees edges, not levels: the module header says so.endword set, read withAcquireafter the tail: a FIN isOk(0), a resetConnectionReset, a timeoutTimedOut, an ICMP failure its own word. What rests on pipe ordering today, that the send pipe closes before the receive pipe so a client probing an empty write learns reset from FIN (TcpStream::ended,Node::pass), is deleted, and a failure says which failure it was (issues/a-streams-failure-reaches-its-client-as-a-reset-whatever-it-was.md).shutdown(Write)stores the send ring'sendafter its last tail and rings; netstack sends the FIN once it has segmented to that tail. A fatalendmakes the send direction ready to every look (ruling 2), awrite's, a registration's, apoll's and netstack's ring back. Awritethen answers the error without waiting, and POSIX'sselectcounts a descriptor "ready for writing when a call to an output function with O_NONBLOCK clear would not block, whether or not the function would transfer data successfully". A writer or aWRITABLEregistration that arms after a reset over a full send ring does not sleep. A FIN is not fatal, so half-close keeps the send direction.OTHER_END_GONE. Today the node cannot learn it once the peer's FIN has been read through (theOWNERLESS_LIFEparagraph instreams.rs). The peer-side bounds on an ownerless stream stay [tcp]'s; the half that exists only because a pipe cannot say its reader left goes.What it deletes
src/pipes.rs; in the node,ToClient,FromClient,ReadRefusal,WriteRefusal,Pipes,PipeEnd,pipe_gone,pipe_broken, the end-ordering of the two ends, theCHUNKread, thedrainedbookkeeping that therx_room/tx_datawants replace, and the listener'sWakepipe; inserve.rs,data_pipes, the datagram socket's pipes and its request per datagram, and two watches a stream (issues/netstack-spends-two-poller-slots-per-piped-connection.md,issues/netstack-watches-both-pipes-of-every-connection-on-every-pass.md). [tcp]'sVecstorage underRingstays, for a passive child before its accept alone (amendment C).toyos::net: the*Pipedmessages,DATA_HANDLES,NOTIFY_HANDLESand the pipe ends in every answer type.make_socket_fd,TcpStream::ended's empty-write probe,UdpSocket's two pipe handles.SYS_CONNECTION_JOINandConnectionEnd::joined.PLACE_BYTESfalls from 12 MiB (two 2 MiB pipes and two 4 MiB buffers) to the 8 MiB region, plus its doorbells' three pipe pages untilissues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.mdcloses; until then it is 14 MiB, where the bell's page is allocated by a stream whose writer once waited for room (amendment A) or that is registered forWRITABLE(amendment D).What a hostile client can do to a region it shares with netstack
netstack reads four kinds of word the client writes. A datagram record's
len,portandaddrare copied out of the region once and judged on the copy (ruling 4): alenpast the bytes the record holds is aViolationthat ends that socket alone, asudp_send_torefuses alennaming bytes never written today (system/netstack/src/serve.rs), and notoyos::netwrites one. That is the one bound netstack's serving code judges: the payload, port and address go to [udp], whosesubmitrefuses a payload pastlimits::MAX_PAYLOAD(Counter::ExceedsMtu), port 0 and an address it cannot send to, each by its counter, as it does today (system/netstack/udp/src/lib.rs;s_udp_us_023_payload_sizes,each_refusal_is_answered_in_the_pipes_word_for_it). A program names any of these throughsend_to, so each refuses that datagram, and the socket serves on. The send ring's tail and the receive ring's head are each believed only as a position at most the ring away, and otherwise are aViolationthat resets that stream alone. The want,endandnodelaywords are each a bit whose worst value costs netstack one doorbell or one pass (any want but 0 is set). The send ring's bytes netstack copies into a frame before it checksums them, so a client that rewrites a byte after publishing it, or between a send and its retransmission, corrupts only its own stream. netstack never reads back a byte it wrote into the receive ring, so out-of-order payload the client can see and overwrite before it is committed is the client's own. A doorbell storm is bounded by the pass: doorbells coalesce, and a stream rung with nothing new published costs one look, and a ring back for each want the client set on a direction that is ready (amendment B). A client that unmaps or closes its handle takes nothing from netstack, which holds its own handle and mapping until it lets the stream go.Amendments A and B, accepted
Each is set out in full, with the alternatives it weighed and refused and its model, in #863 (comment), as this body carried it through round 4.
Amendment A: a writer waiting for send room has a doorbell of its own. Accepted by the orchestrator: #863 (comment).
rx_dataandtx_roomon the connection's one netstack→client pipe, and a pipe's byte goes to one reader: the reader could take the byte rung fortx_room, and the writer sleep on with room in its send ring.rx_datarings the connection;tx_roomrings the stream's bell, a pipe netstack makes at connect and accept and holds the write end of alone (amendment E). When netstack writes the receive ring'send, it exchanges both client wants for 0 and rings each one that was set, so a writer asleep on a stream that ended wakes and reads why; one that arms after a fatalendfinds it at its look (ruling 2). The bell's write end gone reads as the stream gone, as the connection's hang-up does. netstack's own wants,rx_roomandtx_data, share the connection's client→netstack pipe, since its one thread is their only sleeper. A reader, aREADABLEregistration and aPOLLINwait read only the connection; a writer,WRITABLEandPOLLOUTonly the bell.PIPESentry more per stream, and the bell's page, allocated on its first ring: once the stream's writer finds its send ring full, or once it is registered forWRITABLE(amendment D). Untilissues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.mdcloses,PLACE_BYTESis 14 MiB where it is 12 today.want-relaxed, EXIT=101 (the model, its log).Amendment B: within a direction no end is held across a wait, and a take is passed on. Accepted by the orchestrator: #863 (comment).
pollof the same direction watched it. The kernel answers that watch by the pipe's level when its submitter looks after the post, and arms it again if the pipe is empty (kernel/src/inbox/polls.rs;lookinkernel/src/inbox/mod.rs;read_posts_are_readinessis false for a pipe and a connection), so a level-triggeredpollslept over a readable socket.pollwhose interest names the direction, counts itself in under the end before it drains, arms and looks, and out when it returns. A thread that takes the direction's doorbell, by a wait or a drain, and leaves the end with another waiter counted sets the want by exchange again and looks, and rings netstack if the direction is ready (bytes or theendin the receive ring, room or a fatalendfor the send ring). netstack, rung by a client, exchanges each client want whose direction is ready for 0 and rings the one that was set; stage 3 adds this exchange to each ofstream.rs's ends.pollof one direction, the blocking call leaving a byte, theendor room, at most five preemptions as the kernel's own models bound theirs. The rule as written, B less the pass and B underwant-relaxedare each red in all three cases, EXIT=101; B passes all three, EXIT=0 (the model, its log).Amendments C, D and E, accepted
Accepted by the orchestrator: #863 (comment). Each is set out in full, with the alternatives it weighed and refused, in the comment that proposed it: #863 (comment). They answer the triage's three BLOCKERs (#863 (comment)), each read again at e7e1179, and each follows published practice, per #885's rule.
Amendment C: a passive child's bytes wait in netstack until its accept.
stack.rs:418), and the ACK that ends the handshake runs the segment's text as ESTABLISHED (stack.rs:852-864). As first written, the design made the region only at the accept and deleted [tcp]'s heap storage, so that text had nowhere to go.mtcp_accept(tcp_in.c:617-630, api.c:582), and RFC 9293 §3.8.6, under which the SYN-ACK's window "indicates the range of sequence numbers the sender of the window (the data receiver) is currently prepared to accept": the design's paragraph of that name.listeners.rs'sa_listener_answers_a_syn_and_wakes_its_owner_when_the_handshake_ends(listeners.rs:455) is green at e7e1179 and red, EXIT=101, with the child's bytes dropped at its accept, as the design first written dropped them (pre-accept.log). Ruling 1's case, a child holding text in order, text past a hole and its FIN behind that before its accept, the hole filled after it, ran as a scratch test against today's node tests at 04ff95b: green, EXIT=0, and red, EXIT=101, under each control dropping one of the three at the accept. The in-order bytes read[0, 0, 108, 108, 111]and red the first case too, the out-of-order range[104, 101, 108, 0, 0], and the FIN leaves an end never reached (pre-accept.log).Amendment D: a registration arms its wants.
stream.rs:73-74), as tokio'sconnect_mioand first I/O do (tokiostream.rs:155,registration.rs:169). As first written no registration armed a want, soTcpStream::connectand sshserver's first I/O would wait forever.WRITABLE.end, room, and a reset over a full send ring (ruling 2), each made ready before the registration or racing it (registered-ready.log). The design as first written is red in all eight cases, EXIT=101; D passes all eight, EXIT=0; D less the arm and D less the ring are each red in all eight;want-relaxedreds D in the four racing cases;head-only, whose looks read only the send ring's head as the record stood at 148d265, reds D in the two reset cases alone. Each of the 30 reds is loom'sdeadlock, both threads blocked.want-relaxed, and, with a fatalend, red where a look reads only the head.Amendment E: the connection carries one request, then only doorbells.
MsgType::Regiontravelled on a connection where every byte after the answer is a doorbell: netstack would drain the request's 8-byteIpcHeader(ipc.rs:93-99) as rings, and the answer would land in the pipe the stream's readers drain.MsgType::Regiongoes, andtoyos::netrefuses by name a handle the process holds no end for (the handle bullet).make_socket_fd,sdk/std/sys/net/connection.rs), serves the stream through aFileor a child's stdio. Over the region it is the doorbell connection: a child would wait for ever or read doorbells as data, and netstack would drain its writes as doorbells, with no error. Stage 3 refuses both loudly instead (the handle bullets). A stream cannot move between processes:issues/a-stream-cannot-move-between-processes.md.Regionon the connection, with its boundary: a stream's own handle given out and taken back shares its one end, red when giving it out lets the entry go (round 5's BLOCKER 3). And a test thatFile::from_raw_fdand a spawn's stdio refuse a stream's handle, each red when it takes the handle silently (ruling 7).True zero copy: dropped
Received bytes straight from the card into the application's region would remove
cy_ring_write, the one copy the session design keeps, and nothing else:readstill copies from the region into the caller's buffer, becausestd::io::Readhands in the buffer. Round 3 measured that copy at 0.225 ms per MB of 5.447 ms of C0 at K = 32, 4.1%, and the scout read the same share (0.2185 of 5.310). To get it:pcidev::dma_map), which spendsMAX_GRANT_TOTAL, 32 MiB a claim (kernel/src/pcidev/mod.rs): four 8 MiB sessions.For a 4% gain the design would trade [tcp] parsing what the client cannot touch for per-flow hardware state, per-client IOMMU windows and a reassembly that copies in exactly the case loss makes common. Dropped under the owner's condition: he ruled zero copy out unless the design showed a useful benefit, and this reading of the gain, the design's and not his, finds none worth that trade. The session layout above carries no field for it.
Does the pipe design itself have a problem?
What pipes are for once netstack leaves them. Three users, from the code: every IPC connection, which is two pipes (
connect_through,kernel/src/syscall/ipc.rs) carrying frames of at mostipc::MAX_FRAME_LEN, 8 KiB; a process's stdio and the pipelines between programs (sdk/std/sys/pipe.rs,sdk/std/sys/process.rs), the one user whose volume is unbounded, a POSIX byte stream between two processes; and one-byte wakes (toyos::wake, the mio fork's waker, the supervisor's alive and service-end pipes) and logkeeper's stream to each log reader. Bulk data elsewhere already moves through regions: a file server's window (toyos::fs::WINDOW_BYTES), a block session (lib/blockring), sound and the compositor's buffers.Are two copies inherent? For a kernel-buffered byte stream whose reader and writer each hand in their own buffer, yes: the writer's buffer cannot be the reader's, and the kernel holds the bytes between the two calls, so each byte is copied in and copied out. The defect was not the pipe's: it was netstack keeping every byte in its own TCP ring and in the pipe, two buffers of one stream, which is the third copy and the 12 MiB a place. The session design makes the ring the buffer.
What
SYS_PIPE_MAPis for. Nothing, today. Its last caller, netd, mapped its clients' rings only to read the closed flags, and stopped in a237346 when those were found forgeable; since then only guest tests call it. It is the reason a pipe's ring is a whole 2 MiB page, the only page this kernel maps, and whypublish_endsstill writes flags into it: filed asissues/a-pipe-is-a-two-mebibyte-page-for-a-mapping-nothing-makes.md, not fixed here.Is the global
PIPESlock a flaw beyond its issue?issues/the-global-pipe-lock-spans-a-user-copy.mdalready names the whole of it: one table lock is also every ring's serialization, so a copy of a length the caller chooses, and a first write's allocation and 2 MiB zeroing, run under the lock every connection, readiness poll, clone and close in the machine takes. Its fix direction, a per-pipe lock under the table lock and the allocation out from under both, answers it. Nothing found here is outside that issue, so nothing new is filed. Round 3 is the first measurement of it, 0.281 ms held and 0.047 ms spun per MB, and the scout split the hold: the two copies are 73% of it.What it buys, measured and not guessed
Round 3's rows. Round 3's B at K = 32, medians of the six runs per MB downloaded, from the judge logs b 1a b 2a b 3a and their halves posted on #861:
pipe_lock_hold_cyclespipe_lock_spin_cyclescy_ring_writecy_pipe_wpipe_copy_bytesThe lock and its copies are 0.328 ms of 5.447, 6.0%. These rows cannot say how much of the hold is the two copies and how much is everything else the lock covers, and so which part the session design removes and which part the per-pipe lock of the lock's own issue would.
The scout (stage 0). One T14 boot of
internet_download, round 3's B with three kernel counters more and every run told K = 32:pipe_lock_takes, and the stamp ticks oftry_read's andtry_write's copy alone (pipe_read_copy_cyclesR,pipe_write_copy_cyclesW), each timed around thewrite_run/read_runthe ring hands its run to. Patch: the comment; branchwt/toyos-netring-scout2f9e090, measurement only, never landing.The rule, stated before the boot. What the session design removes of the pipe lock on the receive path is S = H − R + P per MB (H the hold, P the spin): the write copy and every non-copy hold and spin, while R stays as the client's own copy out of the region. S ≥ 0.10 ms per MB (about 2% of round 3's C0): the build may claim a CPU saving, and its own T14 boot is judged against the scout's rows. S < 0.10: the design is built on the owner's ruling for what pipes cannot say and the 4 MiB a place, and claims no saving. H − R − W > W: most of the hold is not a copy, and the per-pipe lock takes most of the gain without this design, so that issue goes first. The instrument is sound only if R + W ≤ H in every run and
pipe_copy_bytesis 2.00.The reading (judged; the judge,
cargo test --test toyos-build -- --metal --metal-readback <scratchpad>/netring/scout internet_downloadon 2f9e090, EXIT=0, its log whole and masked in part 1 and part 2; the flash, toyos-metal's lines and the boot). toyos-metal exited 0, and all six runs read the whole 170,439,044-byte file with the right sha256. R + W ≤ H in all six;pipe_copy_bytes2.0028 in all six. Medians per MB: R 0.1197 ms, W 0.0806, H 0.2728, P 0.0397, 185 takes, C0 5.310 ms, 785.95 Mb/s.PIPES, and the takes, 185 per MB against about 20 data writes, are mostly not the stream's copies. So the lock's share it removes lies between W = 0.081 and S = 0.192 ms per MB. netstack's pipe-write syscalls and the client's read syscalls go too, and their entry, lookup and return run outside the hold, in neither number; the doorbells' syscalls come in. Stage 3's boot reads the whole, which may be more than S, and its pull request claims no more than that boot reads.The roast
Cost. By the files it rewrites, not a measurement: netstack's
serve.rs,client.rsand the node'sstreams.rs,listeners.rsanddatagram.rswith their tests (2,866 lines of tests innode/tests/streams.rs,listeners.rsanddatagram.rsalone, 1,440, 1,106 and 320 bywc -l, most of which assert pipe behaviour and are rewritten, not kept);toyos::net; std'sconnection.rs; libc'ssocket.rsandstreamend.rs; [tcp]'sRingover the region. The byte ring and its loom model intoyos-transportare paid: stage 2 (#878) landed them. Production should shrink:pipes.rs, the end-ordering, the per-datagram request,SYS_CONNECTION_JOINand the doubled buffers go, but for a passive child before its accept (amendment C); the client end is new.ABI churn. In the kernel:
SYS_FSTATanswersDeviceMemory, a newFileType, which every reader of a region's file type sees (stage 1, landed), andSYS_CONNECTION_JOINgoes (stage 4);SYS_HANDLE_RECVandSYS_SHM_MAPkeep their shape. In userland: the netstack message set is replaced. Three forks move in stage 3's pull request: mio (itstoyos_stream.rsandtoyos_listener.rs) and socket2 (src/sys/toyos.rs), each a commit on itstoyosbranch and a lockfile bump, andrust/, a commit on its branch and a gitlink bump: itslibrary/std/src/os/fd/owned.rsholds the ToyOS arms ofOwnedFd's drop andBorrowedFd::try_clone_to_ownedthat stage 3 routes throughtoyos::net, and itsos/fdcompiles out fortoyosthe conversions stage 3 adds.Risk.
toyos-transport's discipline and loom model carry over to the byte ring, and itsCONTROLSfeatures (publish-relaxed,no-clamp, and toyos-transport: a byte stream between two ends that do not trust each other, and its wake handshake (netring stage 2) #878'shead-relaxedandwant-relaxed) give the negative controls.PIPES, one byte per sleep rather than per batch, and a stream holds three pipes where it held two (amendment A).READABLEandPOLLIN, and the bell forWRITABLEandPOLLOUT. So two limits serve half the sockets watched both ways that they serve today. libc'spolltakesnfdsup toPoller::MAX_HANDLES, 256 (sdk/libc/src/posix_io.rs). A mio ring holdsMAX_PENDING_WATCHES, 1024 (kernel/src/inbox/mod.rs). A socket watched one way costs what it does today.OwnedFddrop andtry_clone_to_ownedin every ToyOS program, eachFile::from_raw_fd, and each handle a spawn puts in a child's stdio looks intoyos::net's table of the process's stream ends, under its lock. In std on ToyOS onlysdk/std/sys/net/connection.rs's sockets hold anOwnedFd(rg -n OwnedFd sdk/std); another crate's pays at its drop. Not measured: nothing is built.Known limit, a regression from today's ToyOS: a stream does not move between processes (amendment E, ruling 7). A program that hands a stream to another process through a
Fileor a child's stdio, as an inetd-style server does, works today and fails loudly after stage 3; one that takes a moved stream back as aTcpStream, as socket activation does, is refused.CommandExt::inherit_handlepast the stdio slots andCommandExt::endowstill hand one over unrefused (What I am unsure of). Filed asissues/a-stream-cannot-move-between-processes.md, owned by the orchestrator, with E's exit: a stream moved by its handle reads and writes in the receiving process, and the process that moved it holds no end.What it saves. On the receive path, of the pipe lock, between the write copy, 0.081 ms per MB, and S = 0.192, 3.6% of C0, by the scout, and beside it the pipe syscalls' work outside the hold, which no row reads apart; on the send path two copies of every byte and a pipe; the request and connection per datagram; the
PIPEStraffic of every stream; and the structural debts above, three of them filed issues it closes.The alternative it is measured against is to keep the pipes and take the per-pipe lock: no ABI change, the lock's convoy gone, every copy kept. The scout's third rule is where that would have won, and it did not fire.
Staging
The track's stages, each a pull request, built in this order:
fstat'sDeviceMemory, andSharedMemory::lentjudging a lent region before it maps it. Useful alone: fileserver and diskserver judge their lent regions through it.toyos-transport.toyos::net, std's raw-handle conversions forTcpStream, and its refusal of a stream's handle as aFileor a child's stdio. Its exit is the track's:want-relaxed, and, with a fatalend, red where a look reads only the head (ruling 2);toyos::net, handed a stream handle the process holds no end for, refuses it by name and writes nothing on its connection, and that a stream's own handle, given out byinto_raw_fdorFrom<TcpStream> for OwnedFdor made bytry_clone_to_owned, and taken back, shares the one end, red when giving the handle out lets its entry go (amendment E, round 5's BLOCKER 3);File::from_raw_fdof a stream's handle panics with that refusal and a spawn whose stdio slot holds one answers it with no child started, each red when it takes the handle silently (ruling 7);writewith room after a reset answersConnectionResetand copies nothing, red whenwritecopies before it reads theend(ruling 3);git diff 777c96b2b 2f9e090b8, carried onto its head, readingpipe_copy_bytesunder 0.01 per byte downloaded beside the scout's rows.SYS_CONNECTION_JOINwith it. Its exit is the track's: over the region,a_listener_answers_a_syn_and_wakes_its_owner_when_the_handshake_endsand ruling 1's case pass, the case red under each of its three controls; and a datagram record whoselenis past the bytes it holds is refused while netstack serves on, red when netstack takes thatlenas written (ruling 4).Stages 3 and 4 change the ABI with the work that needs it, as root
CLAUDE.mdasks, and each says why userland cannot do what it adds to the kernel, if it adds anything.Checks
This branch changes no production code, source, test or manifest: three issue files, and the merges of
main. Net lines (git diff --shortstat origin/main...0831ca65f,origin/mainat 1acaa36): 3 files changed, 219 insertions(+), all underissues/; production 0, tests 0. Each stage names its own negative control and oracle in its pull request.head-onlyfor ruling 2; and stage 2'swant-relaxedfor A, B and D. E, its cost and rulings 3 to 7 rest on readings, and the exits carry their tests. The independent oracles are RFC 9293, POSIX.1-2024 (IEEE Std 1003.1-2024:accept,select,send,sendto), mTCP's and TAS's sources, mio's and tokio's sources, and virtio 1.2, each at the version cited.Gates
cargo run -- --ci hoston 0831ca6: this branch, withmainat 1acaa36 merged. It ran from 19:26:56Z, after 0831ca6 was committed at 19:26:50Z, to 19:31:04Z with EXIT=0, and its log endsHost: 76 step(s), all green(line 8802). Each of its 49test result: FAILEDlines is inside a[ci] controlstep. The log is posted whole, the worktree masked as<tree>and the home directory as~, and holds none of the shapesbuild/sourcegate.rsrefuses (a vendor MAC, a dotted quad, a global or link-local IPv6 address, a serial number, a personal hostname): a scratch scan for each found only itsHH:MM:SSstamps and a target triple'sx86_64-pc. host.log at evidence 0ed15661..claude/agents/.toyos-transportis unchanged since 7ea0ff9, which A's and B's models ran and D's ran a copy of:git diff --stat 7ea0ff9d6 0831ca65f -- lib/blockring/transport lib/untrustedis empty. Each model case iscargo test --offline --release --test <model> -- --exact <case>, run by itsrun.sh, beside its source, manifest and lockfile.ungated, 0, 0, 0 (log); unbounded,pass_on_bytes_leftstopped at 600 s with no verdict, EXIT=143 (log).issues/and.claude/agents/reviewer.md(git diff --stat 04ff95bc7 0831ca65f):sh run.sh <tree> <dir>applies the case, runscargo test -p toyos-net-node --test listeners, EXIT=0 (32 passed), then each control, EXIT=101, 101, 101, then restores,git status --porcelainempty (log). The first round'snowhere.patch: EXIT=0, then 101 (log).cargo test --test toyos-build -- --metal --metal-readback <scratchpad>/netring/scout internet_downloadonwt/toyos-netring-scout2f9e090: EXIT=2, the staging verdict, which is not a pass: nothing was judged (tests/common/metal.rs). Log and request.txt. Its judge after the boot is under "The reading".What I am unsure of
toyos::nettakingtoyos-transportas a dependency moves the sysroot's identity in a waybuild/identity.rscharges every build for; read before stage 3.want-relaxedall red within the bound. Unbounded, one case gave no verdict in ten minutes.WouldBlockafter its event. A mio program that reads a socket from a thread mio did not wake, and stops short ofWouldBlock, can miss its next event, as mio documents.reregisterthat changes only the token should ring netstack too. As written, D's does.CommandExt::inherit_handlepast the stdio slots andCommandExt::endowhand a child a stream's handle too, which reads doorbells there, and anendowmoves the handle out of this process's table whiletoyos::netkeeps its entry. The spawn's check costs the same lookup for every handle it hands over; whether it covers them is the orchestrator's to rule.send_toerror today (udp_send_to,system/netstack/src/serve.rs;each_refusal_is_answered_in_the_pipes_word_for_it). Over records a send has no answer:send_toreturnsOkand the refusal is only a counter, a payload pastMAX_PAYLOADincluded, which POSIX'ssendtofails[EMSGSIZE], "The message is too large to be sent all at once, as the socket requires". Whethertoyos::netrefuses what it can judge before it writes a record is the orchestrator's to rule.🤖 Generated with Claude Code
https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy