Repository navigation
LAN stage D and B: netstack wakes only on a change, and the I219 takes a 1,000-descriptor ring at a fixed 131 µs throttle - #861
Conversation
…and a watch the kernel holds armed is not submitted again Two costs every connection paid on every pass, both read off issues/netstack-watches-both-pipes-of-every-connection-on-every-pass.md. The kernel: an OTHER_END_GONE poll sat on its own end's data watch, which the other end's every read or write posts, so each read a client made of its receive pipe fired netstack's poll on that pipe for a look that found the reader still there. Each pipe end now has a leaving watch that only the other end's last holder posts (pipe::left_watch); release posts it beside the data watch it always posted, and a close cancels the polls on it with the end's others. A watch that asks a direction as well needs no entry there: the same release posts that direction's watch. netstack: every pass submitted a watch for the acceptor, the card's claim, both pipes of every stream and every other place's pipe, each one replacing an identical poll the kernel still held armed, and each paying its registration on every watch it names. A kernel watch stays armed until it is answered or replaced, and is answered by what the object holds at the look that answers it, so watches::Armed records what is armed by token and a pass submits only a watch whose handle or interest changed or whose answer it has read. Every armed watch is answered once, a closed handle's with NotFound, so no record outlives its poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
m3 mutant: cargo test -p netstack watches (EXIT=101) |
|
cargo test -p netstack at c91e72b (EXIT=0) |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 1 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 2 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 3 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 4 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 5 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 6 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 7 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 8 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 9 of 10 |
|
cargo run -- --ci host at c91e72b (EXIT=0), part 10 of 10 |
|
cargo test (whole guest suite) at c91e72b (EXIT=0), part 1 of 2 |
|
cargo test (whole guest suite) at c91e72b (EXIT=0), part 2 of 2 |
|
netperf_qemu, main arm (wt/toyos-lan-metal 61ed244), whole log; counts only |
|
netperf_qemu, D arm (wt/toyos-lan-headmetal 0ebe830), whole log; counts only |
|
Mutations, as checked patches, each applied, run and reverted in one script
--- a/kernel/src/pipe.rs
+++ b/kernel/src/pipe.rs
@@ -316,7 +316,7 @@ fn release(pipe_id: PipeId, end: End) {
(Released::EndGone, End::Read) => crate::scheduler::wake_pipe_writers(pipe_id),
(Released::EndGone, End::Write) => crate::scheduler::wake_pipe_readers(pipe_id),
}
- left.post();
+ drop(left);
}
fn slot(end: End) -> usize {
--- a/system/netstack/src/watches.rs
+++ b/system/netstack/src/watches.rs
@@ -33,7 +33,7 @@ impl Armed {
/// The kernel answered the watch under `token`: it is armed no more.
pub fn answered(&mut self, token: u64) {
- self.0.remove(&token);
+ let _ = token;
}
}
--- a/system/netstack/src/watches.rs
+++ b/system/netstack/src/watches.rs
@@ -27,7 +27,7 @@ impl Armed {
/// Whether a watch of `flags` on `handle` under `token` has to be
/// submitted, recording it as armed where it does.
pub fn submit(&mut self, token: u64, handle: u32, flags: u32) -> bool {
- self.0.insert(token, (handle, flags)) != Some((handle, flags))
+ self.0.insert(token, (handle, flags)).is_none()
}
/// The kernel answered the watch under `token`: it is armed no more.m1 script (kernel; on wt/toyos-lan-headmetal, D plus the instrument): m2 and m3 (host): |
|
m1 mutant run: cargo test --test toyos-build -- netstack_s (MUTANT EXIT=1) |
|
m1 restored run: same filter (RESTORED EXIT=0) |
|
m2 mutant: cargo test -p netstack watches (EXIT=101) |
|
Round 1, arm A (main + instrument), boot 1: |
|
Round 1, arm B (stage D + instrument), boot 1: |
|
Round 1, arm A (main + instrument), boot 2: |
|
Round 1, arm B (stage D + instrument), boot 2: |
|
Round 1, arm A (main + instrument), boot 3: |
|
Round 1, arm B (stage D + instrument), boot 3: |
|
Round 1, arm U (Ubuntu on the same T14, stage 0's binary, three runs): |
|
Round 1, arm S (the shared boot at c91e72b, |
|
Round 1, arm A boot 1's readback: |
spacing.py now prints the median of the seven Ubuntu SYNs that did not follow a download within 3 ms (first in their process, or after a resolver miss), the group the analysis sets ToyOS's syn_wire against. Its ToyOS header now says what the gaps are measured from: the fetch's own DNS query and the previous download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
|
The connection-setup scout on the T14: the analysis. The scout ran The plan was written before the run: the request, with its rows, tripwires E1-E4, kill conditions and prediction, and the investigation behind it. Every log is whole, masked, at evidence
Marks and units:
Tripwires: all four hold, and nothing is void
One capture failed, outside every condition.
Kill conditions, applied as written
What the verdicts alone do not say1. S2's alive verdict rests on how the Ubuntu arm spaces its fetches. The split below is post hoc and decides nothing (
2. A SYN-only cost of 0.67-0.84 shows in every fetch [D]. It is under (a)'s line, and Ubuntu has no data_rtt to set it against. 3. The local time is paid per netstack request, not per connect (
Where the gap goes, against the predictionIn this session, ToyOS's tcp is 23.331 [M] and Ubuntu's U3 is 14.685 over all 18 [D], a gap of 8.65 [D]; over v4 only it is 8.92 [D]. The "~10 ms" came from rounds 3 and 4 (22.8 and 23.4 against 13.2). ToyOS held its figure; Ubuntu came in 1.5 slower this time.
The Ubuntu predictions:
ToyOS's TLS phase is 0.58 shorter than Ubuntu's [D], so all of the setup gap is in tcp. In shares of the gap:
What the result asks forThese are options for the owner to choose between; none is chosen here. 1. The lookup: 6.3 of the 8.6. netstack keeps no cache: "No cache. Every lookup asks." (
2. The SYN: 1.5-1.8 as written, 0.41 against like-spaced fetches. It is not actionable until a rerun separates the arm's spacing from ToyOS. That rerun could:
3. Local IPC: 0.80 net, 1.30 local sum, about 0.5 a netstack request.
4. NIC/wake: not dead by its 50 µs line, at 0.059 a reply [M]. The tcp path holds two such wakes, about 0.12 [D], and both already sit inside the local rows. Nothing is proposed here until 1-3 move. |
Triage of an external automated review (ChatGPT, supplied by the owner)The external review read a8a14f8. That commit was made at 22:11Z on 10-11, before round 4's logs were posted at 22:43Z. The head is 443fb5d. Since a8a14f8 the branch has merged main, added 1df3ace (STRETCH = 8, from round 4) and deleted the pacing by the owner's ruling. I judged each claim at 443fb5d against the body, the tree and the three review rounds: 614f709e9, 1df3ace22 and 443fb5d10. Counts:
What the review finds
Its table of earlier findings, and its other statements
|
… for the machine that empties with the lease The design accepted on #883, with round 2's named changes. Every lookup asked the lease's resolver across the WAN; on the T14 that round trip was 6.31 ms of the 8.65 ms by which ToyOS's TCP setup trailed Ubuntu's (#861's scout). toyos-dns: the reader reads each record's TTL unsigned and caps it at an hour (RFC 8767 §4), and for a negative answer the alias chain of an NXDOMAIN reply and the authority section's SOA, which counts only where it is owned by the denied name or an ancestor (RFC 2308 §5). A lookup keeps one clock: every TTL it reads counts from its first query's send, so neither a retried query nor an alias asked again stretches what an earlier reply said. `Cache` keeps an answer per view, under the name asked only, bounded at 128 names a view and 32 views; a full view drops its expired entries and then its least recently used, and a new view replaces one whose entries have all expired before the least recently used live one. An emptying is a new generation, and an answer whose lookup started under an older one is not kept. toyos-net-node: the cache sits beside the lease state in one type whose state only its setter writes, and the setter empties it; a link that goes down empties it too. `Node::resolve` takes the view, answers a kept name in the call, holding no lookup place, and keeps an ended lookup's answer. netstack: a lookup's view is the badge on its connection, read per lookup; an unbadged connection's lookup is refused ERR_PERMISSION_DENIED. The two log lines that named a failed lookup's name no longer do. supervisor: every program whose row receives netstack is minted its own connector with its row and session; the shared connector is dropped at once. Boot hands every service its acceptors before it starts any, so a mint on another service's port does not depend on [boot] start's order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Brings in #867, #878, #879, #880, #882 and #885. The one file both sides touch, kernel/src/object/ops.rs, merges without conflict: main's hunk is in `fstat` and this branch's are in `close` and `pipe_end_watch`. The branch's diff against main after the merge is its diff against the old base, 177e72a, line for line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…es is unmeasured The one finding of the owner's external review that held in triage (#861 (comment), item 4). Design B raised `Node::receive`'s budget from main's 64 to the card's ring, 999 on the I219, and a batch's frames share one `now` and run before the deadlines, the transmit opportunity and the clients' answers. That changes latency only for a pass that brings more than 64 frames, and round 4's LAN round trip was read at about 12 a pass. Filed with the LAN track's holder as owner, and as the exit a T14 reading at both budgets that is reported and, as #885 has it, decides nothing about the budget by itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…33b2, every log whole `cargo run -- --ci host` EXIT=0, `cargo run -- --build-only` EXIT=0 and `cargo test -- netstack libc_sockets https_fetch` EXIT=0, run in that order by gates.sh, each exit read from the command itself. merge-delta.sh and its output hold that the merge of origin/main at 36aff86fc leaves the branch's diff against main as it was against 177e72a9e. Local paths are masked as <worktree>, <home> and <tmp>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…ed-ruling #871 closes issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md (cdd090b), which this branch modified: the deletion is taken. This branch's four hunks there are accounted for: the SDM §16.4 and APM §2.5.12 citations backed a save at the switch, which #871 built; the §6.6 to §7.6 correction is at PerCpu::preempt_count's doc on main; the kill line and the exit's "or kept with its refusal reason" were about an exit #871 met in full. The track, which this branch takes up, drops the closed file from its measured stages and its exit list, and drops the "Unexplained" bullet on the syscall's lock saving, which pointed at that file and which cdd090b answers by counting ten stripped sites. Its exit list gains the two issues #871 filed with the track as their owner: every-syscall-takes-the-process-table-lock-twice-to-count-itself and a-thread-resuming-on-the-cpu-that-holds-its-state-restores-it-again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
#884 fixes the `iommu-firmware-left` ring whose overflow panicked every shared-debug boot on the AMD bench, which this branch's global-leaf rows need. #871 declares XCR0, IA32_XSS and CR4.OSXSAVE in the same control_regs declaration this branch adds CR4.PGE to. Four conflicts, each both sides kept: - kernel/cpuvuln/Cargo.toml, src/lib.rs: main's erratum_1386 (AMD's revision guides) beside this branch's invlpg_misses_global (Intel's specification updates), in the description and the crate doc. - kernel/src/arch/x86_64/control_regs.rs: the header names main's XCR0, IA32_XSS and save form, and keeps this branch's one other CR4 writer, flush_global; CR4_REQUIRED carries both PGE and OSXSAVE. supported() merged cleanly: this branch's CPUID.01H:EDX[13] row beside main's CPUID.01H:ECX[26] one. init() writes CR4 through PGE clear, then EFER, XCR0 and IA32_XSS. - tests/checks.rs: main widened each case to (cr0, cr4, xcr0) and moved DECLARED's cr4 to 0x0034_0668 (OSXSAVE); this branch adds bit 7, so 0x0034_06e8, swaps the never-named example from PGE to PKE, and adds "every CPU without PGE": 15 refusals. Clean merges checked by reading: dispatch.rs and syscall.rs (action 28 is still free), tests/toyos.rs (main's control_regs judge names OSXSAVE and XCR0 and not PKE; this branch's PGE row and three metal rows sit beside main's), cpu.rs (main's leaf_d, xgetbv0, xsetbv0 beside this branch's revision read and INVPCID type 3), and idt/tlb.rs, whose entry #871 rewrote and which still calls the same serve_ipi. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
serve.rs's one conflict is its imports: #861 took `Poller` out for `watches::Watches`, and this branch added `toyos::port::Acceptor` for the badge read. Both are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…top's design branch #871 makes CR4.OSXSAVE required and gives user programs AVX state in XCR0, so std's feature detection reports AVX and FMA on a CPU that has them, and rubato's own dispatch takes AvxInterpolator on the T14 and the AMD bench. Round 3 of the design measures the resampler on that and on SseInterpolator, which rubato takes on an AVX CPU without FMA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
No conflicts. #871 moves the user FP/vector save to the switch and changes the context's identity (`runs`); this branch's kernel side touches neither: `KernelHw::kick`'s self-route sets `need_resched`, which the pass still clears before it drains (`sched::driver::pass`), so the pass this branch's kept probe asks for survives into the next poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
…s no tree The finding said both T14 arms predate #861 and asked for a run on a tree with it. Both arms, `89a2dd00f` and `703f474c6`, descend from #861's head `443fb5d10` and carry `system/netstack/src/watches.rs`, so that exit was already met. What they predate is #871's kernel; the exit now asks only for the stamps that name what the slow mode waits on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy
The LAN part of the combined XSAVE/AVX and crypto work: ToyOS's LAN path held to Linux's speed, efficiency and security on modern hardware, with the T14 as the instrument. This branch carries only the netstack and kernel side. TLS still runs on ring here, and its numbers are kept apart from the netstack's. Two stages: D (netstack wakes only on a change) and B (the I219's receive ring fills its grant, the tail moves once a batch, and the throttle is a fixed 131 µs).
State at ec72322, which is 443fb5d with origin/main merged in at 36aff86 and one issue filed; the branch's code is 443fb5d's, line for line. The pacing is deleted, by the owner's ruling on the re-review: no ACK clock, no stretch, no round-trip plumbing, no per-pass interval. What B keeps is what round 4 on the T14 measured. Against Linux it is not there yet: throughput about matches; cycles, busy time and connection setup are behind; energy against Linux is not measured (below).
Stage D: netstack wakes only on a change
Two costs that every connection paid on every pass, both named in
issues/netstack-watches-both-pipes-of-every-connection-on-every-pass.md.The kernel: a pipe's leaving has its own watch. An
OTHER_END_GONEpoll sat on its own end's data watch, which every read or write the other end makes posts. So each read a client made of its receive pipe fired netstack's poll on that pipe, and the look it was owed found the reader still there. Each pipe end now has a leaving watch (pipe::left_watch) that only the other end's last holder posts.releaseposts it beside the data watch it always posted,ops::pipe_end_watchanswers it, andops::closecancels the polls on it along with the end's other polls. A watch that also asks for a direction needs no entry on it, because the same release posts that direction's watch too.netstack: a watch the kernel holds armed is not submitted again. Every pass submitted a watch for the acceptor, the card's claim, both pipes of every stream and every other place's pipe. Each one replaced an identical poll the kernel still held, and each paid its registration on every watch it names. A kernel watch stays armed until it is answered or replaced, and its answer is whatever the object holds at the look that answers it. So
watches::Armedrecords what is armed, keyed by handle as the kernel keys polls, with a token index for answers, and a pass submits a watch only when its token or interest changed or once its answer has been read.Why it lives in netstack and not in
toyos::Poller: every other Poller user resubmits every loop. If some object's readiness flips without a post on its watch, that resubmission hides the missing post, and the same change in the SDK would turn the missing post into a hang for that user. netstack's handles are pipes, a claim, an acceptor and connections, and every one of those posts its watch.Stage B: the ring that fills the grant, the tail once a batch, a fixed throttle
The design, as ruled. The owner chose option (b) of the re-review: delete the ACK clock, the stretch and the round-trip plumbing that served them, and keep the 1,000-descriptor ring and the fixed minimum interval, which the T14 measured. Nothing unmeasured stays.
What the code at the head does:
toyos_i219::RX_RING), filling the 2 MiB span the kernel maps a grant in. Compile-time assertions hold that it fits and that eight more would not.Node::receive(RX_RING − 1on the I219, the queue's 256 on virtio), so one wait's frames are one batch. It still answers whether another pass is owed.RDTwrite (rx_flush), or sooner onceHELD_MAX(a quarter ofRX_SHORT) wait. Readiness is still read from the descriptor.ITR = 512, 131.072 µs, written once at bring-up, as main writes its 500 (128 µs). Nothing writes it again.Why 512 and not 160 µs or main's 128 µs. 512 units is what round 4's B wrote on every pass at a gigabit, read from its code (1df3ace's
cadence.rs): there the ACK clock and the stretch both fall under the 160 µs floor (the stretch is 8 frames of 12.3 µs), the ring's bound read 3.07 ms (below), so every pass asked 160 µs, and the three-leading-bits rounding this head deletes turned that into 512. Its ten runs at S = 8 read a mean applied per pass,itr_us, of 131.09–131.16 µs, a little above 512 units' 131.072; the 443fb5d review found no figure this body states moves on that. 160 µs was never applied, and main's 500 is not what B ran, so neither is taken. The commit message says the same.Deleted with the pacing, beyond what the ruling names. The ring's bound on the interval, the per-pass
ITRwrite and its rounding (throttle_units),rx_free,Card::pace,cadence.rs, and node'sBatch. The ring's bound would have shortened the interval when the ring was short of room, and it could not act, read from 1df3ace's code (the 443fb5d review's derivation):Card::receiveflushed the tail before it paced, and the flush hands every buffer back, sorx_freereadRX_RING − 1= 999 on every pass, the room above the 500-descriptor reserve 499, and the boundframes(499) / 2= 3.07 ms at 1,000 Mb/s, 30.7 ms at 100 and 307 ms at 10: never under the floor at any rate the I219 links at. Deleting it changes no register write round 4's B made. The ring holds half its descriptors in reserve against about ten full-sized frames an interval at a gigabit. The ruling keeps a fixed minimum interval, and a bound recomputed every pass is not fixed; the 443fb5d review judged the deletion inside the ruling.Issues.
issues/netstacks-receiver-round-trip-reads-no-finer-than-a-millisecond.mdstays, rewritten: the cadence paragraph and itsKgo with the code, and what is left is the receive buffer's growth on a sub-millisecond path, which rests onrx.rsas it is on main.issues/closing-a-pipe-end-ends-every-holders-polls-on-it.mdis stage D's, found in review. No other issue existed only for the deleted code:a-receive-flow-with-no-tcp-round-trip-is-paced-at-the-floor.mdwas opened and deleted inside this branch, so it never reached main.issues/the-lan-is-not-yet-production-grade.mdnames a fixed throttle again, with no number.issues/netstacks-latency-under-a-receive-batch-over-64-frames-is-unmeasured.mdfiles the one finding of the owner's external review that held in its triage (item 4): the budget's rise from main's 64 to the ring, with onenowa batch and the deadlines, the transmit opportunity and the clients' answers after its last frame, changes latency only for a pass that brings more than 64 frames, and round 4 read the LAN round trip at about 12 a pass. Its owner is the LAN track's holder, and its exit a T14 reading at both budgets, reported and, as #885 has it, no verdict on the budget by itself.The re-review of 1df3ace, answered
FLOORbuilt without an accepted design: the code is deleted. What stays is the ruling's design.The 443fb5d10 review's NOTE, that the ring bound's deletion rested on
itr_us's mean, which does not show it: "Why 512", "Deleted with the pacing" and "Is round 4's B representative" now rest on 1df3ace's code,rx_freeread after the flush, as the review derived it.What B's gains are: the ring, not a cadence
Round 4 (analysis, rule output) alternated D and B five times on the T14 in one session: one boot each, six downloads of the 170,439,044-byte file a boot. All 60 runs read the whole file with the right sha256. M is read from a counter; D is derived, its derivation named.
On that link, B's interval was the floor in every run. At S = 8 it was 131.1 µs, against main's 128. Neither the ACK clock nor the stretch acted. So what round 4 measured of B at S = 8 is D plus the 1,000-descriptor ring, F1 and 3 µs more of throttle. That is this head's B.
RXOoverrun. B dropped none in any of its 30 runs at any S. Why D's software falls behind is still not known; the larger ring absorbs it and says nothing of its cause.counters::bring_upenables fixed counter 0, thenpmu::armwrites both control registers whole and clears that enable. The instrument is onwt/toyos-lan-metal4, which never lands.mainhas no other PMU user, andkernel/src/arch/x86_64/pmu.rsstates that invariant at the site, so nothing is filed.Is round 4's B representative of this head? Yes, for everything the part does, and no confirming T14 boot is needed for the claims above:
ITR= 512 at open and, read from its code, on every pass at a gigabit (above, "Why 512"). This head writes 512 at bring-up and never again. Same ring, same F1, same budget.rx_free, the interval's arithmetic and a compare; per segment, atake_least_rttminimum.netstack_streams_e1000eruns the driver's bring-up, ring and flush on QEMU's 82574 at this head, and passes.The flood floor, against the remaining code
The bring-up test (
bring_up_programs_what_the_initialization_sections_name) again reads the throttle the part holds after bring-up, as main's did:ITR × 256 = 131,072 ns. Nothing else in the driver writesITR, so no peer's frames wake netstack more often than every 131 µs at any link rate.no_batch_holds_an_i219_under_128_uswent withcadence.rs, the function it tested.Close and the leaving watch (stage D, unchanged since a6ac028)
pipe_other_end_goneholds that an end closed by its own holder, its other end alive, answers itsOTHER_END_GONE-only watch withNotFound, for both ends, within a 10 s bound that fails loudly. Why that tier:closeand the pipe's watches have no host test (the kernel library's host tests reach neither), so the cheapest tier is a metal row, and this test is a member of the T14's shared boot. On the T14 in round 4: S (unmutated) exits 0. SMUT (the review's patch,closeskippingpipe_end_watch) exits 101 atpipe_other_end_gone.rs:88:9,left: [], 10.006 s after spawn, the bound.Against Ubuntu (the owner's bar: match or beat Linux on speed, efficiency and security)
Ubuntu's figures are round 3's (comment): 3 runs, same machine and cable, another session; round 4 ran no Ubuntu arm. D's own cycles moved 10.7% between those sessions (14.289 to 15.821 Mcyc/MB), so differences of that size are inside cross-session drift. Medians per MB; the last column is derived from the two medians beside it.
tcp_ms(name lookup and connect)tls_msissues/the-lan-is-not-yet-production-grade.mdstays unmet. The ruling hands the gap to netring: a session region replaces the pipes between netstack and its clients (design, and the track that stages it) #863's session design and the speed track's kernel work.Negative controls
Each a checked patch applied, run and reverted by one script, tree left clean (script, summary, patches and every log). m1, m2 and the restored run are at 443fb5d, whose lines the merge of origin/main leaves as they were.
THROTTLEback to main's 500cargo test -p toyos-i219bring_up_programs_what_the_initialization_sections_name:left: 128000,right: 131072RX_RINGback to main's 256assertion failed: layout(RX_RING + 8).1 > GRANT_BYTES as usizecloseskipspipe_end_watch, at a8a14f8's kernelpipe_other_end_goneArmedkeyed by token, at a6ac028cargo test -p netstacka_handle_under_another_token_is_submitted_and_so_is_the_first_againan_answer_under_a_replaced_token_lets_the_handles_record_standa6ac028's m20–m21.
watches.rsis unchanged since then.The whole change reverted is round 4's D arm and round 1's main arm, below. Independent oracles: the I219 datasheet's 256 ns unit (§10.2.4.2) and
RDLEN's multiple of 128 (§7.1.8), and the T14, round 4.Evidence at the head
The head is ec72322: 443fb5d with origin/main merged in at 36aff86 (#867, #878, #879, #880, #882 and #885), and the issue above. The merge took no hand:
kernel/src/object/ops.rsis the one file both sides touch, main's hunk infstatand the branch's incloseandpipe_end_watch. The branch's diff against main after the merge is its diff against 177e72a before it, line for line (script, output,diffEXIT=0).Every log is whole, local paths masked, at evidence commit bf902772ab0d2e47c376f66eb041d1e27becd71d (the script that ran the three).
cargo run -- --ci hostcargo run -- --build-onlycargo test -- netstack libc_sockets https_fetch(netstack_streams,netstack_streams_e1000e,netstack_socket_churn,https_fetch,libc_sockets)pipe_other_end_goneon the T14 (S / SMUT), at a8a14f8's kernelThe branch's kernel diff is a8a14f8's, line for line. The whole guest suite was last run on this branch at 777c96b (60 of 61; its one red was #864's). The merge brings main's kernel and guest-test changes, #867's
fstatverdict and its two guest tests among them, and changes no line of the branch's own diff. This round ran the network filter, which covers every guest test netstack's image runs, and not the whole suite.Net lines (
git diff --shortstat origin/main...HEAD): 20 files, +549 −78;system/andkernel/+427 −77. This round: the merge, which adds nothing to that diff, and the issue file, +35.What I am unsure of
issues/netstacks-latency-under-a-receive-batch-over-64-frames-is-unmeasured.md.HELD_MAX, a quarter of the reserve, is a choice and not a measurement.Why no new guest test
What B keeps is reached on cheaper tiers. Types: the ring's sizing, as compile-time assertions. Host tests: the throttle at bring-up, the budget, the tail's batching, the ring's wrap, and the watches' records. Metal: the cost per MB, which is a measurement and not a suite row, and
pipe_other_end_gone's new case, which no host test reaches.netstack_streams_e1000ealready runs the driver's paths under QEMU.Round 3 on the T14 (superseded by round 4; kept as the record)
One session, one machine, one cable: D and B alternated D B D B D B, one boot each, six downloads of the 170,439,044-byte file a boot labelled K = 8 16 32 32 16 8, then Ubuntu three times. D (
wt/toyos-lan-dmetal3d82112e) is stage D plus round 3's instrument; it refuses theK=word, so its labels are only places in the order. B (wt/toyos-lan-bmetal352fea69) is stage B plus the instrument, taking K from the test. All 36 runs read the whole file with the right sha256, and no run was cut by the list bound. Every log is posted whole and masked: judge logs D 1a 1b 2a 2b 3a 3b, B 1a 1b 2a 2b 3a 3b; Ubuntu; kernel logs, which carry the overruns' times, D 1a 1b 2a 2b 3a 3b, B 1a 1b 2a 2b 3a 3b; and the scripts and their outputs every number below comes from (parse.py,rule.py,parts.py, each EXIT=0).K's rule, every K
Each B cell is the median of B's six runs at that K. Each bound is from D's eighteen runs. Bold is a failed guard.
tcp_mstls_msrtt_msrcv_rttat close, msNot guards, read for the roast (medians; D over all eighteen runs):
rtt_msguard reads the boot, not K. The test's five handshake samples are the same in all six runs of a boot.D, B at K = 32, and Ubuntu
Median [min–max]: D over 18 runs, B over its 6 runs at K = 32, Ubuntu over 3. M is read from a counter, D derived from measured rows, the derivation named. Per MB throughout. "Mcyc" is core cycles from APERF, so at each run's observed clock, which is its own row;
cy_*rows are TSC time.tcp_ms(name lookup and connect)tls_mscy_node_frame, ms (TSC)cy_checksum/cy_ring_write/cy_pipe_w, ms (TSC)Where ToyOS stands against Ubuntu, at B with K = 32:
rcv_rtt(15.3 against 12.6 ms) and LAN round trip (7.75 against 6.05 ms) medians are higher, inside the guards.tcp_ms22.95 against 13.2,tls_ms41.1 against 31.5.tcp_msincludes the name lookup.The remaining gap, 0.94 Mcyc/MB, derived from the medians above (the parts overlap, so they do not sum):
cy_node_frame, 1.28 ms of TSC time over 704 frames per MB. ToyOS takes 25.2 passes per MB against Linux's 43.7 interrupts, but 88.7 switches against Linux's 68.3.Roast 4, per-frame cost on the larger ring. In TSC time
cy_node_framerose from 1.00 to 1.28 ms/MB. At each arm's clock that is 2.61 against 2.44 Mcyc/MB, a fall (derived, assuming the clock was uniform across the run).New instrument rows (the instrument never lands):
K's decision rule, as stated in the request before the run (request and instrument). Cycles per MB is the metric. For each K, take B's six runs at that K and compare them with all of A's eighteen runs:
tcp_ms,tls_ms, the handshakertt_ms,rcv_rttat close, and the LAN round trip during the download are each not above A's highest run.K is the passing value with the least median system cycles per MB. If no K passes, B does not land at any K.
Evidence for stage D, at c91e72b
cargo run -- --ci hostcargo test(the whole guest suite, 61 of 61)cargo test -p netstack(11 tests, 5 new inwatches)The guest Rust programs,
pipe_other_end_goneamong them, run only on the T14's shared boot: round 1's S and round 4's S and SMUT.Negative controls. Each mutation is a checked patch, applied, run and reverted in one script, with the tree left clean: patches and script.
cargo test --test toyos-build -- netstack_s: MUTANT EXIT=1.netstack_socket_churngoes red: "net.piped.live is 1 and not 0 20s after a client that shut its sending half down dropped both its ends" (log). Restored, the same filter exits 0 (log).cargo test -p netstack watchesEXIT=101, 2 tests red (log).Counts in QEMU (
netperf_qemu, the measurement branch's job against a host server). These are counts, never timings. Main arm: log. D arm: log.Under TCG the client is CPU-bound, so its reads do not race netstack's park, and the fires do not move there. The T14 reads them below.
Round 1 on the T14: main, D and Ubuntu
One session, one machine, one cable. A (main plus the instrument) and B (D plus the instrument) alternated A B A B A B, one boot each, and each boot downloaded the same 170,439,044-byte file twice. U is Ubuntu on the same T14 straight after, running stage 0's binary three times. S is the shared boot at c91e72b. Every log is posted whole and masked: A 1 2 3, B 1 2 3, U, S (
pipe_other_end_gone, 1 passed), the outlier boot's kernel log, and the scripts and per-arm aggregates the table comes from.Each cell is the median [min-max] over 6 ToyOS runs (3 boots × 2) or 3 Ubuntu runs. "main w/o" is main without its outlier (n = 5). M means measured by a counter. D means derived from measured rows; the derivation is named in the row. All values are per MB.
cy_*rows are TSC time at 2.419 GHz. "Mcyc" rows are core cycles from APERF.wholeline, C0 time)cy_watch, msWhat D did. Submissions fell 4.0x and inbox fires halved, to one per pass. The fire per client read is gone. Looks, passes, kicks, switches and idle exits did not move, and neither did throughput. System cycles fell by 0.81 Mcyc/MB (−5.6%, 14.56 to 13.75). The five main runs (14.47-14.73) and the six D runs (13.36-14.17) do not overlap. The saving falls on netstack (−0.41 Mcyc), on the app (−0.22), whose read syscall was paying for the fire, and on the kernel (−0.11). Its own bookkeeping cost D +0.012 ms/MB in
cy_watch.The outlier is main's, not D's. Arm A, boot 1, run 1 took 35.3 s: 38.6 Mb/s at 11.85 cpu ms/MB. From its judge log and readback:
Where ToyOS stands against Ubuntu, at D:
tcp_ms21.1 against 15.8,tls_ms40.6 against 34.5.tcp_msincludes the name lookup, so the round does not yet say whether the lookup or the connect takes it.What dominates the gap. The app is at parity: 5.34 against 5.46 Mcyc/MB, even though ToyOS's AES-GCM runs without AVX and costs 0.78 against 0.42. The whole gap of 4.6 Mcyc/MB is the network path: netstack 6.68 plus kernel 1.36 makes 8.04, against Linux's 3.58 for its entire kernel and every other task. All of the following is derived from the median rows:
begin_pass, theICRread and write-back: 0.073;cy_rx_done, which holds all 710 writes/MB and their republish, is 0.020 ms/MB.Overruns, both arms. Every boot logged 1 to 4
RXOoverruns withRXDMT0reports, D and main alike, and most fell near a run's end or start.RXDMT0at half of a 256-descriptor ring means software fell at least ~1.5 ms behind at line rate, at about 25% busy. The cause is not known. It is the track's own unmet exit ("error counters asserted zero under load",issues/the-lan-is-not-yet-production-grade.md), and B below must not grow it.After B, in the profile's order
issues/every-interrupt-lands-on-the-boot-cpu.md, an owner-approved track). Each pass pays the forward from CPU 0: one of its 3 idle exits and 2 kicks. That track's sequencing is the orchestrator's to decide.The design B was built from (accepted, kept as the record)
interval = clamp(min(ack_bound, ring_bound), floor, hw_max).ack_bound= the least receiver RTT among streams that recently received in-order data, divided by K.ring_bound= (descriptors held − margin) × the full-sized frame time ÷ 2.floor= the wake path's cost, about 30 µs.Roast, as accepted:
tls_msguard it.The owner's ruling replaced this design; Stage B above is what the head builds.
The instrument (never lands)
Round 1's instrument is the local branch
wt/toyos-lan-metal:internet_downloadand its per-MB table, ported onto the restructured tree, with the judge's fix and asubmitscounter. Round 3 addswt/toyos-lan-metal3(e882ea5) on top of it, with the rows listed under Round 3. Arm A iswt/toyos-lan-dmetal3. Arm B iswt/toyos-lan-bmetal3, which also adds netstack taking K from the test, and its pace counters. None of these branches lands.These faults from round 1 remain:
The lookups the LAN probe adds also land in each run's counters. They are the same in both arms.
Round 4 adds
wt/toyos-lan-metal4(18593df) on round 3's instrument: the D-SACK classes and ages, lost-run lengths, MPC, instructions retired and package energy, and runs labelled by S. Arm D iswt/toyos-lan-dmetal4(b9e2d36); arm B iswt/toyos-lan-bmetal4(6efe8d72d), taking S from the test. None lands. Its instructions-retired row counted nothing (BLOCKER 2, above).🤖 Generated with Claude Code
https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy