From f1ee8dd945c588585a892b90945663296763b7ce Mon Sep 17 00:00:00 2001 From: japabu Date: Mon, 12 Oct 2026 15:55:17 +0200 Subject: [PATCH 1/7] issues: the speed track's stages follow their hardware's manufacturers and the state of the art, and their figures are reported MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner ruled that a design follows its hardware's manufacturers and the published state of the art, and that our bench reports what a design gains and never blocks it (#885, the owner's answers recorded on it). The speed track still judged each design round by a kill condition fixed before the run, and its stages carried kill lines, twins whose gain decided whether a design was built, and exits that let a measured loss or a measured cost reject a change. The track now says what a stage cites and how its figure is used: where the manufacturers or the published state of the art back a stage, its figure is reported and never blocks it, and a slowdown or a gap to Linux is filed as a defect and investigated; a stage they do not cover is reviewed on its correctness and simplicity, and its figure decides only where speed is its only purpose. Each stage that carried a kill line drops it and cites what backs it: - the syscall's FP save and the preempt count's `lock`: SDM 325462-093 Vol. 3A §16.4 and §7.6, AMD APM 24593 rev. 3.42 Vol. 2 §2.5.12; the exit's escape for a change kept on its measured cost goes; - a wake to a busy CPU: Lozi et al., EuroSys 2016, on the work-conserving invariant; - per-CPU lines: Intel's optimization manual 248966-051 Vol. 1 §11.4.5, Coding Rule 17; the exit's escape on a measured cost goes, and the attribution is read again after the change; - the MWAIT doorbell: SDM Vol. 3A §11.10.4, §11.10.5 and §11.10.6.4, and the optimization manual §11.4; the twin and its 20 % line go, and the exit reads the split again after the change; - the TSC deadline: SDM Vol. 3A §13.5.4.1; the twin, its 5 % line and the `rejected` outcome go, and timer lateness is read with both timers, no longer held to the count's. FSGSBASE's exit let a pipe round-trip loss reject it. Intel's Guidance for Enabling FSGSBASE (ID 767615) names what it gives and how an OS takes it, so the stage is now `issues/fsgsbase-is-taken.md` and the figure is read with and without it. The track's global-pages bullet existed to set up a PGE twin; it goes, and the decision stays `issues/page-global-is-a-decision-nobody-has-made.md`'s. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy --- ...s-next-pass-and-no-idle-cpu-can-take-it.md | 8 ++-- ...-and-every-preemption-disable-is-locked.md | 23 +++++----- ...base-is-measured-then-taken-or-rejected.md | 21 --------- issues/fsgsbase-is-taken.md | 25 ++++++++++ ...-so-every-wake-of-an-idle-cpu-is-an-ipi.md | 46 ++++++------------- ...pmm-state-shares-lines-every-cpu-writes.md | 17 +++---- ...e-armed-every-pass-never-a-tsc-deadline.md | 32 ++++--------- ...s-what-modern-hardware-offers-for-speed.md | 45 +++++++----------- 8 files changed, 90 insertions(+), 127 deletions(-) delete mode 100644 issues/fsgsbase-is-measured-then-taken-or-rejected.md create mode 100644 issues/fsgsbase-is-taken.md diff --git a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md index fecbf9befb..8c9331d818 100644 --- a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md +++ b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md @@ -38,9 +38,11 @@ dispatch on the kernel clock; quantiles are bucket upper bounds, accurate to With 7 spinners every wake to a busy target waited out a whole quantum on both machines, while the slowest wake to a sleeping one took 2.16 µs (T14) and 3.17 µs (AMD bench), and the pair fell from about 181,600 (T14) and -205,200 (AMD bench) rounds a second with no spinners to 100. The kill condition fixed before the run, the change -dropped if the busy-target p99 is under about 100 µs with 7 and with 8 -spinners on both benches, did not fire: it reads 68 to 105 times that. +205,200 (AMD bench) rounds a second with no spinners to 100. A woken task +waiting in a busy CPU's mailbox while another CPU idles breaks the invariant +published work holds a scheduler to: ready threads run on available cores, +and no core stays idle while another is overloaded (Lozi et al., "The Linux +Scheduler: a Decade of Wasted Cores", EuroSys 2016). **Exit**: a metal row on the T14 and the AMD bench that wakes a task onto a CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs, and diff --git a/issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md b/issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md index a946a6e994..755958a129 100644 --- a/issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md +++ b/issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md @@ -12,8 +12,12 @@ The syscall stub parks the user's whole FP/SSE image on the kernel stack with thread is switched out, though the kernel is built soft-float and never touches those registers (`build/build.rs:511-522`). Saving at the ring transition is what lets a switch skip it; every syscall that returns to its own thread pays -it anyway. PR #859, open, makes the image an XSAVE area: under XSAVES the T14 -would park 2432 bytes, computed and not yet read from a boot (that PR's body). +it anyway. The SDM has a preemptive OS save the state as part of the switch +and restore it when the task resumes (325462-093, Vol. 3A §16.4), and AMD's +APM names FXSAVE and FXRSTOR as what saves and restores it during a context +switch (24593 rev. 3.42, Vol. 2 §2.5.12). PR #859, open, makes the image an +XSAVE area: under XSAVES the T14 would park 2432 bytes, computed and not yet +read from a boot (that PR's body). Every preemption disable on x86_64, `percpu::preempt_count_up` and `preempt_count_down` (`kernel/src/arch/x86_64/percpu.rs:791-802`), is a @@ -21,8 +25,8 @@ Every preemption disable on x86_64, `percpu::preempt_count_up` and `lock_dec_u32`, `percpu.rs:299-315`). The word is private to the CPU, and the reason given for the prefix, that an interrupt can split a plain `add`'s load and store, is not the hardware's: an interrupt is taken only on an -instruction boundary (SDM Vol. 3A §6.6). The syscall stub's own raise and -lower around the handler (`syscall.rs:63`, `:67`) are the same instructions, +instruction boundary (SDM 325462-093, Vol. 3A §7.6). The syscall stub's own +raise and lower around the handler (`syscall.rs:63`, `:67`) are the same instructions, but they are `issues/syscall-preemption-is-incidental.md`'s: that file's step of `issues/toyos-beats-linuxs-latency-on-the-t14.md` removes the raise, and this file does not. @@ -38,10 +42,8 @@ change made alone and both together: | T14 (2419 MHz) | 646 | 560 | 499 | 423 | 13.3 % | 22.8 % | 34.5 % | | AMD bench (1796 MHz) | 385 | 274 | 328 | 218 | 28.8 % | 14.8 % | 43.4 % | -The shares are (base − arm) / base, derived. The kill condition fixed before -the run, both changes dropped if both together save under about 15 % on both -benches, did not fire. The FP saving matches the pair's own cost timed in -Ring 3 (79 cycles on the T14, 111 on the AMD bench). +The shares are (base − arm) / base, derived. The FP saving matches the pair's +own cost timed in Ring 3 (79 cycles on the T14, 111 on the AMD bench). **Unexplained: where the `lock` saving comes from.** The no-`lock` arm stripped the prefix from the stub's raise and lower and from @@ -55,6 +57,5 @@ a null syscall takes. **Exit**: the `lock` saving split by site, posted; the save gone from a syscall that returns to its own thread, and the `lock` prefix gone from -`preempt_count_up` and `preempt_count_down`, or either kept with its refusal -reason at its site and its measured cost on both benches in the closing -commit; and `syscall_cost` read on both benches before and after. +`preempt_count_up` and `preempt_count_down`; and `syscall_cost` read on both +benches before and after. diff --git a/issues/fsgsbase-is-measured-then-taken-or-rejected.md b/issues/fsgsbase-is-measured-then-taken-or-rejected.md deleted file mode 100644 index 45c5adfcf1..0000000000 --- a/issues/fsgsbase-is-measured-then-taken-or-rejected.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -status: open -kind: track -opened: 2026-09-29 ---- - -# FSGSBASE is measured, then taken or rejected - -`CR4.FSGSBASE` is forbidden (`kernel/src/arch/x86_64/control_regs.rs:62-75`): -GS always holds the kernel's per-CPU base and the kernel runs no `swapgs`. -Taking it costs a `swapgs` entry and the swapgs barriers of -`issues/user-pointer-checks-have-no-spectre-v1-fence-and-smap-is-optional.md`. - -**Exit**: the T14's pipe round trips with and without it, a loss becoming a -`rejected` issue with both figures. Taken: on every proving machine and under -TCG, a `boot-actuators` arm enters vector 2 at CPL 0 while the user GS base is -live, between entry and its `swapgs` or after the exit path's, and the handler -finds the kernel's per-CPU data; `CR4_FORBIDDEN`, the `user-writable-gsbase` -kernel (`build/build.rs:1370`) and the test `gsbase_locked` are deleted. -**Mutation**: an IST entry that decides `swapgs` from the saved CS alone. -**Oracle**: Linux's `paranoid_entry`. diff --git a/issues/fsgsbase-is-taken.md b/issues/fsgsbase-is-taken.md new file mode 100644 index 0000000000..69f43d31a7 --- /dev/null +++ b/issues/fsgsbase-is-taken.md @@ -0,0 +1,25 @@ +--- +status: open +kind: track +opened: 2026-09-29 +--- + +# FSGSBASE is taken + +`CR4.FSGSBASE` is forbidden (`kernel/src/arch/x86_64/control_regs.rs:62-75`): +GS always holds the kernel's per-CPU base and the kernel runs no `swapgs`. +Taking it costs a `swapgs` entry and the swapgs barriers of +`issues/user-pointer-checks-have-no-spectre-v1-fence-and-smap-is-optional.md`. +Intel names what it gives, user programs' FS and GS bases without a system +call and a more efficient kernel context switch, and has it used in production +only with a complete OS implementation (Guidance for Enabling FSGSBASE, ID +767615). + +**Exit**: on every proving machine and under TCG, a `boot-actuators` arm +enters vector 2 at CPL 0 while the user GS base is live, between entry and its +`swapgs` or after the exit path's, and the handler finds the kernel's per-CPU +data; `CR4_FORBIDDEN`, the `user-writable-gsbase` kernel +(`build/build.rs:1370`) and the test `gsbase_locked` are deleted; the T14's +pipe round trips read with and without it. **Mutation**: an IST entry that +decides `swapgs` from the saved CS alone. **Oracle**: Linux's +`paranoid_entry`. diff --git a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md index 409d06eedc..08065a9dce 100644 --- a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md +++ b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md @@ -12,13 +12,19 @@ doorbell (`kernel/pure/sched/mailbox.rs:474-490`) and then sends a kick IPI (`kernel/src/arch/x86_64/apic.rs:256-260`), whose entry and handler run on the woken CPU before its `hlt` returns (`kernel/src/arch/x86_64/idt/kick.rs:3-9`). A CPU that waits in MWAIT on its doorbell's line (CPUID.01H:ECX[3]) is woken -by the store alone and takes no interrupt. The T14 enumerates it (`monitor`, -`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is -unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the +by the store alone and takes no interrupt. The SDM names the IPI a halted CPU +needs as a delay in serving new work and has an OS consider MONITOR/MWAIT in +place of `hlt` in its C1 idle loop (325462-093, Vol. 3A §11.10.4, +§11.10.6.4), and Intel's optimization manual names the C1 loop's +responsiveness as what that buys (248966-051, Vol. 1 §11.4). The T14 +enumerates MONITOR/MWAIT (`monitor`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); +whether the AMD bench does is unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the wait enters; this file is what ends it. The doorbell shares a line with its neighbours' published counters (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`), -so a monitor on it today would also wake on their writes. +so a monitor on it today would also wake on their writes; the SDM has +monitored data padded to the largest monitor line size CPUID.05H reports +(§11.10.5). **Measured** (`wt/toyos-scout-tlb` at `d4c2f8ebe`, `main`'s `177e72a9e` plus the scout; https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198). @@ -36,32 +42,8 @@ send to the woken task running, p50 in TSC ticks per direction: A CPU woken from MWAIT by the store skips all three segments but pays its own wake from the store, which no run has timed and which delivery's C1 exit stands in for here, so the share is an upper bound on what the doorbell -saves. Applied to that bound, the line fixed before the scout, the doorbell -dropped under 20 % of the wake, fires on the T14 and not on the AMD bench. +saves. -**Next: an MWAIT twin** beside a control arm, both timed by the same split in -the same session. The control arm is `main` plus the scout with the doorbell -on a line of its own, which is -`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`'s -change. The twin is the control arm with each idle CPU waiting in MWAIT on its -doorbell's line, hint C1 (EAX = 0), the state `hlt` enters, and the wake's -kick not sent. The two arms differ only in what ends the wait, which is what -the bound above covers. Its kill line, fixed here before it runs: the -doorbell is built ahead of `issues/idle-cpus-enter-the-c-states-cst-names.md` -only if, on a proving machine that enumerates MONITOR/MWAIT, the twin's whole -idle wake p50 is at least 20 % under the control arm's in both directions. -The T14's bound is under that line, so the twin runs on the AMD bench, and -reads CPUID.01H:ECX[3] there first; if the AMD bench does not enumerate it, no -proving machine can clear the line. Its clock control, fixed with the line: -each arm reads APERF and MPERF on CPUs 4 and 5 before and after the split, -and where the two arms' ratios of APERF's advance to MPERF's differ by more -than 1 % on either CPU, both arms' readings are void and decide nothing. - -**Exit**: the twin's reading, posted, and an idle CPU on every proving machine -that enumerates MONITOR/MWAIT woken by its doorbell's store with no IPI. Where -the twin clears the line, that is built ahead of the C-states. Where it does -not, this file stays open behind -`issues/idle-cpus-enter-the-c-states-cst-names.md`: the 20 % prices an idle -path beside `hlt`, which that track builds anyway, and once an idle CPU waits -in MWAIT there, putting its monitor on the doorbell's line and sending no -kick adds no second path. +**Exit**: an idle CPU on every proving machine that enumerates MONITOR/MWAIT +woken by its doorbell's store with no IPI, and the split above read again on +both benches after it. diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index f055905d90..c5bb9efae7 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -40,13 +40,14 @@ the parts it links): only these four; and the load event (D2H/04H) on this CPU also counts clean forwards. -The kill condition fixed before the run, these recorded as structure only if -both figures are under 1 %, did not fire on its HITM half, so the contention -is not dismissed. Nothing in the run attributes a miss to the `BITMAP` lock, -`IN_PASS`, `CpuHandle` or `Frontier`, and the AMD bench was not read. The -next measurement attributes the HITM count to the lines it lands on, on both +Intel's optimization manual has critical data and locks allocated apart, +aligned to no less than the false-sharing threshold, the CLFLUSH line size of +CPUID.01H:EBX[15:8] (248966-051, Vol. 1 §11.4.5, User/Source Coding Rule +17). Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, +`CpuHandle` or `Frontier`, and the AMD bench was not read. The next +measurement attributes the HITM count to the lines it lands on, on both benches. -**Exit**: that attribution, posted; and each of the four either written per -CPU on a line no other CPU writes, or kept with its refusal reason at its -site and its measured cost on both benches in the closing commit. +**Exit**: that attribution, posted, and read again after the change; and +each of the four either written per CPU on a line no other CPU writes, or +kept with its refusal reason at its site. diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md index d38c6b9321..4967f32f6e 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md @@ -20,9 +20,10 @@ https://github.com/ToyOSOrg/ToyOS/pull/861#issuecomment-6113257693). A timer fir calibration spins the boot for 10 ms (`apic.rs:315-320`, from `kernel/src/arch/x86_64/boot.rs:135`). TSC-deadline mode (CPUID.01H:ECX[24]) takes the absolute deadline in TSC units as one `IA32_TSC_DEADLINE` write, -with no calibration of a second timebase. The T14 enumerates it -(`tsc_deadline_timer`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether -the AMD bench does is unread. +with no calibration of a second timebase (SDM 325462-093, Vol. 3A §13.5.4.1). +The T14 enumerates it (`tsc_deadline_timer`, +`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is +unread. **Measured** (`wt/toyos-scout-tlb` at `d4c2f8ebe`, `main`'s `177e72a9e` plus the scout; https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198), @@ -47,23 +48,8 @@ the deadline into a span, and a deadline written as an absolute is an upper bound on what deadline mode saves, which still writes once to arm and once to stop; on the AMD bench both sit at its TSC's floor. -**Next: a TSC-deadline twin on the T14**, `main` plus the scout with the -one-shot armed and stopped through `IA32_TSC_DEADLINE`, timed by the same -split beside `main` in the same session, with the re-arm segment opening at -`set_timer` (`kernel/src/hw.rs:49`) in both arms. Its kill line, fixed here -before it runs: deadline mode is not worth building for speed if what it -takes off a pass, `main`'s re-arm p50 less the twin's, is under 5 % of -`main`'s pick plus switch in either direction (221 and 216 ticks on the -scout's figures). Its clock control, fixed with the line: each arm reads -APERF and MPERF on CPUs 4 and 5 before and after the split, and where the two -arms' ratios of APERF's advance to MPERF's differ by more than 1 % on either -CPU, both arms' readings are void and decide nothing. Where the line fires, -this file becomes `rejected` with both readings, and the boot's 10 ms -calibration spin stays the count's price. - -**Exit**: the twin's reading, posted, and the AMD bench's CPUID.01H:ECX[24] -read. Where the twin clears the line, a deadline is armed on every CPU that -enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the absolute -value, and the T14's 1 ms timer lateness of -`issues/no-program-measures-toyos-against-linux-on-one-machine.md` is no worse -than the count's. +**Exit**: the AMD bench's CPUID.01H:ECX[24] read; a deadline armed on every +CPU that enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the +absolute value; and the split above, and the T14's 1 ms timer lateness of +`issues/no-program-measures-toyos-against-linux-on-one-machine.md`, read with +the count and with the deadline. diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index ba15554fe3..6adb4f581e 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -18,14 +18,17 @@ each that has its feature. A figure is the T14's, and the AMD bench's too where the stage's path differs there, since the EPYC guests are shared CI runners: the median of 11 runs with its spread, taken by the program of `issues/no-program-measures-toyos-against-linux-on-one-machine.md`. -A stage is done at or past that program's Linux figure, or with the gap filed -as a defect carrying both; a feature Linux has no counterpart for lands only -if it beats ToyOS without it. What no proving machine is known to offer is -`issues/features-no-proving-machine-is-known-to-offer.md`. +A stage cites what its hardware's manufacturers document and the published +state of the art. Where they back it, its figure is reported and never blocks +it: a slowdown, or a gap to that program's Linux figure, is filed as a defect +carrying both figures and investigated. A stage they do not cover is reviewed +on its correctness and simplicity, its figure reported and deciding nothing by +itself unless speed is the stage's only purpose. What no proving machine is +known to offer is `issues/features-no-proving-machine-is-known-to-offer.md`. **Measurement first.** Each design round starts from a posted measurement, -never a guess, judged by a kill condition fixed before the run. Three scouts -on `main`'s `177e72a9e` have read both benches, or the T14 against Ubuntu: +never a guess. Three scouts on `main`'s `177e72a9e` have read both benches, +or the T14 against Ubuntu: https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119438569 (the syscall and wake latency), https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198 (the TLB @@ -33,25 +36,18 @@ and the idle wake) and https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119513916 (the clock and contention). -**Measured, and worth building.** Each issue carries its reading. +**Measured.** Each issue carries its reading. - The syscall's FP save and the `lock` prefixes it takes are 34.5 % of a null syscall on the T14 and 43.4 % on the AMD bench: `issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md`. -- A wake to a busy CPU waits out a whole 10 ms quantum on both benches, p99 - 68 to 105 times a 100 µs bar: +- A wake to a busy CPU waits out a whole 10 ms quantum on both benches: `issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md`. - -**Bounded, and judged next by a twin** whose kill line its issue fixes -before it runs. - - An MWAIT doorbell: the IPI's send and delivery and the kick's handler are at most 14.7 to 14.9 % of an idle wake on the T14 and 22.4 to 22.6 % on the - AMD bench; next, an MWAIT twin on the AMD bench - (`issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`). + AMD bench (`issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`). - A TSC deadline: the re-arm is at most 11.7 to 12.1 % of a pass's pick and - switch on the T14, and at the AMD bench's TSC floor; next, a TSC-deadline - twin on the T14 + switch on the T14, and at the AMD bench's TSC floor (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`). **Moot.** PCID on the AMD bench: its CPUs read `pcid=0 invpcid=0`, so there @@ -86,21 +82,12 @@ is nothing to enable. (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`). - **The wake storm's per-waiter gap** between the benches; next, its split (`issues/a-wake-storm-costs-the-t14-twice-the-amd-benchs-time-per-waiter-and-nothing-says-why.md`). -- **Global pages.** CR4.PGE is clear on both benches, so where there is no - PCID every process switch drops the kernel's translations. On the AMD - bench, which has none, a user ping-pong takes 80.7 full page walks per round trip with no - user working set and 88.6 at four pages, which prices at about 7 % of the - round trip (derived, user and kernel walks together, so a ceiling on what - global pages save). On the T14, PCID saves 3.4 % of the same round trip at - four pages (`wt/toyos-scout-tlb` at `d4c2f8ebe`). The decision and its - sequencing are `issues/page-global-is-a-decision-nobody-has-made.md`'s; - next, a PGE twin on the AMD bench. The CPU cap is PR #865's (`wt/toyos-cpus`), a design round of its own. **Exit**: every issue below is closed. They are listed in the order they are -taken: the instruments, then what is measured and worth building, then what -waits on its next measurement, then the design rounds not yet measured. +taken: the instruments, then what is measured, then the design rounds not +yet measured. `issues/user-programs-use-avx-under-xsave.md` is PR #859's, built ahead of that order. @@ -117,6 +104,6 @@ that order. - `issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md` - `issues/the-direct-map-uses-1-gib-leaves-where-the-cpu-has-them.md` - `issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md` -- `issues/fsgsbase-is-measured-then-taken-or-rejected.md` +- `issues/fsgsbase-is-taken.md` - `issues/idle-cpus-enter-the-c-states-cst-names.md` - `issues/diskserver-drives-nvme-apst.md` From 533acff9b5e54950752e85e54325a8c5c31bc67f Mon Sep 17 00:00:00 2001 From: japabu Date: Mon, 12 Oct 2026 17:16:22 +0200 Subject: [PATCH 2/7] issues: the TSC deadline is decided by its twin, and each stage cites what backs it on both makers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 1's review of #887 and the orchestrator's rulings on it. - The TSC-deadline stage gets its twin and kill line back. The SDM documents the mode and recommends it over the count nowhere, and AMD's APM Vol. 3 (24594 rev. 3.36, §E.3.2) has CPUID.01H:ECX[24] reserved, so an AMD CPU keeps the count and its calibration beside any deadline: nothing published covers the choice and speed is its only purpose, the owner's exception, where its reading decides it. The AMD bench's CPUID read leaves the exit, which the APM answers. - The track says "back", as the role prompts do since #885, and "nothing published covers" for the case the owner ruled on. - The busy-wake defect cites Lozi et al.'s §3.3, the Overload-on-Wakeup bug and its fix, with the fix's power condition, in place of the abstract's invariant. Its 8-spinner arm, where no core idles, is one nothing published covers and whose purpose is latency, so its 100 us bound decides that arm's design. - The MWAIT and per-CPU-line stages record AMD's position as read: APM Vol. 3's MONITOR and MWAIT document the wait on another processor's store and recommend no idle loop; APM Vol. 2 and the family 17h models 30h-and-greater optimization guide (56305 rev. 3.02) say nothing on MWAIT or false sharing. - FSGSBASE is backed: Intel's guidance names the complete OS implementation it asks for, and Linux 5.9 meets it and enables it on every CPU that has it. The issue cites both and AMD's APM, states the condition ToyOS's implementation meets, and its exit adds a thread's bases surviving a switch. - APST takes its design from the NVMe base specification 2.1 and the drive's own id-ctrl, not from Linux's policy, and its oracle is the specification's rules. Linux's quirks for drives whose APST failed under it go: the T14's drive's id-ctrl is unread. - The C-states file is left to #870, which takes it up. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy --- ...s-next-pass-and-no-idle-cpu-can-take-it.md | 16 ++++++--- issues/diskserver-drives-nvme-apst.md | 17 ++++++--- issues/fsgsbase-is-taken.md | 27 ++++++++++---- ...-so-every-wake-of-an-idle-cpu-is-an-ipi.md | 11 ++++-- ...pmm-state-shares-lines-every-cpu-writes.md | 4 ++- ...e-armed-every-pass-never-a-tsc-deadline.md | 35 +++++++++++++++---- ...s-what-modern-hardware-offers-for-speed.md | 19 +++++----- 7 files changed, 93 insertions(+), 36 deletions(-) diff --git a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md index 8c9331d818..67f7d94779 100644 --- a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md +++ b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md @@ -38,11 +38,17 @@ dispatch on the kernel clock; quantiles are bucket upper bounds, accurate to With 7 spinners every wake to a busy target waited out a whole quantum on both machines, while the slowest wake to a sleeping one took 2.16 µs (T14) and 3.17 µs (AMD bench), and the pair fell from about 181,600 (T14) and -205,200 (AMD bench) rounds a second with no spinners to 100. A woken task -waiting in a busy CPU's mailbox while another CPU idles breaks the invariant -published work holds a scheduler to: ready threads run on available cores, -and no core stays idle while another is overloaded (Lozi et al., "The Linux -Scheduler: a Decade of Wasted Cores", EuroSys 2016). +205,200 (AMD bench) rounds a second with no spinners to 100. + +With 7 spinners this is the Overload-on-Wakeup bug of Lozi et al., "The Linux +Scheduler: a Decade of Wasted Cores" (EuroSys 2016, §3.3): a thread that was +asleep wakes on an overloaded core while other cores are idle. Its fix +wakes the thread on the core it last ran on if that core is idle, and +otherwise on the core idle longest, where one is; the paper applies it only +where the power policy keeps cores out of low-power states, since waking a +core from one costs power. With 8 spinners no core idles, so §3.3 does not +reach that arm and nothing published covers it; latency is its whole purpose, +so under the track's rule the 100 µs bound below decides its design. **Exit**: a metal row on the T14 and the AMD bench that wakes a task onto a CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs, and diff --git a/issues/diskserver-drives-nvme-apst.md b/issues/diskserver-drives-nvme-apst.md index fbee731b3c..30d86a180a 100644 --- a/issues/diskserver-drives-nvme-apst.md +++ b/issues/diskserver-drives-nvme-apst.md @@ -7,12 +7,19 @@ opened: 2026-09-29 # blockd drives NVMe APST After blockd drives the T14's drive, the blockd step of -`issues/the-kernel-is-small-interrupts-post-and-threads-wait.md`. As -Linux at `Ubuntu-6.8.0-142.142`, APST stays within -`default_ps_max_latency_us` and honours `NVME_QUIRK_NO_APST` and -`NVME_QUIRK_NO_DEEPEST_PS` (`drivers/nvme/host/core.c:63,2495-2598`). +`issues/the-kernel-is-small-interrupts-post-and-threads-wait.md`. The NVMe +base specification defines APST: enabled through Feature 0Ch, the controller +moves itself, once idle for the time a power state's table entry names, to +the non-operational state that entry names, one with no higher idle power; +and the drive reports each state's worst entry and exit latency, a +transition costing the old state's exit latency plus the new one's entry +latency (NVM Express Base Specification 2.1, §5.1.25.1.6, §8.1.17 and +§8.1.17.2). blockd builds the table from the power state descriptors of the +drive's own `id-ctrl`. The specification bounds no latency: blockd's bound is +its design round's, chosen on its merits. **Exit**: host tests of the APST table; on the T14, the one proving machine with a physical NVMe drive, the block figures and the idle minute with the drive idle. **Mutation**: a table that admits a state past the bound. -**Oracle**: Linux's `nvme_configure_apst` over the T14's `id-ctrl`. +**Oracle**: the specification's rules for the table over the T14's +`id-ctrl`. diff --git a/issues/fsgsbase-is-taken.md b/issues/fsgsbase-is-taken.md index 69f43d31a7..5bb3e8b9af 100644 --- a/issues/fsgsbase-is-taken.md +++ b/issues/fsgsbase-is-taken.md @@ -12,14 +12,27 @@ Taking it costs a `swapgs` entry and the swapgs barriers of `issues/user-pointer-checks-have-no-spectre-v1-fence-and-smap-is-optional.md`. Intel names what it gives, user programs' FS and GS bases without a system call and a more efficient kernel context switch, and has it used in production -only with a complete OS implementation (Guidance for Enabling FSGSBASE, ID -767615). +only with a complete OS implementation: one whose context switch manages the +FS and GS bases apart from their selectors and whose ring transitions are +insensitive to the GS base, as Linux's is from 5.9 on (Guidance for Enabling +FSGSBASE, ID 767615). Linux enables it on every CPU that enumerates it +(`arch/x86/kernel/cpu/common.c:1515-1519` at `v5.9`); its switch saves each +thread's bases as the thread left them (`process_64.c:248-262`), and its +paranoid entry assumes nothing of the GS base it finds and loads the CPU's own +from a kernel table (`entry_64.S:823-848`). AMD documents the four +instructions, enabled by CR4.FSGSBASE at every privilege level, and recommends +nothing on their use (APM 24594 rev. 3.36, Vol. 3, RDFSBASE and WRFSBASE). +ToyOS takes it under Intel's condition: its switch saves and restores each +thread's FS and GS bases as the thread left them, and its ring transitions are +insensitive to the GS base they find. Its figure is reported and decides +nothing. **Exit**: on every proving machine and under TCG, a `boot-actuators` arm enters vector 2 at CPL 0 while the user GS base is live, between entry and its `swapgs` or after the exit path's, and the handler finds the kernel's per-CPU -data; `CR4_FORBIDDEN`, the `user-writable-gsbase` kernel -(`build/build.rs:1370`) and the test `gsbase_locked` are deleted; the T14's -pipe round trips read with and without it. **Mutation**: an IST entry that -decides `swapgs` from the saved CS alone. **Oracle**: Linux's -`paranoid_entry`. +data; a thread's FS and GS bases, written with WRFSBASE and WRGSBASE, read +back unchanged after its CPU has run another thread; `CR4_FORBIDDEN`, the +`user-writable-gsbase` kernel (`build/build.rs:1370`) and the test +`gsbase_locked` are deleted; the T14's pipe round trips read with and without +it. **Mutation**: an IST entry that decides `swapgs` from the saved CS alone. +**Oracle**: Linux's `paranoid_entry`. diff --git a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md index 08065a9dce..b5932f97c0 100644 --- a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md +++ b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md @@ -16,9 +16,14 @@ by the store alone and takes no interrupt. The SDM names the IPI a halted CPU needs as a delay in serving new work and has an OS consider MONITOR/MWAIT in place of `hlt` in its C1 idle loop (325462-093, Vol. 3A §11.10.4, §11.10.6.4), and Intel's optimization manual names the C1 loop's -responsiveness as what that buys (248966-051, Vol. 1 §11.4). The T14 -enumerates MONITOR/MWAIT (`monitor`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); -whether the AMD bench does is unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the +responsiveness as what that buys (248966-051, Vol. 1 §11.4). AMD documents +MWAIT as a wait for another processor's store to the range MONITOR set up, +sized by CPUID 0000_0005h, and recommends it for no idle loop (APM 24594 rev. +3.36, Vol. 3, MONITOR and MWAIT); neither its APM Vol. 2 (24593 rev. 3.42) nor +its optimization guide for family 17h models 30h and greater (56305 rev. +3.02) speaks to it. The T14 enumerates MONITOR/MWAIT (`monitor`, +`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is +unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the wait enters; this file is what ends it. The doorbell shares a line with its neighbours' published counters (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`), diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index c5bb9efae7..f90e0d38e8 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -43,7 +43,9 @@ the parts it links): Intel's optimization manual has critical data and locks allocated apart, aligned to no less than the false-sharing threshold, the CLFLUSH line size of CPUID.01H:EBX[15:8] (248966-051, Vol. 1 §11.4.5, User/Source Coding Rule -17). Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, +17). AMD's APM Vol. 2 (24593 rev. 3.42) and its optimization guide for +family 17h models 30h and greater (56305 rev. 3.02) say nothing on false +sharing. Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, `CpuHandle` or `Frontier`, and the AMD bench was not read. The next measurement attributes the HITM count to the lines it lands on, on both benches. diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md index 4967f32f6e..a528102026 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md @@ -22,8 +22,15 @@ calibration spins the boot for 10 ms (`apic.rs:315-320`, from takes the absolute deadline in TSC units as one `IA32_TSC_DEADLINE` write, with no calibration of a second timebase (SDM 325462-093, Vol. 3A §13.5.4.1). The T14 enumerates it (`tsc_deadline_timer`, -`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is -unread. +`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`). AMD documents no such mode: its +CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, +§E.3.2), so the AMD bench keeps the count, its calibration and its spin. + +**Its speed reading decides it.** The SDM documents the mode and recommends it +over the count nowhere, and AMD documents none, so nothing published covers +the choice. Nothing but speed is its purpose: a deadline removes no path, +since a CPU without one keeps the count. Under the track's rule its figure +decides it. **Measured** (`wt/toyos-scout-tlb` at `d4c2f8ebe`, `main`'s `177e72a9e` plus the scout; https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198), @@ -48,8 +55,22 @@ the deadline into a span, and a deadline written as an absolute is an upper bound on what deadline mode saves, which still writes once to arm and once to stop; on the AMD bench both sit at its TSC's floor. -**Exit**: the AMD bench's CPUID.01H:ECX[24] read; a deadline armed on every -CPU that enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the -absolute value; and the split above, and the T14's 1 ms timer lateness of -`issues/no-program-measures-toyos-against-linux-on-one-machine.md`, read with -the count and with the deadline. +**Next: a TSC-deadline twin on the T14**, `main` plus the scout with the +one-shot armed and stopped through `IA32_TSC_DEADLINE`, timed by the same +split beside `main` in the same session, with the re-arm segment opening at +`set_timer` (`kernel/src/hw.rs:49`) in both arms. Its kill line, fixed here +before it runs: deadline mode is not worth building for speed if what it +takes off a pass, `main`'s re-arm p50 less the twin's, is under 5 % of +`main`'s pick plus switch in either direction (221 and 216 ticks on the +scout's figures). Its clock control, fixed with the line: each arm reads +APERF and MPERF on CPUs 4 and 5 before and after the split, and where the two +arms' ratios of APERF's advance to MPERF's differ by more than 1 % on either +CPU, both arms' readings are void and decide nothing. Where the line fires, +this file becomes `rejected` with both readings, and the boot's 10 ms +calibration spin stays the count's price. + +**Exit**: the twin's reading, posted. Where the twin clears the line, a +deadline is armed on every CPU that enumerates TSC-deadline as one +`IA32_TSC_DEADLINE` write of the absolute value, and the T14's 1 ms timer +lateness of `issues/no-program-measures-toyos-against-linux-on-one-machine.md` +is no worse than the count's. diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index 6adb4f581e..71500302ed 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -18,13 +18,14 @@ each that has its feature. A figure is the T14's, and the AMD bench's too where the stage's path differs there, since the EPYC guests are shared CI runners: the median of 11 runs with its spread, taken by the program of `issues/no-program-measures-toyos-against-linux-on-one-machine.md`. -A stage cites what its hardware's manufacturers document and the published -state of the art. Where they back it, its figure is reported and never blocks -it: a slowdown, or a gap to that program's Linux figure, is filed as a defect -carrying both figures and investigated. A stage they do not cover is reviewed -on its correctness and simplicity, its figure reported and deciding nothing by -itself unless speed is the stage's only purpose. What no proving machine is -known to offer is `issues/features-no-proving-machine-is-known-to-offer.md`. +A stage that its hardware's manufacturers or the published state of the art +back follows and cites them, and its figure is reported and never blocks it: a +slowdown, or a gap to that program's Linux figure, is filed as a defect +carrying both figures and investigated. A stage nothing published covers is +reviewed on its correctness and simplicity, its figure reported and deciding +nothing by itself unless speed is the stage's only purpose. What no proving +machine is known to offer is +`issues/features-no-proving-machine-is-known-to-offer.md`. **Measurement first.** Each design round starts from a posted measurement, never a guess. Three scouts on `main`'s `177e72a9e` have read both benches, @@ -47,7 +48,9 @@ and contention). at most 14.7 to 14.9 % of an idle wake on the T14 and 22.4 to 22.6 % on the AMD bench (`issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`). - A TSC deadline: the re-arm is at most 11.7 to 12.1 % of a pass's pick and - switch on the T14, and at the AMD bench's TSC floor + switch on the T14, and at the AMD bench's TSC floor; next, a TSC-deadline + twin on the T14, whose reading decides the stage, since speed is its only + purpose (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`). **Moot.** PCID on the AMD bench: its CPUs read `pcid=0 invpcid=0`, so there From 2d7b881b026821b8798b6d6ae9ecd2725f5fcb4d Mon Sep 17 00:00:00 2001 From: japabu Date: Mon, 12 Oct 2026 19:58:09 +0200 Subject: [PATCH 3/7] issues: the TSC deadline, the busy wake's 8-spinner arm and MWAIT on AMD follow what Linux backs The orchestrator's corrected rulings on #887 (https://github.com/ToyOSOrg/ToyOS/pull/887#issuecomment-6127473561), after round 2's review read Linux where rounds 1 and 2 read only the vendors' manuals: - The TSC deadline is backed. Linux has armed its one-shot through IA32_TSC_DEADLINE since v3.8 (279f1461432c, Intel's Suresh Siddha: no APIC calibration at boot, no uncached re-arm under xAPIC), and since v7.1 hands the deadline over in TSC cycles without reading the TSC (f246ec3478cf). The kill line and the rejected outcome go; the twin stays, its reading reported, its clock control voiding a reading to be taken again rather than a decision. - The busy wake's 8-spinner arm is backed by wakeup preemption: CFS's check_preempt_curr_fair at v2.6.23, by virtual deadline since v6.6 (147f3efaa241). The 100 us bound is the exit's test, not the design's judge. - MWAIT idle is backed on AMD from family 17h by AMD's own commit aebef63cf7ff (v6.0). The issue says per vendor what backs it. - Round 2's NOTE 2: the per-CPU lines issue cites Linux's DEFINE_PER_CPU_SHARED_ALIGNED runqueues and ____cacheline_aligned_in_smp, and AMD's CLFLUSH line-size field and 56305's 64-byte line. Round 2's NOTE 3, the lock half's Linux citation, has no file to go in: #871 closed every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked in the merge before this commit. Each commit's text, and the release that first carries it, is quoted on ToyOSOrg/evidence pr-887's sources.md. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy --- ...s-next-pass-and-no-idle-cpu-can-take-it.md | 11 +++-- ...-so-every-wake-of-an-idle-cpu-is-an-ipi.md | 18 +++++--- ...pmm-state-shares-lines-every-cpu-writes.md | 15 +++++-- ...e-armed-every-pass-never-a-tsc-deadline.md | 42 ++++++++++--------- ...s-what-modern-hardware-offers-for-speed.md | 3 +- 5 files changed, 55 insertions(+), 34 deletions(-) diff --git a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md index 67f7d94779..b0a65c603b 100644 --- a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md +++ b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md @@ -46,9 +46,14 @@ asleep wakes on an overloaded core while other cores are idle. Its fix wakes the thread on the core it last ran on if that core is idle, and otherwise on the core idle longest, where one is; the paper applies it only where the power policy keeps cores out of low-power states, since waking a -core from one costs power. With 8 spinners no core idles, so §3.3 does not -reach that arm and nothing published covers it; latency is its whole purpose, -so under the track's rule the 100 µs bound below decides its design. +core from one costs power. With 8 spinners no core idles, and a wake onto a +CPU running a task is what wakeup preemption decides: Linux's fair class has +preempted the running task for a woken one since CFS (`check_preempt_curr_fair`, +`kernel/sched_fair.c` at v2.6.23), and since v6.6 by virtual deadline +(147f3efaa241, "wakeup preemption is driven by the deadline"), the +vruntime/lag family of ToyOS's fair band (`kernel/pure/sched/fair.rs:1-3`). +That arm follows it. The 100 µs bound below is the exit's test that the stage +works, not the judge of its design. **Exit**: a metal row on the T14 and the AMD bench that wakes a task onto a CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs, and diff --git a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md index b5932f97c0..ea6a701344 100644 --- a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md +++ b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md @@ -16,12 +16,18 @@ by the store alone and takes no interrupt. The SDM names the IPI a halted CPU needs as a delay in serving new work and has an OS consider MONITOR/MWAIT in place of `hlt` in its C1 idle loop (325462-093, Vol. 3A §11.10.4, §11.10.6.4), and Intel's optimization manual names the C1 loop's -responsiveness as what that buys (248966-051, Vol. 1 §11.4). AMD documents -MWAIT as a wait for another processor's store to the range MONITOR set up, -sized by CPUID 0000_0005h, and recommends it for no idle loop (APM 24594 rev. -3.36, Vol. 3, MONITOR and MWAIT); neither its APM Vol. 2 (24593 rev. 3.42) nor -its optimization guide for family 17h models 30h and greater (56305 rev. -3.02) speaks to it. The T14 enumerates MONITOR/MWAIT (`monitor`, +responsiveness as what that buys (248966-051, Vol. 1 §11.4). AMD's APM +documents MWAIT as a wait for another processor's store to the range MONITOR +set up, sized by CPUID 0000_0005h (24594 rev. 3.36, Vol. 3, MONITOR and +MWAIT), and neither it, its Vol. 2 (24593 rev. 3.42) nor its optimization +guide for family 17h models 30h and greater (56305 rev. 3.02) names an idle +loop. AMD's own Linux commit aebef63cf7ff (v6.0, by AMD's Wyes Karny) does: +"AMD Family 17h onwards supports MWAIT as well as MWAIT C1. And it is +preferable to use MWAIT as the default idle state on these systems, as it has +lower exit latencies", and at v7.2 Linux's `prefer_mwait_c1_over_halt` checks +no vendor (`arch/x86/kernel/process.c`). So the stage is backed on +both benches: on Intel by §11.10.4 and §11.10.6.4, and on AMD from family 17h, +the AMD bench's, by aebef63cf7ff. The T14 enumerates MONITOR/MWAIT (`monitor`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the wait enters; this file is what ends it. The doorbell shares a line with its diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index f90e0d38e8..bbd1ebd60a 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -43,9 +43,18 @@ the parts it links): Intel's optimization manual has critical data and locks allocated apart, aligned to no less than the false-sharing threshold, the CLFLUSH line size of CPUID.01H:EBX[15:8] (248966-051, Vol. 1 §11.4.5, User/Source Coding Rule -17). AMD's APM Vol. 2 (24593 rev. 3.42) and its optimization guide for -family 17h models 30h and greater (56305 rev. 3.02) say nothing on false -sharing. Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, +17). Linux keeps each CPU's run queue on lines of its own, +`DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues)` (`kernel/sched/core.c` +at v7.2), which aligns it by `____cacheline_aligned_in_smp` to +`SMP_CACHE_BYTES`, x86's L1 line (`include/linux/percpu-defs.h`, +`include/linux/cache.h`, `include/vdso/cache.h`, +`arch/x86/include/asm/cache.h`). AMD documents the line through the same CPUID +field (APM 24594 rev. 3.36, Vol. 3, CLFLUSH) and gives it as 64 bytes on +family 17h models 30h and greater (56305 rev. 3.02, §2.6.1, §2.6.2), and none +of APM Vol. 2 (24593 rev. 3.42), Vol. 3 and 56305 says anything on false +sharing. So the stage is backed on both benches. + +Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, `CpuHandle` or `Frontier`, and the AMD bench was not read. The next measurement attributes the HITM count to the lines it lands on, on both benches. diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md index a528102026..a728607a68 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md @@ -26,11 +26,18 @@ The T14 enumerates it (`tsc_deadline_timer`, CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, §E.3.2), so the AMD bench keeps the count, its calibration and its spin. -**Its speed reading decides it.** The SDM documents the mode and recommends it -over the count nowhere, and AMD documents none, so nothing published covers -the choice. Nothing but speed is its purpose: a deadline removes no path, -since a CPU without one keeps the count. Under the track's rule its figure -decides it. +**The published state of the art backs it.** Linux has armed its one-shot +through `IA32_TSC_DEADLINE` on every CPU that enumerates the mode since v3.8: +279f1461432c, by Intel's Suresh Siddha, "enables us to skip the APIC +calibration during boot. Also, in xapic mode, this enables us to skip the +uncached apic access to re-arm the APIC timer". At v7.2 its +`calibrate_APIC_clock` returns at once where the mode is, and +`setup_APIC_timer` registers `lapic-deadline` there +(`arch/x86/kernel/apic/apic.c`); its errata table, `deadline_match`, does not +name the T14's family 6 model 140. Since v7.1 the deadline is handed over in +TSC cycles "without reading the TSC at all" (f246ec3478cf), the absolute +write this stage takes. The stage follows them, and its figure is reported +and decides nothing. **Measured** (`wt/toyos-scout-tlb` at `d4c2f8ebe`, `main`'s `177e72a9e` plus the scout; https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198), @@ -58,19 +65,14 @@ and once to stop; on the AMD bench both sit at its TSC's floor. **Next: a TSC-deadline twin on the T14**, `main` plus the scout with the one-shot armed and stopped through `IA32_TSC_DEADLINE`, timed by the same split beside `main` in the same session, with the re-arm segment opening at -`set_timer` (`kernel/src/hw.rs:49`) in both arms. Its kill line, fixed here -before it runs: deadline mode is not worth building for speed if what it -takes off a pass, `main`'s re-arm p50 less the twin's, is under 5 % of -`main`'s pick plus switch in either direction (221 and 216 ticks on the -scout's figures). Its clock control, fixed with the line: each arm reads -APERF and MPERF on CPUs 4 and 5 before and after the split, and where the two -arms' ratios of APERF's advance to MPERF's differ by more than 1 % on either -CPU, both arms' readings are void and decide nothing. Where the line fires, -this file becomes `rejected` with both readings, and the boot's 10 ms -calibration spin stays the count's price. +`set_timer` (`kernel/src/hw.rs:49`) in both arms, its reading reported. Its +clock control: each arm reads APERF and MPERF on CPUs 4 and 5 before and +after the split, and where the two arms' ratios of APERF's advance to MPERF's +differ by more than 1 % on either CPU, both arms' readings are void and are +taken again. -**Exit**: the twin's reading, posted. Where the twin clears the line, a -deadline is armed on every CPU that enumerates TSC-deadline as one -`IA32_TSC_DEADLINE` write of the absolute value, and the T14's 1 ms timer -lateness of `issues/no-program-measures-toyos-against-linux-on-one-machine.md` -is no worse than the count's. +**Exit**: the twin's reading, posted; a deadline armed on every CPU that +enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the absolute +value; and the T14's 1 ms timer lateness of +`issues/no-program-measures-toyos-against-linux-on-one-machine.md` no worse +than the count's. diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index 566642faed..4b3a0fd184 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -46,8 +46,7 @@ and contention). AMD bench (`issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`). - A TSC deadline: the re-arm is at most 11.7 to 12.1 % of a pass's pick and switch on the T14, and at the AMD bench's TSC floor; next, a TSC-deadline - twin on the T14, whose reading decides the stage, since speed is its only - purpose + twin on the T14, its reading reported (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`). **Moot.** PCID on the AMD bench: its CPUs read `pcid=0 invpcid=0`, so there From eded3b73a6d2a3994b95e8143855c4397c700089 Mon Sep 17 00:00:00 2001 From: japabu Date: Mon, 12 Oct 2026 20:59:31 +0200 Subject: [PATCH 4/7] issues: the timer's lateness is reported, the 8-spinner arm follows EEVDF whole, MONITOR's erratum keeps the kick Round 3's review (#887, 6130446811) and the orchestrator's rulings on it (6130566783): - The TSC deadline: the twin and its clock control go. The exit reads the split again on the T14 after the change, as the MWAIT stage's does, and reports the 1 ms timer lateness with the count and with the deadline, where round 3 held it "no worse than the count's". - The busy wake's 8-spinner arm follows EEVDF whole: v6.6 curbs wakeup preemption too (63304558ba5d, RUN_TO_PARITY, on by default at v7.2), so that arm's test is a woken task running within one protected slice, 1 ms (#874's PROTECTED_RUN_NS), and its p99 against 100 us is reported. The 7-spinner arm keeps the 100 us bound as its test. - MWAIT idle: a CPU whose MONITOR loses wakes keeps the kick, as Linux keeps it. c9a4b55431e5 (v6.13, Intel's Len Brown) adds Lunar Lake to Goldmont in X86_BUG_MONITOR; at v7.2 such a CPU's MWAIT idle sets no polling flag, so each wake of it is an IPI. - APST: blockd honours NVME_QUIRK_NO_APST and NVME_QUIRK_NO_DEEPEST_PS, the drives whose APST failed, which round 2 had dropped with Linux's latency policy. No reading in the tree names the T14's drive, so whether it is one of them is unread. - fsgsbase-is-taken: CR4_FORBIDDEN is at control_regs.rs:149-153, and gsbase_locked and its user-writable-gsbase kernel, which 29e100612 cut, are named as stage C of the guest-suite track's. The track's TSC-deadline bullet drops its twin. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy --- ...s-next-pass-and-no-idle-cpu-can-take-it.md | 15 +++++++++---- issues/diskserver-drives-nvme-apst.md | 21 +++++++++++++------ issues/fsgsbase-is-taken.md | 13 ++++++------ ...-so-every-wake-of-an-idle-cpu-is-an-ipi.md | 13 +++++++++--- ...e-armed-every-pass-never-a-tsc-deadline.md | 19 +++++------------ ...s-what-modern-hardware-offers-for-speed.md | 3 +-- 6 files changed, 49 insertions(+), 35 deletions(-) diff --git a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md index b0a65c603b..d22050c616 100644 --- a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md +++ b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md @@ -52,9 +52,16 @@ preempted the running task for a woken one since CFS (`check_preempt_curr_fair`, `kernel/sched_fair.c` at v2.6.23), and since v6.6 by virtual deadline (147f3efaa241, "wakeup preemption is driven by the deadline"), the vruntime/lag family of ToyOS's fair band (`kernel/pure/sched/fair.rs:1-3`). -That arm follows it. The 100 µs bound below is the exit's test that the stage -works, not the judge of its design. +The same release curbs it (63304558ba5d): under `RUN_TO_PARITY`, on by default +at v7.2, a wake does not preempt "until the current task has either matched +the 0-lag point or until is has exhausted it's slice" +(`kernel/sched/features.h`). That arm follows EEVDF whole, the curb included, +so its test is that a woken task runs within one protected slice, 1 ms +(`PROTECTED_RUN_NS`, https://github.com/ToyOSOrg/ToyOS/pull/874), and its p99 +against 100 µs is reported. With 7 spinners the 100 µs bound is the exit's +test that the stage works, not the judge of its design. **Exit**: a metal row on the T14 and the AMD bench that wakes a task onto a -CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs, and -holds its post-to-dispatch p99 under 100 µs. +CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs: +with 7 it holds the post-to-dispatch p99 under 100 µs, and with 8 it holds +that a woken task runs within one protected slice and reports its p99. diff --git a/issues/diskserver-drives-nvme-apst.md b/issues/diskserver-drives-nvme-apst.md index 30d86a180a..b6bf085e97 100644 --- a/issues/diskserver-drives-nvme-apst.md +++ b/issues/diskserver-drives-nvme-apst.md @@ -16,10 +16,19 @@ transition costing the old state's exit latency plus the new one's entry latency (NVM Express Base Specification 2.1, §5.1.25.1.6, §8.1.17 and §8.1.17.2). blockd builds the table from the power state descriptors of the drive's own `id-ctrl`. The specification bounds no latency: blockd's bound is -its design round's, chosen on its merits. +its design round's, chosen on its merits. Drives whose APST failed are +recorded, and blockd honours the record: no APST on a drive marked +`NVME_QUIRK_NO_APST`, and no transition to its deepest state on one marked +`NVME_QUIRK_NO_DEEPEST_PS`, by the drive, firmware or machine Linux lists for +each (`drivers/nvme/host/core.c`'s `core_quirks`, and `pci.c`'s +`nvme_id_table` and `check_vendor_combination_bug`, at v7.2). Whether the +T14's drive is one of them is unread: no reading in the tree names its PCI ID, +model or firmware, and the program of +`issues/no-program-measures-toyos-against-linux-on-one-machine.md` reads its +NVMe identity. -**Exit**: host tests of the APST table; on the T14, the one proving machine -with a physical NVMe drive, the block figures and the idle minute with the -drive idle. **Mutation**: a table that admits a state past the bound. -**Oracle**: the specification's rules for the table over the T14's -`id-ctrl`. +**Exit**: host tests of the APST table, a quirked drive's among them; on the +T14, the one proving machine with a physical NVMe drive, the block figures and +the idle minute with the drive idle. **Mutation**: a table that admits a state +past the bound. **Oracle**: the specification's rules for the table over the +T14's `id-ctrl`. diff --git a/issues/fsgsbase-is-taken.md b/issues/fsgsbase-is-taken.md index 5bb3e8b9af..11a0ca37e3 100644 --- a/issues/fsgsbase-is-taken.md +++ b/issues/fsgsbase-is-taken.md @@ -6,7 +6,7 @@ opened: 2026-09-29 # FSGSBASE is taken -`CR4.FSGSBASE` is forbidden (`kernel/src/arch/x86_64/control_regs.rs:62-75`): +`CR4.FSGSBASE` is forbidden (`kernel/src/arch/x86_64/control_regs.rs:149-153`): GS always holds the kernel's per-CPU base and the kernel runs no `swapgs`. Taking it costs a `swapgs` entry and the swapgs barriers of `issues/user-pointer-checks-have-no-spectre-v1-fence-and-smap-is-optional.md`. @@ -31,8 +31,9 @@ nothing. enters vector 2 at CPL 0 while the user GS base is live, between entry and its `swapgs` or after the exit path's, and the handler finds the kernel's per-CPU data; a thread's FS and GS bases, written with WRFSBASE and WRGSBASE, read -back unchanged after its CPU has run another thread; `CR4_FORBIDDEN`, the -`user-writable-gsbase` kernel (`build/build.rs:1370`) and the test -`gsbase_locked` are deleted; the T14's pipe round trips read with and without -it. **Mutation**: an IST entry that decides `swapgs` from the saved CS alone. -**Oracle**: Linux's `paranoid_entry`. +back unchanged after its CPU has run another thread; `CR4_FORBIDDEN` is +deleted, and so is `gsbase_locked` with its `user-writable-gsbase` kernel from +stage C of `issues/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md`; +the T14's pipe round trips read with and without FSGSBASE. **Mutation**: an +IST entry that decides `swapgs` from the saved CS alone. **Oracle**: Linux's +`paranoid_entry`. diff --git a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md index ea6a701344..9f9d59d144 100644 --- a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md +++ b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md @@ -27,7 +27,14 @@ preferable to use MWAIT as the default idle state on these systems, as it has lower exit latencies", and at v7.2 Linux's `prefer_mwait_c1_over_halt` checks no vendor (`arch/x86/kernel/process.c`). So the stage is backed on both benches: on Intel by §11.10.4 and §11.10.6.4, and on AMD from family 17h, -the AMD bench's, by aebef63cf7ff. The T14 enumerates MONITOR/MWAIT (`monitor`, +the AMD bench's, by aebef63cf7ff. A CPU whose MONITOR loses wakes keeps the +kick, as Linux keeps it: Intel's c9a4b55431e5 (v6.13, by Len Brown) adds Lunar +Lake to Goldmont in `X86_BUG_MONITOR` because "MONITOR wakeups on Lunar Lake +processors can be lost, resulting in significant user-visible delays", "so +that wake_up_idle_cpu() always sends an IPI"; at v7.2 such a CPU's MWAIT idle +sets no polling flag, so every wake of it is an IPI +(`arch/x86/include/asm/mwait.h`), and `prefer_mwait_c1_over_halt` gives it +`hlt`. The T14 enumerates MONITOR/MWAIT (`monitor`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the wait enters; this file is what ends it. The doorbell shares a line with its @@ -56,5 +63,5 @@ stands in for here, so the share is an upper bound on what the doorbell saves. **Exit**: an idle CPU on every proving machine that enumerates MONITOR/MWAIT -woken by its doorbell's store with no IPI, and the split above read again on -both benches after it. +woken by its doorbell's store with no IPI, a CPU whose MONITOR loses wakes +still kicked, and the split above read again on both benches after it. diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md index a728607a68..dbc012f93f 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md @@ -62,17 +62,8 @@ the deadline into a span, and a deadline written as an absolute is an upper bound on what deadline mode saves, which still writes once to arm and once to stop; on the AMD bench both sit at its TSC's floor. -**Next: a TSC-deadline twin on the T14**, `main` plus the scout with the -one-shot armed and stopped through `IA32_TSC_DEADLINE`, timed by the same -split beside `main` in the same session, with the re-arm segment opening at -`set_timer` (`kernel/src/hw.rs:49`) in both arms, its reading reported. Its -clock control: each arm reads APERF and MPERF on CPUs 4 and 5 before and -after the split, and where the two arms' ratios of APERF's advance to MPERF's -differ by more than 1 % on either CPU, both arms' readings are void and are -taken again. - -**Exit**: the twin's reading, posted; a deadline armed on every CPU that -enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the absolute -value; and the T14's 1 ms timer lateness of -`issues/no-program-measures-toyos-against-linux-on-one-machine.md` no worse -than the count's. +**Exit**: a deadline armed on every CPU that enumerates TSC-deadline as one +`IA32_TSC_DEADLINE` write of the absolute value; and on the T14, the split +above read again after it, and the 1 ms timer lateness of +`issues/no-program-measures-toyos-against-linux-on-one-machine.md` read with +the count and with the deadline, each reported. diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index 4b3a0fd184..501f776b0a 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -45,8 +45,7 @@ and contention). at most 14.7 to 14.9 % of an idle wake on the T14 and 22.4 to 22.6 % on the AMD bench (`issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`). - A TSC deadline: the re-arm is at most 11.7 to 12.1 % of a pass's pick and - switch on the T14, and at the AMD bench's TSC floor; next, a TSC-deadline - twin on the T14, its reading reported + switch on the T14, and at the AMD bench's TSC floor (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`). **Moot.** PCID on the AMD bench: its CPUs read `pcid=0 invpcid=0`, so there From 11be3ba557c25e67b5c8e32c7b0828ad16863834 Mon Sep 17 00:00:00 2001 From: japabu Date: Tue, 13 Oct 2026 02:09:45 +0200 Subject: [PATCH 5/7] issues: the speed track's stages stand on the vendors' documents and the literature, and cite no Linux The owner's direction and the orchestrator's re-rulings on round 4's review (#887, 6131140814, 6131154873): - TSC deadline: Intel's SDM defines the mode and recommends nothing, AMD has none (APM, and the AMD bench's PPR 55922 rev 3.06), so the T14's speed reading decides it. A twin reads the 1 ms timer lateness under the count and under the deadline; the better is kept, and the twin is deleted once it has answered. Intel's Tiger Lake specification update 631123-034 lists no TSC, TSC-deadline or APIC-timer erratum. - The busy wake's 8-spinner arm stands on EEVDF (Stoica et al., RTSS 1996, the algorithm of TR-95-22): a quantum goes, at its start, to the eligible request with the earliest virtual deadline. The exit stands. - MWAIT idle is backed on Intel. AMD's PPR for the AMD bench enumerates MONITOR/MWAIT with one C1 sub-state and names no idle loop, so on AMD the stage follows the bench's _CST, still unread. - The kick stays on a CPU a vendor erratum names: Intel's APL30 (334820-010) names Goldmont. Lunar Lake goes: Intel's Core Ultra 200V specification update was not found to read. The verdict is host-tested by CPUID vendor, family and model (round 4's NOTE 1). - FSGSBASE stands on Intel's 767615 and its three elements, which ToyOS meets or has nothing of; its oracle is the vector-2 arm on the T14. - Per-CPU lines: Intel's Rule 17 on the T14; on the AMD bench AMD's family 17h guide (55723 rev 3.00) and PPR give the 64-byte line and the cache-to-cache transfer, and say nothing on false sharing. - APST follows the NVMe specification with no table of drives; a drive that misbehaves is a defect, measured and filed. - The track's rule states ToyOS's own figures; a comparison with another OS lives in the evidence (NOTE 2). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy --- ...s-next-pass-and-no-idle-cpu-can-take-it.md | 22 +++++----- issues/diskserver-drives-nvme-apst.md | 15 ++----- issues/fsgsbase-is-taken.md | 31 +++++++------- ...-so-every-wake-of-an-idle-cpu-is-an-ipi.md | 42 ++++++++++--------- ...pmm-state-shares-lines-every-cpu-writes.md | 19 ++++----- ...e-armed-every-pass-never-a-tsc-deadline.md | 41 ++++++++++-------- ...s-what-modern-hardware-offers-for-speed.md | 3 +- 7 files changed, 85 insertions(+), 88 deletions(-) diff --git a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md index d22050c616..42ff964cc4 100644 --- a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md +++ b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md @@ -47,17 +47,17 @@ wakes the thread on the core it last ran on if that core is idle, and otherwise on the core idle longest, where one is; the paper applies it only where the power policy keeps cores out of low-power states, since waking a core from one costs power. With 8 spinners no core idles, and a wake onto a -CPU running a task is what wakeup preemption decides: Linux's fair class has -preempted the running task for a woken one since CFS (`check_preempt_curr_fair`, -`kernel/sched_fair.c` at v2.6.23), and since v6.6 by virtual deadline -(147f3efaa241, "wakeup preemption is driven by the deadline"), the -vruntime/lag family of ToyOS's fair band (`kernel/pure/sched/fair.rs:1-3`). -The same release curbs it (63304558ba5d): under `RUN_TO_PARITY`, on by default -at v7.2, a wake does not preempt "until the current task has either matched -the 0-lag point or until is has exhausted it's slice" -(`kernel/sched/features.h`). That arm follows EEVDF whole, the curb included, -so its test is that a woken task runs within one protected slice, 1 ms -(`PROTECTED_RUN_NS`, https://github.com/ToyOSOrg/ToyOS/pull/874), and its p99 +CPU running a task is the fair band's pick to decide, which is the +vruntime/lag family (`kernel/pure/sched/fair.rs:1-3`) of EEVDF. There the +resource is "allocated in time quanta of size at most q. At the beginning of +each time quantum a client is selected", and "A new quantum is allocated to +the client which has the eligible request with the earliest virtual deadline" +(Stoica et al., "A Proportional Share Resource Allocation Algorithm for +Real-Time, Time-Shared Systems", RTSS 1996, the algorithm of Stoica and +Abdel-Wahab's TR-95-22), so a task woken inside a quantum is picked no +sooner than the next one's start. That arm follows it with a wake's q the protected +slice, 1 ms (`PROTECTED_RUN_NS`, https://github.com/ToyOSOrg/ToyOS/pull/874), +so its test is that a woken task runs within one protected slice, and its p99 against 100 µs is reported. With 7 spinners the 100 µs bound is the exit's test that the stage works, not the judge of its design. diff --git a/issues/diskserver-drives-nvme-apst.md b/issues/diskserver-drives-nvme-apst.md index b6bf085e97..6e22b4ec3a 100644 --- a/issues/diskserver-drives-nvme-apst.md +++ b/issues/diskserver-drives-nvme-apst.md @@ -16,18 +16,11 @@ transition costing the old state's exit latency plus the new one's entry latency (NVM Express Base Specification 2.1, §5.1.25.1.6, §8.1.17 and §8.1.17.2). blockd builds the table from the power state descriptors of the drive's own `id-ctrl`. The specification bounds no latency: blockd's bound is -its design round's, chosen on its merits. Drives whose APST failed are -recorded, and blockd honours the record: no APST on a drive marked -`NVME_QUIRK_NO_APST`, and no transition to its deepest state on one marked -`NVME_QUIRK_NO_DEEPEST_PS`, by the drive, firmware or machine Linux lists for -each (`drivers/nvme/host/core.c`'s `core_quirks`, and `pci.c`'s -`nvme_id_table` and `check_vendor_combination_bug`, at v7.2). Whether the -T14's drive is one of them is unread: no reading in the tree names its PCI ID, -model or firmware, and the program of -`issues/no-program-measures-toyos-against-linux-on-one-machine.md` reads its -NVMe identity. +its design round's, chosen on its merits. It follows the specification +with no table of drives: a drive that misbehaves under it is a defect, +measured on the bench and filed. -**Exit**: host tests of the APST table, a quirked drive's among them; on the +**Exit**: host tests of the APST table; on the T14, the one proving machine with a physical NVMe drive, the block figures and the idle minute with the drive idle. **Mutation**: a table that admits a state past the bound. **Oracle**: the specification's rules for the table over the diff --git a/issues/fsgsbase-is-taken.md b/issues/fsgsbase-is-taken.md index 11a0ca37e3..341a9c422c 100644 --- a/issues/fsgsbase-is-taken.md +++ b/issues/fsgsbase-is-taken.md @@ -11,21 +11,18 @@ GS always holds the kernel's per-CPU base and the kernel runs no `swapgs`. Taking it costs a `swapgs` entry and the swapgs barriers of `issues/user-pointer-checks-have-no-spectre-v1-fence-and-smap-is-optional.md`. Intel names what it gives, user programs' FS and GS bases without a system -call and a more efficient kernel context switch, and has it used in production -only with a complete OS implementation: one whose context switch manages the -FS and GS bases apart from their selectors and whose ring transitions are -insensitive to the GS base, as Linux's is from 5.9 on (Guidance for Enabling -FSGSBASE, ID 767615). Linux enables it on every CPU that enumerates it -(`arch/x86/kernel/cpu/common.c:1515-1519` at `v5.9`); its switch saves each -thread's bases as the thread left them (`process_64.c:248-262`), and its -paranoid entry assumes nothing of the GS base it finds and loads the CPU's own -from a kernel table (`entry_64.S:823-848`). AMD documents the four -instructions, enabled by CR4.FSGSBASE at every privilege level, and recommends -nothing on their use (APM 24594 rev. 3.36, Vol. 3, RDFSBASE and WRFSBASE). -ToyOS takes it under Intel's condition: its switch saves and restores each -thread's FS and GS bases as the thread left them, and its ring transitions are -insensitive to the GS base they find. Its figure is reported and decides -nothing. +call and a more efficient kernel context switch, and has it used "never in +production environments unless accompanied by a complete OS implementation": +one whose context switch and process-trace system calls manage the FS and GS +bases apart from their selectors, and whose ring transitions are insensitive +to the GS base (Guidance for Enabling FSGSBASE, ID 767615). AMD documents the +four instructions, enabled by CR4.FSGSBASE at every privilege level, and +recommends nothing on their use (APM 24594 rev. 3.36, Vol. 3, RDFSBASE and +WRFSBASE). ToyOS takes it under Intel's condition: its switch saves and +restores each thread's FS and GS bases as the thread left them, its ring +transitions are insensitive to the GS base they find, and it has no system +call that reads or writes another thread's registers +(`sdk/abi/src/syscall.rs`). Its figure is reported and decides nothing. **Exit**: on every proving machine and under TCG, a `boot-actuators` arm enters vector 2 at CPL 0 while the user GS base is live, between entry and its @@ -35,5 +32,5 @@ back unchanged after its CPU has run another thread; `CR4_FORBIDDEN` is deleted, and so is `gsbase_locked` with its `user-writable-gsbase` kernel from stage C of `issues/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md`; the T14's pipe round trips read with and without FSGSBASE. **Mutation**: an -IST entry that decides `swapgs` from the saved CS alone. **Oracle**: Linux's -`paranoid_entry`. +IST entry that decides `swapgs` from the saved CS alone. **Oracle**: the +vector-2 arm on the T14. diff --git a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md index 9f9d59d144..eeedb94ed1 100644 --- a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md +++ b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md @@ -21,22 +21,21 @@ documents MWAIT as a wait for another processor's store to the range MONITOR set up, sized by CPUID 0000_0005h (24594 rev. 3.36, Vol. 3, MONITOR and MWAIT), and neither it, its Vol. 2 (24593 rev. 3.42) nor its optimization guide for family 17h models 30h and greater (56305 rev. 3.02) names an idle -loop. AMD's own Linux commit aebef63cf7ff (v6.0, by AMD's Wyes Karny) does: -"AMD Family 17h onwards supports MWAIT as well as MWAIT C1. And it is -preferable to use MWAIT as the default idle state on these systems, as it has -lower exit latencies", and at v7.2 Linux's `prefer_mwait_c1_over_halt` checks -no vendor (`arch/x86/kernel/process.c`). So the stage is backed on -both benches: on Intel by §11.10.4 and §11.10.6.4, and on AMD from family 17h, -the AMD bench's, by aebef63cf7ff. A CPU whose MONITOR loses wakes keeps the -kick, as Linux keeps it: Intel's c9a4b55431e5 (v6.13, by Len Brown) adds Lunar -Lake to Goldmont in `X86_BUG_MONITOR` because "MONITOR wakeups on Lunar Lake -processors can be lost, resulting in significant user-visible delays", "so -that wake_up_idle_cpu() always sends an IPI"; at v7.2 such a CPU's MWAIT idle -sets no polling flag, so every wake of it is an IPI -(`arch/x86/include/asm/mwait.h`), and `prefer_mwait_c1_over_halt` gives it -`hlt`. The T14 enumerates MONITOR/MWAIT (`monitor`, -`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is -unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the +loop. Nor does the AMD bench's own reference: it enumerates MONITOR/MWAIT +unless firmware disables them, gives the monitor line as 64 bytes, and has +MWAIT support one C1 sub-state (PPR 55922 rev. 3.06, `CPUID_Fn00000001_ECX` +bit 3, `CPUID_Fn00000005`). So the stage is backed on Intel by §11.10.4 and +§11.10.6.4. On AMD it follows the firmware's `_CST` (the ground of +`issues/idle-cpus-enter-the-c-states-cst-names.md`): a C1 the AMD bench's +`_CST` enters through Functional Fixed Hardware is the firmware declaring C1 +entered by MWAIT, and that `_CST` is unread. A CPU whose vendor's erratum says +a store may not wake MWAIT keeps the kick: Intel's APL30, "A Store +Instruction May Not Wake Up MWAIT", names Goldmont (family 6 model 5Ch), and +its workaround is that "Software needs to use interrupts to wake processors +from MWAIT-induced sleep states" (specification update 334820-010). The T14's +lists no MONITOR or MWAIT erratum (631123-034). The T14 enumerates +MONITOR/MWAIT (`monitor`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); +whether the AMD bench does is unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the wait enters; this file is what ends it. The doorbell shares a line with its neighbours' published counters (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`), @@ -62,6 +61,11 @@ wake from the store, which no run has timed and which delivery's C1 exit stands in for here, so the share is an upper bound on what the doorbell saves. -**Exit**: an idle CPU on every proving machine that enumerates MONITOR/MWAIT -woken by its doorbell's store with no IPI, a CPU whose MONITOR loses wakes -still kicked, and the split above read again on both benches after it. +**Exit**: an idle CPU on every Intel proving machine that enumerates +MONITOR/MWAIT woken by its doorbell's store with no IPI, and the split above +read again on the T14 after it; the kick kept on a CPU a vendor erratum names, +by a verdict that is a pure function of CPUID's vendor, family and model and +is host-tested, red where Goldmont is woken by the store alone or the T14 is +kicked; and the AMD bench's `_CST` read, the doorbell taken there and the +split read again where its C1 is FFH MWAIT, and `hlt` with the kick kept where +it is not. diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index bbd1ebd60a..dcc9143486 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -43,16 +43,15 @@ the parts it links): Intel's optimization manual has critical data and locks allocated apart, aligned to no less than the false-sharing threshold, the CLFLUSH line size of CPUID.01H:EBX[15:8] (248966-051, Vol. 1 §11.4.5, User/Source Coding Rule -17). Linux keeps each CPU's run queue on lines of its own, -`DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues)` (`kernel/sched/core.c` -at v7.2), which aligns it by `____cacheline_aligned_in_smp` to -`SMP_CACHE_BYTES`, x86's L1 line (`include/linux/percpu-defs.h`, -`include/linux/cache.h`, `include/vdso/cache.h`, -`arch/x86/include/asm/cache.h`). AMD documents the line through the same CPUID -field (APM 24594 rev. 3.36, Vol. 3, CLFLUSH) and gives it as 64 bytes on -family 17h models 30h and greater (56305 rev. 3.02, §2.6.1, §2.6.2), and none -of APM Vol. 2 (24593 rev. 3.42), Vol. 3 and 56305 says anything on false -sharing. So the stage is backed on both benches. +17), which backs the stage on the T14. AMD documents the line through the +same CPUID field (APM 24594 rev. 3.36, Vol. 3, CLFLUSH), as 8 quadwords on the +AMD bench (PPR 55922 rev. 3.06, `CPUID_Fn00000001_EBX`), and its family's +optimization guide gives the 64-byte line and has a line another core's L2 +holds reach the reader by a cache-to-cache transfer (Software Optimization +Guide for AMD Family 17h Processors, 55723 rev. 3.00, §2.6.1, §2.6.4). None +of it, the APM's Vol. 2 (24593 rev. 3.42) and Vol. 3, nor the guide for +models 30h and greater (56305 rev. 3.02) says anything on false sharing, so on +the AMD bench AMD documents the cost and recommends nothing. Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, `CpuHandle` or `Frontier`, and the AMD bench was not read. The next diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md index dbc012f93f..b080d22dd6 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md @@ -24,20 +24,16 @@ with no calibration of a second timebase (SDM 325462-093, Vol. 3A §13.5.4.1). The T14 enumerates it (`tsc_deadline_timer`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`). AMD documents no such mode: its CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, -§E.3.2), so the AMD bench keeps the count, its calibration and its spin. +§E.3.2), and so does the AMD bench's own (PPR 55922 rev. 3.06, +`CPUID_Fn00000001_ECX`), so the AMD bench keeps the count, its calibration +and its spin. -**The published state of the art backs it.** Linux has armed its one-shot -through `IA32_TSC_DEADLINE` on every CPU that enumerates the mode since v3.8: -279f1461432c, by Intel's Suresh Siddha, "enables us to skip the APIC -calibration during boot. Also, in xapic mode, this enables us to skip the -uncached apic access to re-arm the APIC timer". At v7.2 its -`calibrate_APIC_clock` returns at once where the mode is, and -`setup_APIC_timer` registers `lapic-deadline` there -(`arch/x86/kernel/apic/apic.c`); its errata table, `deadline_match`, does not -name the T14's family 6 model 140. Since v7.1 the deadline is handed over in -TSC cycles "without reading the TSC at all" (f246ec3478cf), the absolute -write this stage takes. The stage follows them, and its figure is reported -and decides nothing. +**Its speed reading decides it.** The SDM defines the mode and recommends it +over the count nowhere, and AMD documents none, so nothing published covers +the choice. Intel's specification update for the T14's processor lists no +TSC, TSC-deadline or APIC-timer erratum (631123-034, the UP3 line, CPUID +000806C1h). Nothing but speed is its purpose: a deadline removes no path, +since a CPU without one keeps the count. **Measured** (`wt/toyos-scout-tlb` at `d4c2f8ebe`, `main`'s `177e72a9e` plus the scout; https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198), @@ -62,8 +58,17 @@ the deadline into a span, and a deadline written as an absolute is an upper bound on what deadline mode saves, which still writes once to arm and once to stop; on the AMD bench both sit at its TSC's floor. -**Exit**: a deadline armed on every CPU that enumerates TSC-deadline as one -`IA32_TSC_DEADLINE` write of the absolute value; and on the T14, the split -above read again after it, and the 1 ms timer lateness of -`issues/no-program-measures-toyos-against-linux-on-one-machine.md` read with -the count and with the deadline, each reported. +**Next: a TSC-deadline twin on the T14**, `main` plus the scout with the +one-shot armed and stopped through `IA32_TSC_DEADLINE`, beside `main` plus +the scout in the same session. Each arm reads the 1 ms timer lateness of +`issues/no-program-measures-toyos-against-linux-on-one-machine.md` and the +split above, its re-arm segment opening at `set_timer` (`kernel/src/hw.rs:49`) +in both. The arm whose lateness measures better is kept, and the split is +reported. The twin is a measurement arm, deleted once it has answered. + +**Exit**: the twin's reading, posted, and the twin deleted. Where the +deadline's lateness measures better, a deadline is armed on every CPU that +enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the absolute +value, and the split above is read again after it; where the count's does, +this file becomes `rejected` with both readings, and the boot's 10 ms +calibration spin stays the count's price. diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index 501f776b0a..8d3e002cfb 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -20,8 +20,7 @@ runners: the median of 11 runs with its spread, taken by the program of `issues/no-program-measures-toyos-against-linux-on-one-machine.md`. A stage that its hardware's manufacturers or the published state of the art back follows and cites them, and its figure is reported and never blocks it: a -slowdown, or a gap to that program's Linux figure, is filed as a defect -carrying both figures and investigated. A stage nothing published covers is +slowdown is filed as a defect carrying both figures and investigated. A stage nothing published covers is reviewed on its correctness and simplicity, its figure reported and deciding nothing by itself unless speed is the stage's only purpose. What no proving machine is known to offer is From b1c292c7679caafdae1be2544162712541eeff38 Mon Sep 17 00:00:00 2001 From: japabu Date: Tue, 13 Oct 2026 02:48:20 +0200 Subject: [PATCH 6/7] issues: the speed track answers round 5 of #887 Applies the orchestrator's rulings on round 5's review (https://github.com/ToyOSOrg/ToyOS/pull/887#issuecomment-6135434292): - MWAIT idle is Intel's alone; the AMD bench keeps hlt and the kick, and its half is the orchestrator's once its _CST is read. No FFH reading. - The Goldmont-only verdict table goes. The Core Ultra 200V specification update is recorded as unread, and the exit reads it: a 2020+ Intel MONITOR or MWAIT erratum brings the verdict and its host test back with that row. - Per-CPU lines on AMD stand on ToyOS's rule over AMD's documented line (PPR 55922 CPUID_Fn00000001_EBX[15:8], 55723 section 2.6.1); "documents the cost" goes. - The T14's MONITOR/MWAIT and TSC-deadline support no longer rest on a capture taken under another OS: both exits gain a boot line printing CPUID.01H:ECX, cited once a T14 boot carries it. - The twin's "measures better" is named: 11 runs per arm, each run's median 1 ms timer lateness, the median of 11 with its spread, and overlapping spreads keep the count. - The track's comparisons against Ubuntu leave for the evidence; it states ToyOS's own readings. - The 7-spinner arm says it does not take Lozi et al.'s power-policy condition, and the 8-spinner arm says it is blocked on #874's PROTECTED_RUN_NS. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy --- ...s-next-pass-and-no-idle-cpu-can-take-it.md | 34 ++++++++++------- ...-so-every-wake-of-an-idle-cpu-is-an-ipi.md | 33 ++++++++--------- ...pmm-state-shares-lines-every-cpu-writes.md | 14 +++---- ...e-armed-every-pass-never-a-tsc-deadline.md | 19 ++++++---- ...s-what-modern-hardware-offers-for-speed.md | 37 ++++++++----------- 5 files changed, 67 insertions(+), 70 deletions(-) diff --git a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md index 42ff964cc4..2f053c2280 100644 --- a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md +++ b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md @@ -42,26 +42,32 @@ and 3.17 µs (AMD bench), and the pair fell from about 181,600 (T14) and With 7 spinners this is the Overload-on-Wakeup bug of Lozi et al., "The Linux Scheduler: a Decade of Wasted Cores" (EuroSys 2016, §3.3): a thread that was -asleep wakes on an overloaded core while other cores are idle. Its fix -wakes the thread on the core it last ran on if that core is idle, and -otherwise on the core idle longest, where one is; the paper applies it only -where the power policy keeps cores out of low-power states, since waking a -core from one costs power. With 8 spinners no core idles, and a wake onto a -CPU running a task is the fair band's pick to decide, which is the -vruntime/lag family (`kernel/pure/sched/fair.rs:1-3`) of EEVDF. There the +asleep wakes on an overloaded core while other cores are idle. Its fix wakes +the thread on the core it last ran on if that core is idle, and otherwise on +the core idle longest, where one is; the paper applies it only where the power +policy keeps cores out of low-power states, since waking a core from one costs +power. The 7-spinner arm does not take that condition: a wake goes to an idle +core in whatever state it waits, so once +`issues/idle-cpus-enter-the-c-states-cst-names.md` lets cores sleep deeper, +the 100 µs row's p99 also holds the woken core's exit from its state, and the +row names the states its idle CPUs entered. With 8 spinners no core idles, and +a wake onto a CPU running a task is the fair band's pick to decide, which is +the vruntime/lag family (`kernel/pure/sched/fair.rs:1-3`) of EEVDF. There the resource is "allocated in time quanta of size at most q. At the beginning of each time quantum a client is selected", and "A new quantum is allocated to the client which has the eligible request with the earliest virtual deadline" (Stoica et al., "A Proportional Share Resource Allocation Algorithm for Real-Time, Time-Shared Systems", RTSS 1996, the algorithm of Stoica and -Abdel-Wahab's TR-95-22), so a task woken inside a quantum is picked no -sooner than the next one's start. That arm follows it with a wake's q the protected -slice, 1 ms (`PROTECTED_RUN_NS`, https://github.com/ToyOSOrg/ToyOS/pull/874), -so its test is that a woken task runs within one protected slice, and its p99 -against 100 µs is reported. With 7 spinners the 100 µs bound is the exit's -test that the stage works, not the judge of its design. +Abdel-Wahab's TR-95-22), so a task woken inside a quantum is picked no sooner +than the next one's start. That arm follows it with a wake's q the protected +slice, 1 ms (`PROTECTED_RUN_NS`, https://github.com/ToyOSOrg/ToyOS/pull/874, +not on `main`, so that arm is blocked on #874), so its test is that a woken +task runs within one protected slice, and its p99 against 100 µs is reported. +With 7 spinners the 100 µs bound is the exit's test that the stage works, not +the judge of its design. **Exit**: a metal row on the T14 and the AMD bench that wakes a task onto a CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs: with 7 it holds the post-to-dispatch p99 under 100 µs, and with 8 it holds -that a woken task runs within one protected slice and reports its p99. +that a woken task runs within one protected slice and reports its p99, once +#874 has landed `PROTECTED_RUN_NS`. diff --git a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md index eeedb94ed1..93f113f200 100644 --- a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md +++ b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md @@ -25,17 +25,14 @@ loop. Nor does the AMD bench's own reference: it enumerates MONITOR/MWAIT unless firmware disables them, gives the monitor line as 64 bytes, and has MWAIT support one C1 sub-state (PPR 55922 rev. 3.06, `CPUID_Fn00000001_ECX` bit 3, `CPUID_Fn00000005`). So the stage is backed on Intel by §11.10.4 and -§11.10.6.4. On AMD it follows the firmware's `_CST` (the ground of -`issues/idle-cpus-enter-the-c-states-cst-names.md`): a C1 the AMD bench's -`_CST` enters through Functional Fixed Hardware is the firmware declaring C1 -entered by MWAIT, and that `_CST` is unread. A CPU whose vendor's erratum says -a store may not wake MWAIT keeps the kick: Intel's APL30, "A Store -Instruction May Not Wake Up MWAIT", names Goldmont (family 6 model 5Ch), and -its workaround is that "Software needs to use interrupts to wake processors -from MWAIT-induced sleep states" (specification update 334820-010). The T14's -lists no MONITOR or MWAIT erratum (631123-034). The T14 enumerates -MONITOR/MWAIT (`monitor`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); -whether the AMD bench does is unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the +§11.10.6.4, and is Intel's alone: the AMD bench keeps `hlt` and the kick, and +whether it takes the doorbell is the orchestrator's to decide once that bench's +`_CST` is read. A 2020+ Intel CPU whose specification update names an erratum +by which a store may not wake MWAIT keeps the kick. The T14's lists no MONITOR +or MWAIT erratum (631123-034); Intel's Core Ultra 200V specification update is +unread. Whether the T14 and the AMD bench enumerate MONITOR/MWAIT is ToyOS's +own read to make, and no boot prints CPUID.01H:ECX yet. +`issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the wait enters; this file is what ends it. The doorbell shares a line with its neighbours' published counters (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`), @@ -61,11 +58,11 @@ wake from the store, which no run has timed and which delivery's C1 exit stands in for here, so the share is an upper bound on what the doorbell saves. -**Exit**: an idle CPU on every Intel proving machine that enumerates +**Exit**: a boot line printing CPUID.01H:ECX, and a T14 boot carrying it +cited for bit 3; an idle CPU on every Intel proving machine that enumerates MONITOR/MWAIT woken by its doorbell's store with no IPI, and the split above -read again on the T14 after it; the kick kept on a CPU a vendor erratum names, -by a verdict that is a pure function of CPUID's vendor, family and model and -is host-tested, red where Goldmont is woken by the store alone or the T14 is -kicked; and the AMD bench's `_CST` read, the doorbell taken there and the -split read again where its C1 is FFH MWAIT, and `hlt` with the kick kept where -it is not. +read again on the T14 after it; the Core Ultra 200V specification update read, +and where it or any other 2020+ Intel specification update names a MONITOR or +MWAIT erratum, the kick kept on that CPU by a verdict that is a pure function +of CPUID's vendor, family and model, host-tested with that row; and the AMD +bench on `hlt` and the kick. diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index dcc9143486..d0308f1b0f 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -43,15 +43,11 @@ the parts it links): Intel's optimization manual has critical data and locks allocated apart, aligned to no less than the false-sharing threshold, the CLFLUSH line size of CPUID.01H:EBX[15:8] (248966-051, Vol. 1 §11.4.5, User/Source Coding Rule -17), which backs the stage on the T14. AMD documents the line through the -same CPUID field (APM 24594 rev. 3.36, Vol. 3, CLFLUSH), as 8 quadwords on the -AMD bench (PPR 55922 rev. 3.06, `CPUID_Fn00000001_EBX`), and its family's -optimization guide gives the 64-byte line and has a line another core's L2 -holds reach the reader by a cache-to-cache transfer (Software Optimization -Guide for AMD Family 17h Processors, 55723 rev. 3.00, §2.6.1, §2.6.4). None -of it, the APM's Vol. 2 (24593 rev. 3.42) and Vol. 3, nor the guide for -models 30h and greater (56305 rev. 3.02) says anything on false sharing, so on -the AMD bench AMD documents the cost and recommends nothing. +17), which backs the stage on the T14. On the AMD bench the stage stands on +ToyOS's rule that per-CPU data goes on lines of its own, resting on the line +AMD documents: 8 quadwords by CPUID.01H:EBX[15:8] (PPR 55922 rev. 3.06, +`CPUID_Fn00000001_EBX[15:8]`), and 64 bytes (Software Optimization Guide for +AMD Family 17h Processors, 55723 rev. 3.00, §2.6.1). Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, `CpuHandle` or `Frontier`, and the AMD bench was not read. The next diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md index b080d22dd6..badc6fa8a1 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md @@ -21,8 +21,8 @@ calibration spins the boot for 10 ms (`apic.rs:315-320`, from `kernel/src/arch/x86_64/boot.rs:135`). TSC-deadline mode (CPUID.01H:ECX[24]) takes the absolute deadline in TSC units as one `IA32_TSC_DEADLINE` write, with no calibration of a second timebase (SDM 325462-093, Vol. 3A §13.5.4.1). -The T14 enumerates it (`tsc_deadline_timer`, -`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`). AMD documents no such mode: its +Whether the T14 enumerates it is ToyOS's own read to make, and no boot prints +CPUID.01H:ECX yet. AMD documents no such mode: its CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, §E.3.2), and so does the AMD bench's own (PPR 55922 rev. 3.06, `CPUID_Fn00000001_ECX`), so the AMD bench keeps the count, its calibration @@ -63,12 +63,17 @@ one-shot armed and stopped through `IA32_TSC_DEADLINE`, beside `main` plus the scout in the same session. Each arm reads the 1 ms timer lateness of `issues/no-program-measures-toyos-against-linux-on-one-machine.md` and the split above, its re-arm segment opening at `set_timer` (`kernel/src/hw.rs:49`) -in both. The arm whose lateness measures better is kept, and the split is -reported. The twin is a measurement arm, deleted once it has answered. +in both. Each arm runs 11 times; a run's figure is the median lateness of +its 1 ms timers, and an arm's is the median of its 11 with their spread, +lowest to highest. The deadline measures better only where its median is the +lower and the two spreads do not overlap; where they overlap, the count, the +path every CPU has, is kept. The split is reported. The twin is a measurement +arm, deleted once it has answered. -**Exit**: the twin's reading, posted, and the twin deleted. Where the +**Exit**: a boot line printing CPUID.01H:ECX, and a T14 boot carrying it +cited for bit 24; the twin's reading, posted, and the twin deleted. Where the deadline's lateness measures better, a deadline is armed on every CPU that enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the absolute -value, and the split above is read again after it; where the count's does, -this file becomes `rejected` with both readings, and the boot's 10 ms +value, and the split above is read again after it; where it does not, this +file becomes `rejected` with both readings, and the boot's 10 ms calibration spin stays the count's price. diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index 8d3e002cfb..a507437f05 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -27,8 +27,7 @@ machine is known to offer is `issues/features-no-proving-machine-is-known-to-offer.md`. **Measurement first.** Each design round starts from a posted measurement, -never a guess. Three scouts on `main`'s `177e72a9e` have read both benches, -or the T14 against Ubuntu: +never a guess. Three scouts on `main`'s `177e72a9e` have read both benches: https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119438569 (the syscall and wake latency), https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198 (the TLB @@ -52,26 +51,20 @@ is nothing to enable. **Unexplained, and the measurement each needs next.** -- **The T14's clock against Ubuntu's.** Over the `internet_download` row's - download windows the medians read 2.67 against 3.08 GHz effective - (`wt/toyos-scout-clock` at `fa6a2a829`). Ruled out: the power limits - (5.65 W median against a 20 W PL1, no power, thermal or current limit - bit logged), the HWP request (both `0x80002a04`) and C1E promotion (both - allow it). The one limit bit ToyOS logged, multi-core turbo, Ubuntu logged - in every window too. What differs: ToyOS's cores never go below C1/C1E and - its package never leaves PC0, where Ubuntu's cores spend 57 to 79 % of a - run in C7; and ToyOS's idle package draws 3.16 W against Ubuntu's 1.18 to - 1.57 W, which is `issues/idle-cpus-enter-the-c-states-cst-names.md`'s to - answer. That the idle state reaches HWP's clock choice is inference, and - the gap is not stable across sessions: round 1 of PR #861, `main`'s - column, read ToyOS at 2.53 against Ubuntu's 2.03 GHz. Next: a T14 scout - whose idle CPUs wait in a deep C-state, read against the same windows. -- **Cycles per MB.** ToyOS spends 1.98 times Ubuntu's C0 time per MB (6.09 - against 3.08 ms); the clock is about 1.15 of that and cycles per MB about - 1.58 (16.20 against 10.22 Mcyc), from about 1.17 times the instructions at - an IPC of 0.94 against about 1.26 (Ubuntu's instructions derived from - turbostat; ratios of medians). Next: cycles and instructions per MB split - by where they run, app, network stack and kernel, on both systems. +- **The T14's clock under load.** Over the `internet_download` row's + download windows ToyOS's median effective clock reads 2.67 GHz + (`wt/toyos-scout-clock` at `fa6a2a829`), and read 2.53 GHz in round 1 of + PR #861, and what holds it there is unread. Ruled out: the power limits + (5.65 W median against a 20 W PL1, no power, thermal or current limit bit + logged), the HWP request (`0x80002a04`) and C1E promotion (allowed); the one + limit bit logged is multi-core turbo. ToyOS's cores never go below C1/C1E, + its package never leaves PC0, and its idle package draws 3.16 W, which is + `issues/idle-cpus-enter-the-c-states-cst-names.md`'s to answer; that the + idle state reaches HWP's clock choice is inference. Next: a T14 scout whose + idle CPUs wait in a deep C-state, read over the same windows. +- **Cycles per MB.** ToyOS spends 6.09 ms of C0 time and 16.20 Mcyc per MB + over the same windows, at an IPC of 0.94. Next: cycles and instructions per + MB split by where they run, app, network stack and kernel. - **Global-line contention** is bounded at 2 to 3 % of C0 on the T14 and not attributed; next, the HITM count attributed to its lines on both benches (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`). From 72def236f276a247bfab48ab0770624fa5e80a5f Mon Sep 17 00:00:00 2001 From: japabu Date: Tue, 13 Oct 2026 03:00:31 +0200 Subject: [PATCH 7/7] issues: the speed track's clock item takes Intel's rated frequencies as its target Round 6 of #887 asked that each "Unexplained" reading state the target it falls short of. Per the orchestrator's ruling, the T14's clock item now cites Intel's specification of the i5-1135G7 (Max Turbo Frequency 4.20 GHz, configurable TDP-up base 2.40 GHz at 28 W, TDP-down 900 MHz at 12 W), and the 2.67 GHz reading sits above that base and 1.53 GHz short of that turbo. The C0-time and cycles-per-MB item has no published figure to stand on, so it leaves the list for the evidence repository. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy --- ...s-what-modern-hardware-offers-for-speed.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index a507437f05..eb8fda4097 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -51,20 +51,23 @@ is nothing to enable. **Unexplained, and the measurement each needs next.** -- **The T14's clock under load.** Over the `internet_download` row's - download windows ToyOS's median effective clock reads 2.67 GHz - (`wt/toyos-scout-clock` at `fa6a2a829`), and read 2.53 GHz in round 1 of - PR #861, and what holds it there is unread. Ruled out: the power limits +- **The T14's clock under load.** Intel rates the T14's i5-1135G7 at a Max + Turbo Frequency of 4.20 GHz, the most one core reaches under Turbo Boost, + over a base of 2.40 GHz at its 28 W configurable TDP-up and 900 MHz at its + 12 W TDP-down (Intel's specification of the part, + https://www.intel.com/content/www/us/en/products/sku/208658/intel-core-i51135g7-processor-8m-cache-up-to-4-20-ghz/specifications.html). + Over the `internet_download` row's download windows ToyOS's median + effective clock reads 2.67 GHz (`wt/toyos-scout-clock` at `fa6a2a829`), and + read 2.53 GHz in round 1 of PR #861: above that base and 1.53 GHz short of + that turbo, and what holds it there is unread. Ruled out: the power limits (5.65 W median against a 20 W PL1, no power, thermal or current limit bit logged), the HWP request (`0x80002a04`) and C1E promotion (allowed); the one - limit bit logged is multi-core turbo. ToyOS's cores never go below C1/C1E, + limit bit logged is multi-core turbo, whose ceiling per active core count + that specification does not give. ToyOS's cores never go below C1/C1E, its package never leaves PC0, and its idle package draws 3.16 W, which is `issues/idle-cpus-enter-the-c-states-cst-names.md`'s to answer; that the idle state reaches HWP's clock choice is inference. Next: a T14 scout whose idle CPUs wait in a deep C-state, read over the same windows. -- **Cycles per MB.** ToyOS spends 6.09 ms of C0 time and 16.20 Mcyc per MB - over the same windows, at an IPC of 0.94. Next: cycles and instructions per - MB split by where they run, app, network stack and kernel. - **Global-line contention** is bounded at 2 to 3 % of C0 on the T14 and not attributed; next, the HITM count attributed to its lines on both benches (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`).