diff --git a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md index fecbf9befb..2f053c2280 100644 --- a/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md +++ b/issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md @@ -38,10 +38,36 @@ dispatch on the kernel clock; quantiles are bucket upper bounds, accurate to With 7 spinners every wake to a busy target waited out a whole quantum on both machines, while the slowest wake to a sleeping one took 2.16 µs (T14) and 3.17 µs (AMD bench), and the pair fell from about 181,600 (T14) and -205,200 (AMD bench) rounds a second with no spinners to 100. The kill condition fixed before the run, the change -dropped if the busy-target p99 is under about 100 µs with 7 and with 8 -spinners on both benches, did not fire: it reads 68 to 105 times that. +205,200 (AMD bench) rounds a second with no spinners to 100. + +With 7 spinners this is the Overload-on-Wakeup bug of Lozi et al., "The Linux +Scheduler: a Decade of Wasted Cores" (EuroSys 2016, §3.3): a thread that was +asleep wakes on an overloaded core while other cores are idle. Its fix wakes +the thread on the core it last ran on if that core is idle, and otherwise on +the core idle longest, where one is; the paper applies it only where the power +policy keeps cores out of low-power states, since waking a core from one costs +power. The 7-spinner arm does not take that condition: a wake goes to an idle +core in whatever state it waits, so once +`issues/idle-cpus-enter-the-c-states-cst-names.md` lets cores sleep deeper, +the 100 µs row's p99 also holds the woken core's exit from its state, and the +row names the states its idle CPUs entered. With 8 spinners no core idles, and +a wake onto a CPU running a task is the fair band's pick to decide, which is +the vruntime/lag family (`kernel/pure/sched/fair.rs:1-3`) of EEVDF. There the +resource is "allocated in time quanta of size at most q. At the beginning of +each time quantum a client is selected", and "A new quantum is allocated to +the client which has the eligible request with the earliest virtual deadline" +(Stoica et al., "A Proportional Share Resource Allocation Algorithm for +Real-Time, Time-Shared Systems", RTSS 1996, the algorithm of Stoica and +Abdel-Wahab's TR-95-22), so a task woken inside a quantum is picked no sooner +than the next one's start. That arm follows it with a wake's q the protected +slice, 1 ms (`PROTECTED_RUN_NS`, https://github.com/ToyOSOrg/ToyOS/pull/874, +not on `main`, so that arm is blocked on #874), so its test is that a woken +task runs within one protected slice, and its p99 against 100 µs is reported. +With 7 spinners the 100 µs bound is the exit's test that the stage works, not +the judge of its design. **Exit**: a metal row on the T14 and the AMD bench that wakes a task onto a -CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs, and -holds its post-to-dispatch p99 under 100 µs. +CPU running a CPU-bound thread, with 7 and with 8 such threads on 8 CPUs: +with 7 it holds the post-to-dispatch p99 under 100 µs, and with 8 it holds +that a woken task runs within one protected slice and reports its p99, once +#874 has landed `PROTECTED_RUN_NS`. diff --git a/issues/diskserver-drives-nvme-apst.md b/issues/diskserver-drives-nvme-apst.md index fbee731b3c..6e22b4ec3a 100644 --- a/issues/diskserver-drives-nvme-apst.md +++ b/issues/diskserver-drives-nvme-apst.md @@ -7,12 +7,21 @@ opened: 2026-09-29 # blockd drives NVMe APST After blockd drives the T14's drive, the blockd step of -`issues/the-kernel-is-small-interrupts-post-and-threads-wait.md`. As -Linux at `Ubuntu-6.8.0-142.142`, APST stays within -`default_ps_max_latency_us` and honours `NVME_QUIRK_NO_APST` and -`NVME_QUIRK_NO_DEEPEST_PS` (`drivers/nvme/host/core.c:63,2495-2598`). +`issues/the-kernel-is-small-interrupts-post-and-threads-wait.md`. The NVMe +base specification defines APST: enabled through Feature 0Ch, the controller +moves itself, once idle for the time a power state's table entry names, to +the non-operational state that entry names, one with no higher idle power; +and the drive reports each state's worst entry and exit latency, a +transition costing the old state's exit latency plus the new one's entry +latency (NVM Express Base Specification 2.1, §5.1.25.1.6, §8.1.17 and +§8.1.17.2). blockd builds the table from the power state descriptors of the +drive's own `id-ctrl`. The specification bounds no latency: blockd's bound is +its design round's, chosen on its merits. It follows the specification +with no table of drives: a drive that misbehaves under it is a defect, +measured on the bench and filed. -**Exit**: host tests of the APST table; on the T14, the one proving machine -with a physical NVMe drive, the block figures and the idle minute with the -drive idle. **Mutation**: a table that admits a state past the bound. -**Oracle**: Linux's `nvme_configure_apst` over the T14's `id-ctrl`. +**Exit**: host tests of the APST table; on the +T14, the one proving machine with a physical NVMe drive, the block figures and +the idle minute with the drive idle. **Mutation**: a table that admits a state +past the bound. **Oracle**: the specification's rules for the table over the +T14's `id-ctrl`. diff --git a/issues/fsgsbase-is-measured-then-taken-or-rejected.md b/issues/fsgsbase-is-measured-then-taken-or-rejected.md deleted file mode 100644 index 45c5adfcf1..0000000000 --- a/issues/fsgsbase-is-measured-then-taken-or-rejected.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -status: open -kind: track -opened: 2026-09-29 ---- - -# FSGSBASE is measured, then taken or rejected - -`CR4.FSGSBASE` is forbidden (`kernel/src/arch/x86_64/control_regs.rs:62-75`): -GS always holds the kernel's per-CPU base and the kernel runs no `swapgs`. -Taking it costs a `swapgs` entry and the swapgs barriers of -`issues/user-pointer-checks-have-no-spectre-v1-fence-and-smap-is-optional.md`. - -**Exit**: the T14's pipe round trips with and without it, a loss becoming a -`rejected` issue with both figures. Taken: on every proving machine and under -TCG, a `boot-actuators` arm enters vector 2 at CPL 0 while the user GS base is -live, between entry and its `swapgs` or after the exit path's, and the handler -finds the kernel's per-CPU data; `CR4_FORBIDDEN`, the `user-writable-gsbase` -kernel (`build/build.rs:1370`) and the test `gsbase_locked` are deleted. -**Mutation**: an IST entry that decides `swapgs` from the saved CS alone. -**Oracle**: Linux's `paranoid_entry`. diff --git a/issues/fsgsbase-is-taken.md b/issues/fsgsbase-is-taken.md new file mode 100644 index 0000000000..341a9c422c --- /dev/null +++ b/issues/fsgsbase-is-taken.md @@ -0,0 +1,36 @@ +--- +status: open +kind: track +opened: 2026-09-29 +--- + +# FSGSBASE is taken + +`CR4.FSGSBASE` is forbidden (`kernel/src/arch/x86_64/control_regs.rs:149-153`): +GS always holds the kernel's per-CPU base and the kernel runs no `swapgs`. +Taking it costs a `swapgs` entry and the swapgs barriers of +`issues/user-pointer-checks-have-no-spectre-v1-fence-and-smap-is-optional.md`. +Intel names what it gives, user programs' FS and GS bases without a system +call and a more efficient kernel context switch, and has it used "never in +production environments unless accompanied by a complete OS implementation": +one whose context switch and process-trace system calls manage the FS and GS +bases apart from their selectors, and whose ring transitions are insensitive +to the GS base (Guidance for Enabling FSGSBASE, ID 767615). AMD documents the +four instructions, enabled by CR4.FSGSBASE at every privilege level, and +recommends nothing on their use (APM 24594 rev. 3.36, Vol. 3, RDFSBASE and +WRFSBASE). ToyOS takes it under Intel's condition: its switch saves and +restores each thread's FS and GS bases as the thread left them, its ring +transitions are insensitive to the GS base they find, and it has no system +call that reads or writes another thread's registers +(`sdk/abi/src/syscall.rs`). Its figure is reported and decides nothing. + +**Exit**: on every proving machine and under TCG, a `boot-actuators` arm +enters vector 2 at CPL 0 while the user GS base is live, between entry and its +`swapgs` or after the exit path's, and the handler finds the kernel's per-CPU +data; a thread's FS and GS bases, written with WRFSBASE and WRGSBASE, read +back unchanged after its CPU has run another thread; `CR4_FORBIDDEN` is +deleted, and so is `gsbase_locked` with its `user-writable-gsbase` kernel from +stage C of `issues/the-guest-suite-runs-only-what-no-cheaper-tier-reaches.md`; +the T14's pipe round trips read with and without FSGSBASE. **Mutation**: an +IST entry that decides `swapgs` from the saved CS alone. **Oracle**: the +vector-2 arm on the T14. diff --git a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md index 409d06eedc..93f113f200 100644 --- a/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md +++ b/issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md @@ -12,13 +12,33 @@ doorbell (`kernel/pure/sched/mailbox.rs:474-490`) and then sends a kick IPI (`kernel/src/arch/x86_64/apic.rs:256-260`), whose entry and handler run on the woken CPU before its `hlt` returns (`kernel/src/arch/x86_64/idt/kick.rs:3-9`). A CPU that waits in MWAIT on its doorbell's line (CPUID.01H:ECX[3]) is woken -by the store alone and takes no interrupt. The T14 enumerates it (`monitor`, -`kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether the AMD bench does is -unread. `issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the +by the store alone and takes no interrupt. The SDM names the IPI a halted CPU +needs as a delay in serving new work and has an OS consider MONITOR/MWAIT in +place of `hlt` in its C1 idle loop (325462-093, Vol. 3A §11.10.4, +§11.10.6.4), and Intel's optimization manual names the C1 loop's +responsiveness as what that buys (248966-051, Vol. 1 §11.4). AMD's APM +documents MWAIT as a wait for another processor's store to the range MONITOR +set up, sized by CPUID 0000_0005h (24594 rev. 3.36, Vol. 3, MONITOR and +MWAIT), and neither it, its Vol. 2 (24593 rev. 3.42) nor its optimization +guide for family 17h models 30h and greater (56305 rev. 3.02) names an idle +loop. Nor does the AMD bench's own reference: it enumerates MONITOR/MWAIT +unless firmware disables them, gives the monitor line as 64 bytes, and has +MWAIT support one C1 sub-state (PPR 55922 rev. 3.06, `CPUID_Fn00000001_ECX` +bit 3, `CPUID_Fn00000005`). So the stage is backed on Intel by §11.10.4 and +§11.10.6.4, and is Intel's alone: the AMD bench keeps `hlt` and the kick, and +whether it takes the doorbell is the orchestrator's to decide once that bench's +`_CST` is read. A 2020+ Intel CPU whose specification update names an erratum +by which a store may not wake MWAIT keeps the kick. The T14's lists no MONITOR +or MWAIT erratum (631123-034); Intel's Core Ultra 200V specification update is +unread. Whether the T14 and the AMD bench enumerate MONITOR/MWAIT is ToyOS's +own read to make, and no boot prints CPUID.01H:ECX yet. +`issues/idle-cpus-enter-the-c-states-cst-names.md` is which state the wait enters; this file is what ends it. The doorbell shares a line with its neighbours' published counters (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`), -so a monitor on it today would also wake on their writes. +so a monitor on it today would also wake on their writes; the SDM has +monitored data padded to the largest monitor line size CPUID.05H reports +(§11.10.5). **Measured** (`wt/toyos-scout-tlb` at `d4c2f8ebe`, `main`'s `177e72a9e` plus the scout; https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198). @@ -36,32 +56,13 @@ send to the woken task running, p50 in TSC ticks per direction: A CPU woken from MWAIT by the store skips all three segments but pays its own wake from the store, which no run has timed and which delivery's C1 exit stands in for here, so the share is an upper bound on what the doorbell -saves. Applied to that bound, the line fixed before the scout, the doorbell -dropped under 20 % of the wake, fires on the T14 and not on the AMD bench. +saves. -**Next: an MWAIT twin** beside a control arm, both timed by the same split in -the same session. The control arm is `main` plus the scout with the doorbell -on a line of its own, which is -`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`'s -change. The twin is the control arm with each idle CPU waiting in MWAIT on its -doorbell's line, hint C1 (EAX = 0), the state `hlt` enters, and the wake's -kick not sent. The two arms differ only in what ends the wait, which is what -the bound above covers. Its kill line, fixed here before it runs: the -doorbell is built ahead of `issues/idle-cpus-enter-the-c-states-cst-names.md` -only if, on a proving machine that enumerates MONITOR/MWAIT, the twin's whole -idle wake p50 is at least 20 % under the control arm's in both directions. -The T14's bound is under that line, so the twin runs on the AMD bench, and -reads CPUID.01H:ECX[3] there first; if the AMD bench does not enumerate it, no -proving machine can clear the line. Its clock control, fixed with the line: -each arm reads APERF and MPERF on CPUs 4 and 5 before and after the split, -and where the two arms' ratios of APERF's advance to MPERF's differ by more -than 1 % on either CPU, both arms' readings are void and decide nothing. - -**Exit**: the twin's reading, posted, and an idle CPU on every proving machine -that enumerates MONITOR/MWAIT woken by its doorbell's store with no IPI. Where -the twin clears the line, that is built ahead of the C-states. Where it does -not, this file stays open behind -`issues/idle-cpus-enter-the-c-states-cst-names.md`: the 20 % prices an idle -path beside `hlt`, which that track builds anyway, and once an idle CPU waits -in MWAIT there, putting its monitor on the doorbell's line and sending no -kick adds no second path. +**Exit**: a boot line printing CPUID.01H:ECX, and a T14 boot carrying it +cited for bit 3; an idle CPU on every Intel proving machine that enumerates +MONITOR/MWAIT woken by its doorbell's store with no IPI, and the split above +read again on the T14 after it; the Core Ultra 200V specification update read, +and where it or any other 2020+ Intel specification update names a MONITOR or +MWAIT erratum, the kick kept on that CPU by a verdict that is a pure function +of CPUID's vendor, family and model, host-tested with that row; and the AMD +bench on `hlt` and the kick. diff --git a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md index f055905d90..d0308f1b0f 100644 --- a/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md +++ b/issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md @@ -40,13 +40,20 @@ the parts it links): only these four; and the load event (D2H/04H) on this CPU also counts clean forwards. -The kill condition fixed before the run, these recorded as structure only if -both figures are under 1 %, did not fire on its HITM half, so the contention -is not dismissed. Nothing in the run attributes a miss to the `BITMAP` lock, -`IN_PASS`, `CpuHandle` or `Frontier`, and the AMD bench was not read. The -next measurement attributes the HITM count to the lines it lands on, on both +Intel's optimization manual has critical data and locks allocated apart, +aligned to no less than the false-sharing threshold, the CLFLUSH line size of +CPUID.01H:EBX[15:8] (248966-051, Vol. 1 §11.4.5, User/Source Coding Rule +17), which backs the stage on the T14. On the AMD bench the stage stands on +ToyOS's rule that per-CPU data goes on lines of its own, resting on the line +AMD documents: 8 quadwords by CPUID.01H:EBX[15:8] (PPR 55922 rev. 3.06, +`CPUID_Fn00000001_EBX[15:8]`), and 64 bytes (Software Optimization Guide for +AMD Family 17h Processors, 55723 rev. 3.00, §2.6.1). + +Nothing in the run attributes a miss to the `BITMAP` lock, `IN_PASS`, +`CpuHandle` or `Frontier`, and the AMD bench was not read. The next +measurement attributes the HITM count to the lines it lands on, on both benches. -**Exit**: that attribution, posted; and each of the four either written per -CPU on a line no other CPU writes, or kept with its refusal reason at its -site and its measured cost on both benches in the closing commit. +**Exit**: that attribution, posted, and read again after the change; and +each of the four either written per CPU on a line no other CPU writes, or +kept with its refusal reason at its site. diff --git a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md index d38c6b9321..badc6fa8a1 100644 --- a/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md +++ b/issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md @@ -20,9 +20,20 @@ https://github.com/ToyOSOrg/ToyOS/pull/861#issuecomment-6113257693). A timer fir calibration spins the boot for 10 ms (`apic.rs:315-320`, from `kernel/src/arch/x86_64/boot.rs:135`). TSC-deadline mode (CPUID.01H:ECX[24]) takes the absolute deadline in TSC units as one `IA32_TSC_DEADLINE` write, -with no calibration of a second timebase. The T14 enumerates it -(`tsc_deadline_timer`, `kernel/cpuvuln/fixtures/t14/cpuinfo.txt:20`); whether -the AMD bench does is unread. +with no calibration of a second timebase (SDM 325462-093, Vol. 3A §13.5.4.1). +Whether the T14 enumerates it is ToyOS's own read to make, and no boot prints +CPUID.01H:ECX yet. AMD documents no such mode: its +CPUID reference has CPUID.01H:ECX[24] reserved (APM 24594 rev. 3.36, Vol. 3, +§E.3.2), and so does the AMD bench's own (PPR 55922 rev. 3.06, +`CPUID_Fn00000001_ECX`), so the AMD bench keeps the count, its calibration +and its spin. + +**Its speed reading decides it.** The SDM defines the mode and recommends it +over the count nowhere, and AMD documents none, so nothing published covers +the choice. Intel's specification update for the T14's processor lists no +TSC, TSC-deadline or APIC-timer erratum (631123-034, the UP3 line, CPUID +000806C1h). Nothing but speed is its purpose: a deadline removes no path, +since a CPU without one keeps the count. **Measured** (`wt/toyos-scout-tlb` at `d4c2f8ebe`, `main`'s `177e72a9e` plus the scout; https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198), @@ -48,22 +59,21 @@ is an upper bound on what deadline mode saves, which still writes once to arm and once to stop; on the AMD bench both sit at its TSC's floor. **Next: a TSC-deadline twin on the T14**, `main` plus the scout with the -one-shot armed and stopped through `IA32_TSC_DEADLINE`, timed by the same -split beside `main` in the same session, with the re-arm segment opening at -`set_timer` (`kernel/src/hw.rs:49`) in both arms. Its kill line, fixed here -before it runs: deadline mode is not worth building for speed if what it -takes off a pass, `main`'s re-arm p50 less the twin's, is under 5 % of -`main`'s pick plus switch in either direction (221 and 216 ticks on the -scout's figures). Its clock control, fixed with the line: each arm reads -APERF and MPERF on CPUs 4 and 5 before and after the split, and where the two -arms' ratios of APERF's advance to MPERF's differ by more than 1 % on either -CPU, both arms' readings are void and decide nothing. Where the line fires, -this file becomes `rejected` with both readings, and the boot's 10 ms -calibration spin stays the count's price. +one-shot armed and stopped through `IA32_TSC_DEADLINE`, beside `main` plus +the scout in the same session. Each arm reads the 1 ms timer lateness of +`issues/no-program-measures-toyos-against-linux-on-one-machine.md` and the +split above, its re-arm segment opening at `set_timer` (`kernel/src/hw.rs:49`) +in both. Each arm runs 11 times; a run's figure is the median lateness of +its 1 ms timers, and an arm's is the median of its 11 with their spread, +lowest to highest. The deadline measures better only where its median is the +lower and the two spreads do not overlap; where they overlap, the count, the +path every CPU has, is kept. The split is reported. The twin is a measurement +arm, deleted once it has answered. -**Exit**: the twin's reading, posted, and the AMD bench's CPUID.01H:ECX[24] -read. Where the twin clears the line, a deadline is armed on every CPU that +**Exit**: a boot line printing CPUID.01H:ECX, and a T14 boot carrying it +cited for bit 24; the twin's reading, posted, and the twin deleted. Where the +deadline's lateness measures better, a deadline is armed on every CPU that enumerates TSC-deadline as one `IA32_TSC_DEADLINE` write of the absolute -value, and the T14's 1 ms timer lateness of -`issues/no-program-measures-toyos-against-linux-on-one-machine.md` is no worse -than the count's. +value, and the split above is read again after it; where it does not, this +file becomes `rejected` with both readings, and the boot's 10 ms +calibration spin stays the count's price. diff --git a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md index ba15554fe3..eb8fda4097 100644 --- a/issues/toyos-uses-what-modern-hardware-offers-for-speed.md +++ b/issues/toyos-uses-what-modern-hardware-offers-for-speed.md @@ -18,14 +18,16 @@ each that has its feature. A figure is the T14's, and the AMD bench's too where the stage's path differs there, since the EPYC guests are shared CI runners: the median of 11 runs with its spread, taken by the program of `issues/no-program-measures-toyos-against-linux-on-one-machine.md`. -A stage is done at or past that program's Linux figure, or with the gap filed -as a defect carrying both; a feature Linux has no counterpart for lands only -if it beats ToyOS without it. What no proving machine is known to offer is +A stage that its hardware's manufacturers or the published state of the art +back follows and cites them, and its figure is reported and never blocks it: a +slowdown is filed as a defect carrying both figures and investigated. A stage nothing published covers is +reviewed on its correctness and simplicity, its figure reported and deciding +nothing by itself unless speed is the stage's only purpose. What no proving +machine is known to offer is `issues/features-no-proving-machine-is-known-to-offer.md`. **Measurement first.** Each design round starts from a posted measurement, -never a guess, judged by a kill condition fixed before the run. Three scouts -on `main`'s `177e72a9e` have read both benches, or the T14 against Ubuntu: +never a guess. Three scouts on `main`'s `177e72a9e` have read both benches: https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119438569 (the syscall and wake latency), https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119443198 (the TLB @@ -33,25 +35,15 @@ and the idle wake) and https://github.com/ToyOSOrg/ToyOS/pull/866#issuecomment-6119513916 (the clock and contention). -**Measured, and worth building.** Each issue carries its reading. +**Measured.** Each issue carries its reading. -- The syscall's FP save and the `lock` prefixes it takes are 34.5 % of a null - syscall on the T14 and 43.4 % on the AMD bench: - `issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md`. -- A wake to a busy CPU waits out a whole 10 ms quantum on both benches, p99 - 68 to 105 times a 100 µs bar: +- A wake to a busy CPU waits out a whole 10 ms quantum on both benches: `issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md`. - -**Bounded, and judged next by a twin** whose kill line its issue fixes -before it runs. - - An MWAIT doorbell: the IPI's send and delivery and the kick's handler are at most 14.7 to 14.9 % of an idle wake on the T14 and 22.4 to 22.6 % on the - AMD bench; next, an MWAIT twin on the AMD bench - (`issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`). + AMD bench (`issues/idle-is-hlt-so-every-wake-of-an-idle-cpu-is-an-ipi.md`). - A TSC deadline: the re-arm is at most 11.7 to 12.1 % of a pass's pick and - switch on the T14, and at the AMD bench's TSC floor; next, a TSC-deadline - twin on the T14 + switch on the T14, and at the AMD bench's TSC floor (`issues/the-timer-is-a-lapic-count-re-armed-every-pass-never-a-tsc-deadline.md`). **Moot.** PCID on the AMD bench: its CPUs read `pcid=0 invpcid=0`, so there @@ -59,54 +51,39 @@ is nothing to enable. **Unexplained, and the measurement each needs next.** -- **The T14's clock against Ubuntu's.** Over the `internet_download` row's - download windows the medians read 2.67 against 3.08 GHz effective - (`wt/toyos-scout-clock` at `fa6a2a829`). Ruled out: the power limits - (5.65 W median against a 20 W PL1, no power, thermal or current limit - bit logged), the HWP request (both `0x80002a04`) and C1E promotion (both - allow it). The one limit bit ToyOS logged, multi-core turbo, Ubuntu logged - in every window too. What differs: ToyOS's cores never go below C1/C1E and - its package never leaves PC0, where Ubuntu's cores spend 57 to 79 % of a - run in C7; and ToyOS's idle package draws 3.16 W against Ubuntu's 1.18 to - 1.57 W, which is `issues/idle-cpus-enter-the-c-states-cst-names.md`'s to - answer. That the idle state reaches HWP's clock choice is inference, and - the gap is not stable across sessions: round 1 of PR #861, `main`'s - column, read ToyOS at 2.53 against Ubuntu's 2.03 GHz. Next: a T14 scout - whose idle CPUs wait in a deep C-state, read against the same windows. -- **Cycles per MB.** ToyOS spends 1.98 times Ubuntu's C0 time per MB (6.09 - against 3.08 ms); the clock is about 1.15 of that and cycles per MB about - 1.58 (16.20 against 10.22 Mcyc), from about 1.17 times the instructions at - an IPC of 0.94 against about 1.26 (Ubuntu's instructions derived from - turbostat; ratios of medians). Next: cycles and instructions per MB split - by where they run, app, network stack and kernel, on both systems. -- **The syscall's `lock` saving** exceeds what its prefixes cost in Ring 3; - next, the prefix stripped one site at a time (its issue). +- **The T14's clock under load.** Intel rates the T14's i5-1135G7 at a Max + Turbo Frequency of 4.20 GHz, the most one core reaches under Turbo Boost, + over a base of 2.40 GHz at its 28 W configurable TDP-up and 900 MHz at its + 12 W TDP-down (Intel's specification of the part, + https://www.intel.com/content/www/us/en/products/sku/208658/intel-core-i51135g7-processor-8m-cache-up-to-4-20-ghz/specifications.html). + Over the `internet_download` row's download windows ToyOS's median + effective clock reads 2.67 GHz (`wt/toyos-scout-clock` at `fa6a2a829`), and + read 2.53 GHz in round 1 of PR #861: above that base and 1.53 GHz short of + that turbo, and what holds it there is unread. Ruled out: the power limits + (5.65 W median against a 20 W PL1, no power, thermal or current limit bit + logged), the HWP request (`0x80002a04`) and C1E promotion (allowed); the one + limit bit logged is multi-core turbo, whose ceiling per active core count + that specification does not give. ToyOS's cores never go below C1/C1E, + its package never leaves PC0, and its idle package draws 3.16 W, which is + `issues/idle-cpus-enter-the-c-states-cst-names.md`'s to answer; that the + idle state reaches HWP's clock choice is inference. Next: a T14 scout whose + idle CPUs wait in a deep C-state, read over the same windows. - **Global-line contention** is bounded at 2 to 3 % of C0 on the T14 and not attributed; next, the HITM count attributed to its lines on both benches (`issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md`). - **The wake storm's per-waiter gap** between the benches; next, its split (`issues/a-wake-storm-costs-the-t14-twice-the-amd-benchs-time-per-waiter-and-nothing-says-why.md`). -- **Global pages.** CR4.PGE is clear on both benches, so where there is no - PCID every process switch drops the kernel's translations. On the AMD - bench, which has none, a user ping-pong takes 80.7 full page walks per round trip with no - user working set and 88.6 at four pages, which prices at about 7 % of the - round trip (derived, user and kernel walks together, so a ceiling on what - global pages save). On the T14, PCID saves 3.4 % of the same round trip at - four pages (`wt/toyos-scout-tlb` at `d4c2f8ebe`). The decision and its - sequencing are `issues/page-global-is-a-decision-nobody-has-made.md`'s; - next, a PGE twin on the AMD bench. The CPU cap is PR #865's (`wt/toyos-cpus`), a design round of its own. **Exit**: every issue below is closed. They are listed in the order they are -taken: the instruments, then what is measured and worth building, then what -waits on its next measurement, then the design rounds not yet measured. +taken: the instruments, then what is measured, then the design rounds not +yet measured. `issues/user-programs-use-avx-under-xsave.md` is PR #859's, built ahead of that order. - `issues/no-test-can-hold-a-thread-on-a-named-cpu.md` - `issues/no-program-measures-toyos-against-linux-on-one-machine.md` -- `issues/every-syscall-saves-the-fp-state-and-every-preemption-disable-is-locked.md` - `issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md` - `issues/per-cpu-scheduler-and-pmm-state-shares-lines-every-cpu-writes.md` - `issues/a-wake-storm-costs-the-t14-twice-the-amd-benchs-time-per-waiter-and-nothing-says-why.md` @@ -117,6 +94,8 @@ that order. - `issues/kernel-forward-copies-and-fills-are-one-rep-movsb-or-stosb-on-every-cpu.md` - `issues/the-direct-map-uses-1-gib-leaves-where-the-cpu-has-them.md` - `issues/a-tlb-shootdown-reaches-only-the-cpus-that-ran-the-space.md` -- `issues/fsgsbase-is-measured-then-taken-or-rejected.md` +- `issues/fsgsbase-is-taken.md` - `issues/idle-cpus-enter-the-c-states-cst-names.md` - `issues/diskserver-drives-nvme-apst.md` +- `issues/every-syscall-takes-the-process-table-lock-twice-to-count-itself.md` +- `issues/a-thread-resuming-on-the-cpu-that-holds-its-state-restores-it-again.md`