A wake to another CPU that is busy is drained at its next preemption point and runs within a bound

[Japabu]

A wake posted to a busy CPU waited for that CPU's next pass, which with a CPU-bound thread running is the 10 ms quantum, and no idle CPU could take it. This branch builds the design accepted in the acceptance comment, with K4 added to the metal judgement. It takes up issues/a-wake-to-a-busy-cpu-waits-for-its-next-pass-and-no-idle-cpu-can-take-it.md and does not close it: that exit also asks for the case where all eight CPUs run spinners of one process, which this design leaves to the fair band (roast 1) and which issues/a-woken-thread-waits-out-a-sibling-spinners-quantum-when-every-cpu-is-busy.md now records for S4.

Head 0ab013e77 on main b680a973b. Evidence is on ToyOSOrg/evidence branch pr-874: d170207 holds this round's gates; 4a6b534 the metal readbacks and judge logs of rounds 3 and 4, whole and masked; ba98b45 holds this round; c940595 round 3; d63b878 the second metal judgement; cfb4aa7 the first; 20056cc round 2; b036b48 the investigation and the first whole-change control.

Where this stands

• The design and its gates. The first judgement sent the design back on K3. The orchestrator then applied the owner's ruling (A design follows what its hardware's manufacturers or the published state of the art back, and a bench verdict never blocks it #885): the forward to an idle CPU, the threshold-1 steal and the 1 ms protected preemption follow the published work Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872 cites, so their K2–K4 readings are reported; the busy-target kick has no cited source and speed is its whole purpose, so K1 and the row wake_onto_busy gate it. The second judgement passed both gates on both benches.
• Round 3's metal run of a5e247249: the T14 passed 300 of 301, red on wake_storm_cost (judge log, exit 101); the control, the kick reverted, was red on wake_onto_busy at p99 10,001 µs on both benches (below). The AMD bench passed 287 of 301 (log, exit 101). Round 4 found the T14's red and fixed it.
• Round 4's metal run of 4118fbbeb, whose kernel this head carries unchanged but for main's x86-64 kernel leaves are global under a declared CR4.PGE, and only a kernel-half change drops them #873: the T14 passed 300 of 300; the AMD bench passed 286 of 300, red on the 14 rows tabled below and on nothing else. Every line stated before the run holds (## Metal: judged). Round 5 deleted one host test, corrected an issue, filed one and merged main, and owes no new metal run.

Round 4: the T14's wake_storm_cost red, and the AMD bench's

Root cause: the self kick ran a pass inside the waker's syscall

• What the row saw. One futex_wake of 16 waiters cost the waker 26,331 cycles and one of 64 cost 132,867, past the row's linear bound of five times.
• Metal says it is the kick. Across every T14 readback on this host (table), the three boots that carried this branch's kick read 132,867, 144,038 and 153,899 cycles at 64 waiters. The 50 boots without it read 52,910 to 74,328, round 3's control among them at 62,205. Round 3's control differs from its head by the one line of Doorbell::ring.
• Which half of the kick. The kick has two arms: the IPI to another busy CPU, and the poster's own CPU, which it answers with need_resched. On an 8-vCPU guest the storm syscall was instrumented (patches, scripts and logs; a guest's numbers decide nothing):
  • At the head, the 64-waiter user reading was 105,000 to 709,000 over eleven runs. With the kick reverted it was 58,000 to 63,000 over five. With only the own-CPU arm's need_resched removed it was 63,000 to 129,000 over four.
  • Inside the walk (post_n itself), cycles barely moved between the head and the revert.
  • After the walk, the head's storm syscall held about one pass at 64 waiters and none with the kick reverted (summary, reverted). That pass runs in exit_to_user and places every waiter parked on the waker's CPU. From the walk's end to the syscall's exit took a mean of 704,300 cycles at the head against 19,200 reverted. In that span the waker's CPU received 0.2 kick IPIs on average and switched no task.
  • Where the pass's time went: waiting on the process's fair-share lock. It read 5,500 and 7,200 cycles at 16 waiters, and 1,291,100 and 616,300 at 64 (run a, run b). That lock is the one every one of the storm's 64 threads takes as it dispatches and re-parks, so the wait grows faster than the storm.
  • On main the same placements run in the waker's next pass, at its block, after the burst.
• The fix. An ordinary wake posted on the CPU it goes to rings nothing (decision 1, amended). With it, the guest's 64-waiter reading was 80,000, 62,000 and 59,000 (runs, fix-*.log).
  • Its unit tests are above, and its mutations below.
  • Its metal control reds where the line said it would (## Metal: judged).
  • What it leaves is filed for Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's S3: issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md.
  • Accepted (A wake to another CPU that is busy is drained at its next preemption point and runs within a bound #874 (comment)): a change to decision 1.

The AMD bench's 14 reds

Round 3's judge lines and one boot line each names are excerpted here. Round 4's AMD judge log (whole, 117,795 lines) carries the same 14 FAIL lines, row for row, and no other.

main's record of the AMD bench. main has no whole-suite run of the AMD bench anywhere.

• The bench's record in the host's store (~/.cache/toyos/metal/, one file per machine) holds timings only, and no verdicts.
• ToyOSOrg/evidence holds no whole-suite AMD run on any branch.
• What main records are its issues, from PR The metal suite targets a second bench: --metal-host, every machine's record in the host's store, the home read off the machine, and the stick read off the record #852's AMD boots, and round 3's offline judgement of the second judgement's base testcases boot. That boot is main plus a scout, and it was red on 6 of its 19 rows (log).

Every red names hardware or firmware the bench has and the T14 has not, or lacks. None is a scheduling outcome:

row	the boot's cause	red at main's base	issue
hda_tone, hda_client_stall	two HDA controllers answer; the kernel refuses both	yes	the-amd-bench-answers-on-two-hda-controllers-and-plays-on-the-null-sink.md
acpi_server_events, acpi_tables_loaded, battery_read	the DSDT refused for a SystemMemory read in type-11 memory; with no namespace there is no EC and no battery	not run there	acpiserver-refuses-the-amd-benchs-dsdt-for-a-read-in-memory-mapped-io.md
acpi_server_death	the row's needle is the T14's ACPI_ENABLE 0xf0 to SMI_CMD 0xb2; the bench wrote 0xa0 to 0xb0	not run there	a-metal-row-names-no-hardware-and-a-bench-without-it-reds.md
claim_reuses_its_remapping_entry, claim_refused_without_remapping	the job claims the T14's I219 (8086:15fc): NotFound	yes / not run there	a-metal-row-names-no-hardware-and-a-bench-without-it-reds.md, which names both
usbd_drives_the_type_c_controller	"neither spare controller is on this machine"	not run there	a-metal-row-names-no-hardware-and-a-bench-without-it-reds.md
domain_ends_below_the_host_bridges	"no unit on this machine translates": AMD-Vi is switched off	yes	an-amd-vi-machine-isolates-no-device.md
counters	the needle is pm_enable=1, HWP's; the bench declares no performance request ("no HWP")	not run there	the-perf-state-declaration-refuses-hybrid-intel-and-amd-cppc.md
smp_roster_and_tsc_trail	8 CPUs given, 16 in the MADT: the kernel's MAX_CPUS is 8	yes	filed: the-kernel-brings-up-at-most-eight-cpus.md, byte for byte the file open #865 adds, so the red is recorded whichever lands first and the second merges cleanly
loader_watchdog_arms	no TCO block, for the loader or the kernel	not run there	filed: no-watchdog-arms-on-an-amd-machine.md
hard_lockup_ends_a_deaf_cpu	"CPUID states no architectural performance counter"	not run there	filed: the-hard-lockup-detector-samples-no-amd-cpu.md

"Not run there" means the base boot that was judged did not carry that row: it was the second judgement's filtered testcases image, or another boot. Each of those rows' causes is a CPUID leaf, a PCI function, an ACPI table or a row's needle, which this branch's scheduler does not touch.

Round 3: the review's findings (at a5e247249)

BLOCKER 1, the S4 issue: filed. issues/a-woken-thread-waits-out-a-sibling-spinners-quantum-when-every-cpu-is-busy.md carries both judgements' phase-2 halving (500 → 250 rounds, every head wake one quantum on the waker's CPU, both benches), the AMD bench's phase-3 p99 (≤ 10,485,759 ns at base; ≤ 15,728,639 and ≤ 12,582,911 ns at head), and the T14's one 10,002,218 ns phase-5 wake. Owner: #872's S4. Exit: S4's re-run of K1–K4, as the ruling orders.

BLOCKER 2, a gate that reds without the kick.

• The review's revert against the row as it stood. On a 4-vCPU guest from the judge-head tree (b84bd8157, whose kernel/ is d5165c28a's), the row read p50 279 µs, p99 2,586 µs unpatched and p50 74 µs, p99 2,149 µs with busy-target-unkicked (unpatched, reverted, script; both smokes EXIT=0). No slower without the kick: the review's reading holds. Metal was not asked.
• The row now wakes a fresh partner onto the busy CPU every round. 200 partners are staged parked on one CPU, other than the waker's and x2APIC id 0, before a spinner is kept there. Each round wakes one never woken before; it says where it ran and parks again where it is. The waker spins from the first placement to the last round, so its CPU is never idle and it is never woken. When the kernel's balance path moves the spinner to an idle CPU, which a 4-vCPU guest did at both trees, a new spinner is kept on the busy CPU before the next round, up to eight times.
• On the same guest at a5e247249 (head, reverted, through a scratch harness entry): 200 wakes each, all run off the busy CPU, no spinner kept again; p50 267 µs, p99 1,398 µs at the head, p50 14,998 µs, p99 16,556 µs with the kick reverted. A guest's numbers decide nothing. On metal, round 3's control (aa7b41581, the kick reverted) was red on wake_onto_busy: p99 10,001 µs on the T14 (judge log, exit 101; kernel log) and on the AMD bench (log, exit 101), where every head since reads p99 5 to 25 µs.

BLOCKER 3, the steal's published count.

• Merged first. On the merged tree, before any fix, main's I15 caught exactly the hole the review describes: "cpu1 has been idle for 5 rounds while cpu0 holds 1 queued fair task(s) (bound 4 rounds)", in the lopsided cases at 2, 8 and 16 CPUs (log, EXIT=101).
• The review's fix. The pass publishes fair_len() and answers probes from stealable(), and the three drain asserts are exact again (best == floor). With that alone, I15 held (log). At this head the asserts hold: drained 7/7, 15/15, 63/63, 63/63 (log).
• It opened one more red, and the fix that closes it is mine, not the review's. A probe drained in the very pass that switched away from the only queued task was consumed and declined. The re-arm policy's probe gap at 2 CPUs rose to 24 ms against its derived 13 ms; in seed 14 both of the thief's re-arms landed on the victim's quantum-end passes (trace). Proposed for acceptance: a probe declined only for the task the CPU stands on is kept, and the pass asks for its own next pass (need_resched), which, past the switch, hands that task over. At this head the gap is 10 ms worst over 20 seeds (log). Roast: it is a second reason for a pass to ask for a pass, and it costs one more pass where it fires; the alternative, consuming the probe, leaves the pull path's recovery to the push alone.
• main's topology suite at threshold 1. Two of its cases were staged for PUSH_THRESHOLD 2:
  • its negative gate for a push one above the threshold now stages PUSH_THRESHOLD + 1 threads for cpus + 3 quanta, the victim that push leaves unannounced, outlasting I15's bound;
  • the_push_keeps_an_idle_cpu_beside_work_inside_its_bound loses its tightness half. Each probe now kicks the victim, whose extra pass pushes the next sleeper, so over 200 seeds the worst stretch is 5 rounds against 10 on the T14's shape and 6 against 18 on the AMD bench's (log). The bound is still the derivation's worst case.
  • group_imbalance's neighbours now hold stealable work, so it no longer stages an idle CPU whose CCX holds nothing a probe takes.
  • The last two are issues/the-work-conservation-cases-were-staged-for-a-push-threshold-of-two.md, owned by S2.
• cargo test -p kernel-sim at this head: EXIT=0 (log).

BLOCKER 4, the rows riding the testcases boot, judged offline from the second judgement's readbacks with --metal-readback, each from the tree that built it, under a scratch HOME whose metal record is a copy, so no judging here wrote the host's record (script):

bench	base 4cbf3c381	head b84bd8157
T14	18 of 19, EXIT=1: only wake_onto_busy red (log)	19 of 19, EXIT=0 (log)
AMD	13 of 19, EXIT=1 (log)	14 of 19, EXIT=1 (log)

• On the AMD bench the same five rows red at both trees, and none of them is wake_onto_busy. They are hda_tone, hda_client_stall, smp_roster_and_tsc_trail (8 CPUs given, 16 in the MADT), claim_reuses_its_remapping_entry and domain_ends_below_the_host_bridges: hardware the bench lacks, issues/a-metal-row-names-no-hardware-and-a-bench-without-it-reds.md. Base adds wake_onto_busy.
• What the rows printed, base → head, one boot each:
  • spawn_cost median 1,025 → 1,029 µs (T14), 2,054 → 2,086 µs (AMD);
  • syscall_cost 651 → 648 cycles (T14), 388 → 388 (AMD);
  • during audio_tone, soundserver's worst wake latency was 212 and 600 µs at base and 223 and 427 µs at head on the T14, with 0 underruns at both;
  • wake_storm_cost per waiter: 386 → 976 ns on the T14 and 151 → 328 ns on the AMD bench. Its row asserts only the walk's linear shape, and passes. Read from the code, not traced, that is the busy-target kick's cost to a waker: a busy target the walk reaches is now sent an IPI on each 0→1 edge, where base elided it. It is reported, not gated. Round 4 traced it to the self kick, which is gone; at this kernel the T14 reads 401 ns (## Metal: judged).
• Every other row was owed then, and round 4's whole-suite run on both benches judged them (## Metal: judged).

BLOCKER 5: apic_id() is deleted; the row reads x2apic_id() (leaf 0BH), as acpi_mediated and acpi_release do, and arch/cpu.rs is main's again.

NOTEs.

• interactive-mix-crosses-the-fairness-bound-at-two-cpus.md is owned by S4, and measured again at 17c40459d, whose kernel/ is this head's: 197 of 2,000 runs cross, worst 254 ms against 217 ms.
• idle_sibling's answering(now) term is deleted. It was dead behind !kick_pending(), whose standing edge is what now refuses a stopped CPU after its first forward. answering's contract says so. placement-ignores-staleness still reds a_stopped_cpu_stops_taking_work under --ci host.
• The RT forward refusing a kicked sleeper is an RT-latency change. The audio rows above, one boot each, show no difference. Whether they reach that case is not known.
• running_fair is inlined into place.
• The sync-wake forward is issues/a-wake-whose-waker-parks-next-is-forwarded-to-an-idle-cpu.md, owned by S3, with spawn_cost's reading.
• The NSDI'22 row. Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's survey says choice at enqueue, on top of stealing, does not help. The forward's case here is not work conservation, which the steal already gives. It is the hops a halted idle CPU costs the steal path: a probe waits for the idle CPU's wake, and an answer waits for the victim's next pass. The forward spends one IPI at the drain instead. busy_home bounds that at two hops.
• K4, the T14's 10 ms phase-5 wake, is evidence in the S4 issue, with the mechanism the code offers: the pick serving a queued spinner whose stale key sits below the woken thread's.
• The I14 trace is re-shrunk on the merged tree with its provenance notes kept. Two marker patches show which path it takes: the forward marker fires ("place forwards the corpse, rt=false"), and the steal marker does not (run, patches). The trace says so.
• This body is rewritten for the second judgement and the ruling.

What changed, per decision

1. A busy target is kicked on the doorbell's edge (mailbox.rs, Doorbell::ring). Urgency::Normal sends on the 0→1 KICK_PENDING edge whether or not the target is SLEEPING. A target already owed a pass costs nothing more. An ordinary wake posted on the CPU it goes to rings nothing (park::deliver, new this round): that CPU's own next pass drains it, as on main. A boosted wake there still rings, and a kick aimed at the poster's own CPU sets its need_resched and sends no IPI (KernelHw::kick, SimHw::kick). Kicker::runs_on answers whose CPU the poster is: the kernel from its CPU id, the simulator from its poster.

2. At the drain, a fair task that would queue behind a running fair task goes to an idle sibling (CpuSched::place), the RT-behind-RT forward extended to the fair band. idle_sibling takes a CPU that is SLEEPING and has no kick coming, so a burst drained in one pass spreads over the idle CPUs.

3. A fair task that stays, with a key strictly below the running fair task's, cuts that task's quantum to end PROTECTED_RUN_NS (1 ms) after the running task's dispatch, or now if it has run that long. Strictly, because threads of one process share a pot and so a key.

4. One queued task is surplus (PUSH_THRESHOLD 1). The pass publishes its whole fair band, which overstates by at most the task it still stands on. It answers probes from stealable(), the band less that task. A probe that only that task could answer is kept, and the pass asks for its next pass, which answers it past the switch (Machine::need_resched's second case).

Where the build departs from the accepted design

• The dispatch time is a field (dispatched_at), because a cut moves quantum_end. a_second_owed_wake_does_not_cut_inside_the_protected_run holds it, and dispatch-from-quantum-end reds it.
• idle_sibling refuses a CPU with a kick pending. a_burst_of_wakes_forwards_one_per_idle_sibling holds it, and forward-onto-kicked reds it.
• The published surplus counts the loaded task, and a declined probe is kept (decision 4): the review's fix, plus the probe kept, accepted provisionally and upheld by round 4.

Published backing, per the ruling

The forward, the threshold-1 steal and the protected preemption follow the work-conservation, stealing and preemption results #872 cites, and their bench numbers are reported. The forward's case is the hops above, not enqueue-time choice. The busy-target kick has no cited source; K1 and the row gate it.

The simulator, loom and the harness

• New cases: a_wake_to_a_busy_cpu_beside_an_idle_one_runs_within_two_hops (busy_home), a_wake_owed_the_cpu_waits_out_the_protected_run_and_no_more (interactive_mix(1, 1)), a_sibling_that_naps_does_not_preempt_the_one_that_spins (sleep_spin_siblings).
• A late kick is counted apart (Vm::late_kick_wakes).
• Gates restaged because the kick changed their schedules: double_drop_exit_race, lend_then_block, a_pass_inside_the_registration_window_is_caught, the corpus traces, and this round main's two topology cases above.
• The merge keeps the simulator's self-route beside main's per-level IPI record, which counts only kicks that send one.
• loom_sleep models the IPI by the need_resched it sets. Under it, the doorbell-kick-relaxed control reds.
• Docs this change made false were deleted or cut back.
• Issues filed, all true of main as well:
  • interactive-mix-crosses-the-fairness-bound-at-two-cpus.md
  • the-simulators-gate-command-waits-for-overlong-pass-to-abort.md
  • the-simulator-rotates-spawn-placement-unlike-the-kernel.md
  • round 4's three and round 5's the-kernel-brings-up-at-most-eight-cpus.md, above.
• The metal row wake_onto_busy, above, rides the testcases boot and reds at a p99 ≥ 1000 µs. That bound is a tenth of the quantum a wake left behind the spinner waits. The row is in RUST_SKIP, because its verdict is a duration.

No new QEMU guest test. Every decision is reached by a SchedPass host test or a simulator case, and every timing claim belongs to metal. acpi_mediated and futex_wake_counts changed in round 1 and not since. acpi_mediated now asks from itself first when it is already off cpu0, because a thread started from its busy CPU goes to the idle one. The ACPI fixed hardware's mediated access is only a guest's to reach.

Checks

Negative control: the whole change reverted onto main. origin/main's kernel/pure and kernel/src/hw.rs were put under this head's simulator, with a shim declaring PROTECTED_RUN_NS. cargo test -p kernel-sim EXIT=101 (log, script). Six cases red:

• busy_home: "only 0 of 40 runs handed the sleeper to the idle CPU";
• the owed wake: "a second wake in one run waited 7000000 ns against 3200000 ns";
• the lopsided drain, the re-arm and the push, at their exact drain;
• the corpus traces.

The row's control is metal's, and it reds: round 3's control, the kick reverted, read wake_onto_busy p99 10,001 µs on both benches (T14, AMD, each exit 101).

Mutations at this head (results, patches and logs beside it). Each is a checked patch, applied, run and restored. Every one exits 101, red:

patch	what it breaks	red in
surplus-stealable	the review's count, reverted	sim policy (I15, drain, re-arm, push); topology
probe-not-kept	the kept probe	unit a_probe_for_the_task_the_cpu_stands_on_is_answered_a_pass_later; sim re-arm
strict-to-leq	< becomes ≤	unit a_wake_that_ties…; sim a_sibling_that_naps…
dispatch-from-quantum-end	the design's derivation	unit a_second_owed_wake…
no-fair-forward	arm 2's forward	unit a_wake_to_a_busy_cpu_is_handed…; sim busy_home
no-cut	arm 2's cut	unit a_wake_with_a_lower_key…; sim owed wake
busy-target-unkicked	the old ring	unit doorbell_kick_policy; sim owed wake; sim busy_home
lone-task-not-surplus	the old floor	unit a_lone_queued_task_is_published…
loaded-counted	the loaded task handed over	unit a_probe_for_the_task…
forward-onto-kicked	the !kick_pending term	unit a_burst_of_wakes…
sim-no-self-route	the simulator's self-kick	the corpus replays
late-kicks-uncounted	the late-kick instrument	sim the_two_cures…

No behavioural test reaches the simulator's self-route, and none reaches the kernel's (KernelHw::kick).

Mutations of round 4's fix (patches and logs, script), each applied, run and restored:

patch	what it breaks	result
self-wake-rung	the fix reverted: a self-posted ordinary wake rings and kicks	unit an_ordinary_wake_posted_on_its_own_cpu_rings_nothing EXIT=101; sim EXIT=0
boosted-self-unrung	a boosted self wake left unrung too	unit a_boosted_wake_posted_on_its_own_cpu_still_kicks EXIT=101; sim EXIT=0
every-wake-unrung	every ordinary wake unrung, wherever posted	unit EXIT=0; sim EXIT=101 (busy_home, the owed wake, the band wait, the storm, the cures)
sim-runs-on-never	the model's runs_on always false	every scenario EXIT=0 (at fa3a31a06)

self-wake-rung is green in the simulator: no scenario's verdict rests on a self-posted wake's timing, and none on the model's runs_on, a one-line accessor read at its site. The fix's control is metal's, and it reds where the line said: the T14's 64-waiter storm reading at the control is 168,679 cycles, above 74,328 (## Metal: judged).

Independent oracle: the T14 and the AMD bench; and loom, the third-party model checker, for the ordering.

Gates

At 0ab013e77 (r5/gates/, script, whose run log stamps the head and the tree clean before and after), on main b680a973b:

• cargo run -- --ci host: EXIT=0 (log).
• cargo test -p kernel-sim: EXIT=0 (log).
• cargo run -- --build-only: EXIT=0 (log).
• Guest suite, cargo test, at fa3a31a06: EXIT=0, 62 of 62 (log). Since then the branch changed one host test and two issues and merged main's x86-64 kernel leaves are global under a declared CR4.PGE, and only a kernel-half change drops them #873; the guest suite was not run on the merge.
• Round 3's loom runs (loom_sleep, loom_push, loom_mailbox) were not repeated: round 4's change to them is the runs_on each model's kicker answers false, and --ci host builds them.

Metal: judged

The orchestrator ran round 4's request on both benches; every readback image hashed to the request's sha256. Each judge is cargo test --test toyos-build -- --metal --metal-readback <readback dir> (with --metal-host <user>@<amd-bench> on the AMD bench, and the row wake_storm_cost for the control), from the tree that staged it: head 4118fbbeb, control 15d088559, the head's kernel with self-wake-rung. Exits are cargo's, read from each log's own trailer.

	T14	AMD bench
head, every row	300 passed, 0 failed, 20 boots; exit 0 (judge log, boots)	286 passed, 14 failed, 20 boots; exit 101 (judge log, boots)
control, wake_storm_cost	1 passed; exit 0 (judge log, kernel log)	1 passed; exit 0 (judge log, kernel log)

The lines, stated before any number, against what the boots read:

• storm, holds. T14: 73,484 cycles at 64 waiters at the head (kernel log), inside the 50 kick-free boots' 52,910–74,328; 168,679 at the control, above 74,328. The control's ratio row passed, as the line allowed. AMD bench, reported: 22,752 at the head (kernel log) against 26,460 at the control.
• per waiter, reported: 401 ns at the head against 1,077 ns at the control on the T14, and 386 ns at round 3's judged main base; 161 ns against 196 ns on the AMD bench, and 151 ns at that base.
• wake_onto_busy, holds: green at the head on both benches. T14 p99 5 µs, 200 of 200 run off CPU 5, the spinner kept there again 0 times; AMD bench p99 25 µs, 200 of 200 run off CPU 6, 0 times.
• every other row, holds: green on the T14; on the AMD bench red on exactly the 14 rows tabled above.

Round 3's run is published beside it: T14 head (300 of 301, red on wake_storm_cost, exit 101; its testcases kernel log), T14 control, AMD head, AMD control. The orchestrator's boot lists are m874-r4-t14.txt, m874-r4-amd.txt and m874-r3-t14.txt; masking leaves their paths as <path>. Every published file was masked with the orchestrator's mask.sh and counts 0 for the vendor, model, firmware, host and home-path names it checks.

Net lines

git diff --shortstat origin/main...HEAD: 42 files, +1,704 −393.

• The scheduler's files (git diff --numstat): cpu.rs +376 −63, of whose additions 214 are in its test module; mailbox.rs +27 −15, 8 of them tests; park.rs +51 −2, 38 of them tests; queue.rs +5; sched/hw.rs +9 −3; kernel/src/hw.rs +10 −6.
• Each test or model kicker gains a four-line runs_on.
• The simulator, loom, the rows and the harness carry the rest, and ten issues.

What I am unsure of

• The unrung self wake changes decision 1 as accepted: that kicked the poster’s own CPU, and this round stops it for ordinary wakes and files what it leaves. It was accepted in 6134663906.

• Why the self pass grew faster than the storm on metal. The share-lock reading is the guest's; the T14's split is unread.

• The kept probe answers a red the review’s fix opened. Its provisional acceptance was upheld by round 4.

• What the kick to other CPUs costs a waker. wake_storm_cost per waiter reads 401 ns at the head against 386 ns at main's judged base on the T14, and 161 against 151 ns on the AMD bench, one boot each; the T14's 64-waiter reading is inside the 50 kick-free boots' range, so one boot does not separate them. The kick's case is K1 and the row.

• How the head reached the unstaged trio on metal. The readbacks show the end state and not the path.

• The sync wake (filed).

• Whether the audio rows reach an RT wake behind running RT with only kicked sleepers.

• Late kicks on the audio workload are unmeasured on metal.

• busy_home measures only the half of its runs whose cpu1 ever published SLEEPING.

• From the design: C, a preemption's cost, is unmeasured, so M is a policy number.

Roast

1. Phases 2 and 3 with one process, where every CPU is busy and the keys tie, wait for the fair band, and metal showed the change's arms reaching that arrangement where base's rotation did not. It is filed for S4, not fixed.
2. The busy kick costs an interrupt on targets where nothing changes. At this kernel a waker's 64-waiter storm reads 401 ns per waiter on the T14 against main's 386, one boot each; round 3's 2.5× was the self kick, now gone.
3. The strict comparison's gate (sleep_spin_siblings) bounds a model granularity, not a metal number.
4. M = 1 ms rests on an estimated C.
5. The target's pass is charged to the interrupted task.
6. PUSH_THRESHOLD was the parameter of an owner decision. At 1, main's topology suite lost a tightness check and a staging (filed for S2).
7. idle_sibling is still an O(CPUs) scan, and it refuses kicked CPUs.
8. L includes the target's longest preemption-off window, recorded at 1.4 ms and 3.8 ms on the T14.
9. The key comparison is a stepping stone to the pick's own order.
10. The row is staging machinery of its own: spawn-and-check for a placement the kernel does not offer, and a spinner kept again when the balance path moves it.
11. Gates that rest on a schedule's shape were restaged twice. A scheduler change can quietly blind such a gate.
12. A thread that wakes a task parked on its own CPU and keeps running leaves it waiting for that CPU's next pass, as on main; an RT thread woken without a boost is among them. A device interrupt that posts such a wake sets need_resched before its EOI, so the CPU it interrupted runs that pass on the way out. Filed: issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

[Japabu]

Design acceptance, recorded by the orchestrator for the owner. He answered "Accept + extra phases": the design is accepted with K4, the cross-process spinner phases that exercise the preemption path, added to the judging measurement, beside K1–K3. A parallel design round on the scheduler's topology (#872) is being redone to the owner's bar of "the best current scheduler theory has to offer". Where that redesign touches place or the wake path, it says what becomes of this design.

[Japabu]

Metal judgement, both benches: the design goes back. K3 fails on both benches, in phases 2 and 3. K2 fails on the T14. K1 and K4 pass on both. The row's negative control is green on base on both benches, because the row never puts a wake on a busy CPU.

Setup. The plan and K1–K4 are in metal/request.txt, with K4 added at acceptance.

• Base is 5beb7f63a and head is b031cfa6c. Every boot's supervisor line names its tree's commit and reads clean.
• Each bench ran one boot per image per tree. Every boot's verdict is passed.
• The T14 judge exited 0 at base and at head, and so did the AMD bench's --metal-host run.

The whole logs, masked, are at cfb4aa7:

• T14 judges: base, head.
• AMD runs: base, head.
• Every boot's readback sits beside them.

Measured or derived. Every number below is measured, copied from a line in those logs, unless it is marked (d). A derived number has its arithmetic beside it.

The scout's quantiles are bucket upper bounds, eight buckets to an octave (upper() in kernel/src/scout_wake.rs). For example, "≤ 1,791" means the value lies between 1,664 and 1,791 ns (d). Scout times are in ns.

K1, the purpose: passes on both benches

Phase 1, head:

bench	base busy n, p99	base rounds	head busy n, p99	head rounds	head max, any class
T14	500, ≤ 10,485,759	500	1, ≤ 6,143	741,789 in 5,000 ms	17,885 (same)
AMD	500, ≤ 10,485,759	500	0	803,709 in 5,000 ms	16,422 (same)

• Rounds are 29.7× and 32.1× the 25,000 floor (d).
• The busy p99 rests on one sample on the T14 and none on the AMD bench.
  • At head, nearly every phase-1 wake went to the waker's own CPU. The same class has n = 1,483,577 with p99 ≤ 2,559 on the T14, and n = 1,607,418 with p99 ≤ 2,559 on the AMD bench.
  • Base split its wakes 500 sleeping and 500 busy, so the pair started on two CPUs. Head then served every later wake on one CPU. The reading is that the forward put the pair together, on the T14 after its single busy wake and on the AMD bench before the record was armed.
• What carries K1 is the rounds and the any-class maximum, both of which pass.

K2, the quiet path: fails on the T14, passes on the AMD bench

Phase 0:

bench	base rounds	head rounds	change	base sleeping p99	head sleeping p99	buckets above
T14	542,401	508,079	−6.33 % (d)	≤ 1,791	≤ 2,047	+2 (d)
AMD	616,747	616,750	+0.0005 % (d)	≤ 1,919	≤ 1,919	0 (d)

• The T14 fails both halves. Rounds are outside 5 %, and the sleeping p99 is two buckets above base, against a limit of one. By the request, the busy kick's cost is not paid for on the T14.
• The judgement rests on one boot per tree. Phase 0's DUMP ran on cpu4 at base and on cpu0 at head, so the two T14 runs did not place the pair alike. One boot each cannot separate placement from the kick's cost. The AMD bench reads identical to within three rounds.
• The kill line is applied as written.

K3, no idle CPU: fails on both benches, in both phases

bench	phase	base busy n, p99	base rounds	head busy n, p99	head rounds	reading
T14	2	1,000, ≤ 7,340,031	500	0 (same: 500, ≤ 10,485,759)	250	rounds lower: fail
T14	3	334, ≤ 16,777,215	167	336, ≤ 29,360,127	168	p99 six buckets higher (d): fail
AMD	2	1,000, ≤ 7,864,319	500	0 (same: 500, ≤ 10,485,759)	250	rounds lower: fail
AMD	3	500, ≤ 10,485,759	250	500, ≤ 15,728,639	250	p99 five buckets higher (d): fail

Phase 2 is the plain failure.

• On both benches a head round takes 20.0 ms, against 10.0 ms at base (d). That is 5,001 ms over 250 rounds and over 500 on the T14, and 5,002 ms over each on the AMD bench.
• Every head phase-2 wake was posted to the waker's own CPU and waited about one quantum. The same class's p50 and p99 are both ≤ 10,485,759. Its max is 10,003,301 on the T14 and 10,004,984 on the AMD bench. There are two such wakes per round (n = 500 over 250 rounds).
• At base the pair sat on two CPUs, and each busy wake waited a p50 of ≤ 3,407,871 on the T14 and ≤ 2,621,439 on the AMD bench.
• So head leaves the pair on one CPU behind a spinner of its own process. There the keys tie and the strict comparison cuts nothing (roast 1), so the pair pays a whole quantum per wake where base paid a part of one. How the pair came to share that CPU is not in these logs.

Phase 3 keeps its rounds. Its p99 is the 4th worst sample of 336 on the T14 and the 5th of 500 on the AMD bench (d: 1 % of n), from one boot each. Both benches moved the same way.

K4, cross-process: passes on both benches

bench	phase	head busy n	p99	max	rounds, base → head
T14	4	9,965	≤ 1,048,575	1,009,654	500 → 4,983
T14	5	9,965	≤ 1,048,575	1,000,896	500 → 4,983
AMD	4	9,966	≤ 1,048,575	999,558	500 → 4,983
AMD	5	144	≤ 4,095	7,714	167 → 738,439

• Every p99 bucket tops out under 1.1 ms.
• In the three rows near 1 ms, at least 10 % of wakes waited 983,040 ns or more (d: the p90 sits in the same bucket as the p99). That is the protected run, as designed.
• AMD phase 5 put the pair on one CPU. The same class has n = 1,471,707 with p999 ≤ 3,327, and at least one of those wakes waited a quantum (max 10,005,025). That class is outside K4.

The row, and why base is green

bench	base	head
T14	p50 5, p99 5, max 7 µs; exit 5, PASS	p50 6, p99 6, max 22 µs; exit 6, PASS
AMD	p50 4, p99 5, max 8 µs; exit 5, PASS	p50 6, p99 7, max 29 µs; exit 7, PASS

Each run is 20,000 rounds beside 7 spinners on 8 CPUs. Head is green, and so is base, on both benches: the negative control fails.

1. The base kernel lacks the change.
   • Both base boots print supervisor: build 5beb7f63a… clean.
   • git diff 5beb7f63a b031cfa6c -- kernel is exactly A wake to another CPU that is busy is drained at its next preemption point and runs within a bound #874's kernel diff: 21 files, +863 −368, the same as git diff 177e72a9e 831fbf13f -- kernel. Outside kernel/ and issues/, the two trees differ only in acpi_mediated.rs and futex_wake_counts.rs.
   • Base's mailbox.rs:483 still rings on prev & SLEEPING != 0 && prev & KICK_PENDING == 0.
   • Base's own scout boot shows the old behaviour on both benches: phase 1 has 500 rounds, with busy max 9,999,351 on the T14 and 10,000,285 on the AMD bench.
2. The row never puts a wake on a busy CPU.
   • At base, a wake to a CPU running a spinner waits for that CPU's pass, which takes up to a quantum (the maxima just above).
   • The row's base max was 7 µs and 8 µs. Not one of its 40,000 timed wakes (d: 2 × 20,000) reached a busy CPU on either bench.
   • Its premise, "each wake lands on a busy CPU", does not hold on metal.
3. Why, read from the code: derived, not traced.
   • How a spawn is placed. Every spawn, process or thread, is placed by driver.rs::placement. It takes a global rotating start, then runs CpuHandles::place's scan for the least queued CPU, in which a CPU running one thread and queuing none reads 0. On a quiet machine, a fresh process's main thread lands at the start, s, and each later spawn lands one CPU further on.
   • The row. Its 7 spinners take s+1 to s+7. Its partner, the ninth spawn, wraps to s: main's own CPU, which no spinner holds. Every wake then goes to the waker's own CPU, and any kernel serves that in microseconds.
   • The scout. Its phase 1 has the same code shape, but phase 0's partner spawn has already moved the rotation on by one. So phase 1's last spinner wraps onto main's CPU, and its partner lands alone on the free one. That is the split base printed on both benches: sleeping n = 500, busy n = 500.
   • What follows. The row's verdict rests on the spawn rotation, not on the wake path.
4. As written, the row holds nothing about A wake to another CPU that is busy is drained at its next preemption point and runs within a bound #874 and cannot serve as its negative control. Its pair has to start apart before it can red on base. Not changed here.

Verdict

	T14	AMD bench	consequence
K1	pass	pass	not dropped
K2	fail	pass	the busy kick's cost is not paid for on the T14
K3	fail (phases 2, 3)	fail (phases 2, 3)	the design goes back
K4	pass	pass	the preemption arm stands
row	head green, base green	head green, base green	the negative control fails: the row does not tell base from head

The design goes back on K3. Phase 1's purpose and phases 4 and 5 hold on both benches: rounds rise from 500 to 741,789 and 803,709 in phase 1, and from 500 or fewer to 4,983 or more in phases 4 and 5. But with every CPU busy and one process, head halves phase 2's rounds on both benches and raises phase 3's p99. The row has to be restaged before it can stand as the negative control.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

[Japabu]

The orchestrator, applying the owner's standing ruling (#885: a design the manufacturers or the published state of the art back is followed, and its bench numbers are reported, never gated; a design nothing published covers is reviewed on merits, and speed decides only when it is the whole purpose):

• The forward to an idle CPU, the threshold-1 steal and the 1 ms protected preemption follow the published work-conservation, stealing and preemption results Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872 cites. Their K2–K4 readings are reported.
• The busy-target kick has no cited source, and speed is its whole purpose. So K1 gates it, and wake_onto_busy, the kick's purpose row, stays a gate.
• The unstaged K3 regression (phase 2 halved on both benches: the pair together behind a same-process spinner, with tied vruntime keys) is a slowdown seen, so it is investigated, not used to reject. The proposal is accepted: judge the staged arrangements (phases 9 and 10) as reports, and file the unstaged regression as an issue owned by Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's S4 (ordering threads within a process), exiting at S4's re-run of K1–K4. File it on this branch.

[Japabu]

Second metal judgement, both benches: the gates pass. K1 passes on both benches. wake_onto_busy is green at head and red on base with a measured p99 on both, so the row now tells base from head. Under the ruling, K2–K4 and the staged phases 9 and 10 are reported:

• staged K3 and K4 hold on both benches;
• K2 holds on the T14, and cannot be read on the AMD bench, because every head quiet phase moved;
• the unstaged phase 2 halves again on both benches, as at the first judgement.

Setup. The plan is r2/metal/request.txt, byte for byte the one fixed before the run.

• Base is wt/toyos-busywake-judge-base at 4cbf3c381, and head is wt/toyos-busywake-judge-head at b84bd8157. Every boot's supervisor line names its tree's commit and reads clean.
  • Their kernel/ trees differ by A wake to another CPU that is busy is drained at its next preemption point and runs within a bound #874's kernel diff (21 files, +863 −368).
  • Outside kernel/ and issues/, they differ only in acpi_mediated.rs and futex_wake_counts.rs.
• Each bench ran one boot per image per tree, and every boot's verdict is passed. The four T14 images hash to the plan's sha256s.
• The T14 judges exited 1 at base and 0 at head. The AMD bench's --metal-host runs exited 1 at base and 0 at head.

The whole logs, masked, are at d63b878:

• T14 judges: base, head. Each boot's flash log and readback sit beside them.
• AMD runs: base, head. Each boot's readback sits beside them.

Measured or derived. Every number below is copied from a line in those logs unless it is marked (d), which has its arithmetic beside it.

• The scout's quantiles are bucket upper bounds in ns, eight buckets to an octave (upper() in kernel/src/scout_wake.rs). "One bucket" is one step of that table (d).
• moved counts how often a task of the pair's process was switched in on a CPU other than the one it last ran on.

Base's red is the row, at its staged placement, and nothing else

bench	base	head
T14	180 rounds; waker on APIC 3, partner beside a spinner on APIC 5; p50 10,004, p99 10,004, max 10,005 µs; exit 10004: FAIL	20,000 rounds on the same CPUs; p50 5, p99 5, max 38 µs; exit 5: PASS
AMD	180 rounds; waker on 5, partner beside a spinner on 6; p50 10,002, p99 10,003, max 10,003 µs; exit 10003: FAIL	20,000 rounds; waker on 5, partner on 7; p50 4, p99 6, max 22 µs; exit 6: PASS

• It is a measurement, not a refusal. The exit code is the p99 in µs. A refusal is negative: −5 when the row is not staged, or when the spinner left the partner's CPU.
• It is at its staged placement.
  • The row names CPUs by CPUID 01H's initial APIC id. Every partner is off the waker's CPU and off APIC id 0.
  • On the T14, APIC ids 3 and 5 are the kernel's cpu5 and cpu6, by the boot's SMP: AP lines. On the AMD bench, the ids and the CPU numbers coincide.
• Base waited one quantum every round. p50 equals p99 at about 10 ms, and 200 rounds (20 of them warm-up) filled the 2 s span (d: 2,000 ms / 200 = 10 ms a round). That is what the PR body derived.
• Nothing else is red.
  • Each judge's only FAIL is wake_onto_busy on base. scout_wake passes at both trees on both benches.
  • On the T14, every other job in the testcases boot exited 0 at both trees.
  • On the AMD bench, the same boot's test_rs_pci_reclaim exited 1 (claim 1 of 8086:15fc was refused: NotFound), and test_rs_hda_client_stall exited 101 (soundserver on the null sink). Both say the same words at base and at head. Their rows were outside this run's filter, and both are issues/a-metal-row-names-no-hardware-and-a-bench-without-it-reds.md.

K1, the gate: passes on both benches

Phase 1, 7 spinners:

bench	base busy: n, p99, max	base rounds	head busy: n, p99	head same: n, p99, max	head rounds
T14	500, ≤ 10,485,759, 9,999,390	500	1, ≤ 6,143	1,502,917, ≤ 2,559, 3,195	751,459 in 5,000 ms
AMD	500, ≤ 10,485,759, 10,000,864	500	0	1,802,220, ≤ 2,303, 15,149	901,110 in 5,000 ms

• Rounds are 30.1× and 36.0× the 25,000 floor (d), against 500 at base.
• The busy p99 rests on one sample on the T14 and on none on the AMD bench, as at the first judgement: head serves nearly every phase-1 wake on the waker's own CPU.
• So what carries K1 is the rounds, and head's any-class maximum: 5,757 ns on the T14 (busy) and 15,149 ns on the AMD bench (same). The row is the gate that tells base from head on a staged busy target.

K2, reported: holds on the T14; cannot be read on the AMD bench

Staged phases 6–8: the pair alone on two CPUs, 2 s each. Each cell gives waker/partner CPU, moved, rounds and sleeping p99.

bench	tree	phase 6	phase 7	phase 8
T14	base	1/2, 0, 368,961, ≤ 1,791	1/2, 0, 368,894, ≤ 1,791	1/2, 0, 369,054, ≤ 1,791
T14	head	1/2, 3, 372,416, ≤ 1,663	5/6, 0, 374,481, ≤ 1,663	5/6, 0, 375,189, ≤ 1,663
AMD	base	1/2, 0, 425,040, ≤ 1,791	1/2, 0, 422,782, ≤ 1,791	1/2, 0, 422,400, ≤ 1,791
AMD	head	1/2, 2, 419,462, ≤ 1,919	1/2, 1, 418,152, ≤ 1,791	1/2, 1, 417,196, ≤ 1,919

• The T14 holds.
  • Base's median is 368,961 rounds; head's, over phases 7 and 8, is 374,835: +1.59 % (d).
  • The sleeping p99 median is ≤ 1,791 at base and ≤ 1,663 at head: one bucket below (d).
  • Head staged phases 7 and 8 on cpu5 and cpu6, because the staging thread held cpu1; base staged on cpu1 and cpu2. Those are the same two cores, on the other thread of each (d: the boot's map puts cpu1 and cpu5 at APIC 2 and 3, and cpu2 and cpu6 at APIC 4 and 5; the PR body places the T14's SMT siblings four CPU numbers apart).
• The AMD bench cannot be read. Every head quiet phase moved (2, 1 and 1), so the plan's rule leaves no phase to read.
  • Beside it, not judged: head's median is 418,152 rounds against base's 422,782, −1.10 % (d), and all three head phases are below all three base phases.
  • The sleeping p99 median is ≤ 1,919 at head against ≤ 1,791 at base: one bucket above (d).
• Phase 0, natural, read beside it.
  • T14: 545,792 → 538,673 rounds, −1.30 % (d), sleeping p99 ≤ 1,791 at both trees; head moved 2.
  • AMD: 621,263 → 621,132 rounds, −0.02 % (d), ≤ 1,919 at both trees.
  • The first judgement's T14 shortfall (−6.33 %, two buckets up) does not recur on this boot.

K3 staged, reported: holds on both benches

bench	phase (woken class)	base: CPUs, n, p99, rounds	head: CPUs, n, p99, rounds	rounds	p99
T14	9 (busy)	1/2, 800, ≤ 10,485,759, 400 in 4,002 ms	5/6, 800, ≤ 9,437,183, 400 in 4,001 ms	0 %	−1 bucket
T14	10 (same)	1/1, 400, ≤ 10,485,759, 200 in 4,003 ms	1/1, 404, ≤ 10,485,759, 202 in 4,000 ms	+1.0 %	0
AMD	9 (busy)	1/2, 800, ≤ 9,437,183, 400 in 4,001 ms	1/2, 800, ≤ 7,340,031, 400 in 4,001 ms	0 %	−3 buckets
AMD	10 (same)	1/1, 400, ≤ 10,485,759, 200 in 4,001 ms	1/1, 400, ≤ 10,485,759, 200 in 4,001 ms	0 %	0

• Every phase staged with 8 spinners, and every one reads moved=0 at both trees on both benches.
• The derivation holds at both trees. Phase 9 takes 10.0 ms a round and phase 10 takes 19.8–20.0 ms (d: span / rounds), against the plan's expected 10 and 20.

Natural phases 2 and 3, read beside K3: the unstaged regression again

bench	phase	base: woken class, n, p99, rounds	head: woken class, n, p99, rounds	first judgement's rounds, base → head
T14	2	busy, 1,000, ≤ 6,815,743, 500	same, 500, ≤ 10,485,759, 250	500 → 250
AMD	2	busy, 1,000, ≤ 7,864,319, 500	same, 500, ≤ 10,485,759, 250	500 → 250
T14	3	busy, 334, ≤ 18,874,367, 167	busy, 500, ≤ 16,777,215, 250	167 → 168
AMD	3	busy, 500, ≤ 10,485,759, 250	busy, 500, ≤ 12,582,911, 250	250 → 250

• Phase 2 halves again on both benches.
  • A round takes 20.0 ms at head and 10.0 ms at base (d).
  • Every head wake went to the waker's own CPU and waited about one quantum: max 10,003,344 on the T14 and 10,004,238 on the AMD bench.
  • This is the arrangement the first judgement found.
• Phase 3 is a draw of placement, as before.
  • On the T14, head beat base this time: 250 rounds against 167, with the p99 one bucket lower (d).
  • On the AMD bench, the rounds tie and head's p99 is two buckets above (d). The first judgement had it five above.

K4, reported: holds on both benches

bench	phase	head busy n	p99	max	rounds, base → head
T14	4	9,966	≤ 1,048,575	998,856	500 → 4,983
T14	5	9,946	≤ 1,048,575	10,002,218	500 → 4,973
AMD	4	9,968	≤ 1,048,575	996,796	500 → 4,984
AMD	5	9,966	≤ 983,039	997,246	250 → 4,983

• Every p99 bucket tops out under 1.1 ms.
• One T14 phase-5 wake waited a whole quantum.
  • Its p999 is ≤ 1,048,575, so at most 9 of 9,946 wakes waited longer (d: the scout's p999 is the ⌈0.999 n⌉-th sample, the 9,937th), and at least one waited 10,002,218 ns.
  • The protected run should end such a wait about 1 ms in, and the scout does not say why this one did not. The first judgement's T14 phase-5 max was 1,000,896.

Verdict

	T14	AMD bench	under the ruling
K1	pass	pass	gate: passes
row	head green (p99 5 µs), base red (10,004 µs)	head green (6 µs), base red (10,003 µs)	gate: passes; the negative control works
K2, staged	holds (+1.59 %, one bucket below)	cannot be read: every head phase moved	reported
K3, staged (phases 9, 10)	holds	holds	reported
K4	holds	holds	reported
phases 2 and 3, unstaged	phase 2 halved; phase 3 better	phase 2 halved; phase 3's p99 two buckets up	the S4 issue

The gates pass: the busy-target kick stands on K1 and on the row.

What calls for investigation under the ruling:

• Phase 2's halving, reproduced on both benches, and the AMD bench's phase-3 p99. This is the slowdown the ruling orders filed as an issue owned by Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's S4, and these runs are more evidence for it. It is not on the branch yet: the head is still d5165c28a, and its issues are the three the body lists.
• Nothing else meets the ruling's bar of a slowdown seen. Two readings are recorded here and not escalated:
  • the single T14 phase-5 wake that waited 10.0 ms, in K4's reported arm and outside the protected run's bound. Phase 5's rounds (4,973) are within 0.2 % of phase 4's (4,983) on the same boot (d), and S4's re-run of K1–K4 will read it again;
  • K2 on the AMD bench, which cannot be read because head moved its pair at least once in every quiet phase. Its rounds are 1.10 % below base's, inside K2's 5 %.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

[Japabu]

Review of d5165c28a against .claude/agents/reviewer.md on origin/main (cc52b456c). This is round 1.

Net lines, from git diff --shortstat origin/main...d5165c28a: 29 files, +1321 −380.

• Production: +153 −72 (cpu.rs +123 −52, mailbox.rs +17 −13, queue.rs +5, sched/hw.rs +2 −1, src/hw.rs +6 −6).
• Tests, simulator, loom, the row and the harness: +1079 −308.
• Issues: +89.

BLOCKER

• issues/ — The issue the ruling ordered filed on this branch is missing. The ruling (comment 6125653705) ordered an issue for the unstaged phase-2 halving and the AMD bench's phase-3 p99. It is owned by Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's S4, and it exits at S4's re-run of K1–K4 with unstaged phases 2 and 3 no worse than base. d5165c28a adds only the three issues the body lists. This issue is owed. Without it, a regression measured twice on both benches lands unrecorded.
• tests/guest/rust-tests/src/bin/wake_onto_busy.rs:42 — The row the ruling makes the busy-target kick's gate cannot fail without the kick.
  • Its only wake onto the busy CPU is round 0, which falls inside the 20 discarded WARMUP rounds.
  • The drain that wake reaches forwards the partner to an idle CPU, whether a kick starts that drain or the spinner's quantum end does (place, cpu.rs:995-999). Every timed round after it wakes a sleeping CPU.
  • Patch to run on both benches: set kernel/pure/sched/mailbox.rs:487 back to if prev & SLEEPING != 0 && prev & KICK_PENDING == 0 (the body's busy-target-unkicked). wake_onto_busy must turn red. I expect it to stay green.
  • K1 does not cover the kick either. Its rounds come from the pair consolidated on one CPU, which the forward reaches after one unkicked quantum. Its busy-class p99 rests on one T14 sample and none on the AMD bench.
  • As staged, the row holds the forward. The kick is the one arm whose only purpose is speed, and nothing that reds without it gates it.
• kernel/pure/sched/cpu.rs:1726 — The threshold-1 steal's published surplus errs in the unsafe direction.
  • The pass publishes stealable(), which subtracts the loaded task the pass has just re-queued. From the switch onward that task is stealable, but nothing republishes until the next quantum-end pass, and that pass re-queues and hides the other task.
  • So two fair tasks alternating on one CPU publish 0 at every quantum-end pass, and a CPU running two hogs has no other pass. A CPU that goes idle after the first swap posts no probe (best_victim) and is never pushed (push_on_surplus). It stays idle beside a runnable task until some unrelated message happens to reach that CPU.
  • Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's survey backs this arm with Ipanema's work conservation, in which published state errs only by overestimating, by at most one thread. This count errs the other way.
  • The branch loosened the drain's law from assert_eq!(best, floor) to best + 1 >= floor (kernel/sim/tests/policy.rs:779, :1082, :1188) to pass the one-short drains this hole produces: 6/7, 62/63 and 62/63, the same migration counts base reached under threshold 2. a_lone_queued_task_the_cpu_stands_on_is_not_surplus (cpu.rs:3970) pins the 0.
  • Patch to run: publish self.cpu.rq.fair_len() at cpu.rs:1726, keep answering on stealable(), and restore the three assert_eq!s. They should hold with the patch and red without it.
  • origin/main now carries I15, work conservation (kernel/sim/src/invariants.rs:872 on main). Its victim predicate is fair_len() >= PROBE_WORTH_IT, where PROBE_WORTH_IT = PUSH_THRESHOLD. Merged at threshold 1, I15 judges exactly this state, and the branch's simulator evidence predates it. Merge origin/main and run cargo test -p kernel-sim.
• Evidence — No metal row other than wake_onto_busy and the scout has a verdict at this kernel, although the change rewrites the wake, spawn and balance paths every metal row runs on.
  • The second judgement flashed the whole testcases boot on both benches, but it reported that boot's other jobs by exit code only. The judges of the rows that ride it were never read:
    • wake_storm_cost, the wake path's cycle ratio;
    • spawn_cost, which covers the adopt that now forwards;
    • audio_tone and audio_idle_suspend, which answer root CLAUDE.md's rule on audible quality;
    • syscall_cost, and the rest of that boot's rows.
  • Those readbacks exist. cargo test --test toyos-build -- --metal --metal-readback <head testcases readback> judges them without touching a machine.
  • The boots the judgement did not flash are owed too.
• tests/guest/rust-tests/src/arch/cpu.rs:26 — apic_id() (leaf 01H) is a sibling of x2apic_id() (leaf 0BH) in the same module.
  • It answers the same question on every CPU either bench has, and it costs the module an allow(dead_code) (:4).
  • The stated reason, that every x86-64 CPU answers 01H, "Zen 2 included", implies the AMD bench lacks 0BH. Nothing measures that, and Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's S1 assumes 0BH on both benches.
  • Use x2apic_id(), or measure that a target lacks 0BH.

NOTE

• issues/interactive-mix-crosses-the-fairness-bound-at-two-cpus.md — This issue records INTERACTIVE_I5_SAMPLE, a gate allowance the branch added because two of its own seeds cross the bound. Crossings go from 90 to 196 of 2,000 under the change. The issue is status: open with no owner. Its mechanism, frozen insertion keys, is S4's, which is the owner the ruling names for the sibling regression.
• kernel/pure/sched/cpu.rs:935 — answering(now) is !kick_pending() || recent, so the !kick_pending() term before it makes it a dead term. Delete it. Because of it, the placement-ignores-staleness control no longer reaches idle_sibling.
• kernel/pure/sched/cpu.rs:934 — The RT forward now refuses a sleeper that has a kick pending. An RT wake behind running RT therefore queues locally even when the pending kick is only for fair work. This is an RT-latency change that nothing measures; the audio rows in the Evidence BLOCKER are where it would show.
• kernel/pure/sched/cpu.rs:942 — running_fair has one caller (place). Inline it.
• kernel/pure/sched/cpu.rs:995 — The sync-wake forward is a cost no issue records. A wakee is handed off a CPU whose running task is about to park; spawn-then-join is the same case, and acpi_mediated met it. The body names the cost (an IPI and a cold cache), but no issue records it with an owner and an exit. Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872 §3.4's "recorded, not designed" sits in an unlanded body.
• The ruling cites Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872's survey for the forward, but that survey's own NSDI'22 row says that choice at enqueue, on top of stealing, does not help. The forward's case is the hops a halted idle CPU costs the steal path. The record should say that, rather than cite work conservation alone.
• K4, T14 phase 5: one wake waited 10,002,218 ns. The mechanism the code offers is that at the cut, the pick serves a queued same-process spinner whose stale key sits below the woken task's. So the protected run bounds an owed wake only when nothing queued sits below it. This is evidence for the owed S4 issue.
• kernel/sim/corpus/old_migrate_kept_the_corpse_i14.trace — The re-cut drops the provenance tying the trace to the T14's recorded failure. Show that the new replay still migrates the corpse through place's forward, the path cpu.rs:977-982 says this gate stages, and not only through answer_steal_requests.
• PR body — "Where this stands", "Proposed for acceptance" and "Metal: requested again" predate the second judgement and the ruling.

SEND BACK

[Japabu]

Accepted: the change to decision 1 at 4118fbbeb. An ordinary wake posted on its own CPU no longer rings, as on main, and boosted wakes still kick. The T14 readings back it: 132,867-153,899 cycles at 64 waiters with the self-kick, against 52,910-74,328 without it. The wait it leaves a self-posted wake is filed for #872 S3. The metal run is next, once the bench is free.

[Japabu]

Review of 4118fbbeb against .claude/agents/reviewer.md on origin/main (5096551ca). This is round 4. The only review on record is round 1's (comment 6126386560), so its BLOCKERs are the ones judged below. Since a5e247249, this round reviews 4b70d8898, the two merges and 4118fbbeb.

Net lines (git diff --shortstat origin/main...4118fbbeb): 41 files, +1,692 −393.

• kernel/pure and kernel/src: +491 −89, of which about +273 are in-file tests and test kickers. That leaves about +218 −89 of production.
• Simulator, loom, the row and the harness: +922 −304.
• Issues: +279.
• Round 4's own production is about +20 lines: post_unrung, Kicker::runs_on, the branch in deliver, and KernelHw::runs_on. The storm reading below is the reason for them, and I accept it.

Earlier BLOCKERs

• B1, the S4 issue: CLOSED. issues/a-woken-thread-waits-out-a-sibling-spinners-quantum-when-every-cpu-is-busy.md is at the head, owned by S4, with an exit.
• B2, a row that reds without the kick: OPEN in the record.
  • The guest A/B (r3/row-smoke) read p99 1,398 µs at the head and 16,556 µs reverted. A guest's numbers decide nothing.
  • The metal control that would close this is the body's line "the control was red on wake_onto_busy as wanted". No judge log or readback for it is on ToyOSOrg/evidence: r3/metal/ holds only the request and staging.
• B3, the published surplus: CLOSED.
  • cpu.rs:1725 publishes fair_len().
  • policy.rs:778 is assert_eq!(best, floor) again.
  • cargo test -p kernel-sim exits 0 at 4118fbbeb (sim2.log).
  • The kept probe overstates the surplus by at most the one task the CPU stands on. That is the direction, and the bound, of the work-conservation result Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872 cites. It costs one self-requested pass per declined probe (cpu.rs:1947), and that pass hands the task over. I find no hole, so the provisional acceptance stands.
• B4, metal verdicts for every row: OPEN in the record.
  • The orchestrator's round-4 run settles it in substance. On the T14, the head judge reads 300 passed, 0 failed, 20 boot(s). On the AMD bench it reads 286 passed, 14 failed. Those 14 FAIL lines are, row for row, the 14 in round 3's r4/amd/head-reds.txt and in the body's table.
  • All 40 head readback images and both control images hash to the request's sha256 values.
  • None of it is published or cited in the body. The body still says "Metal: requested".
• B5, apic_id(): CLOSED. tests/guest/rust-tests/src/arch/cpu.rs keeps only x2apic_id(), and the diff no longer touches it.

The request's STORM line, from the kernel logs

• T14 control (15d088559, image 5cfacbf0…): 168,679 cycles at 64 waiters. That is above 74,328, so the line holds. Per waiter: 1,077 ns. The ratio row passed (judge exit 0), as the line said it might.
• T14 head (4118fbbeb, image 477c4ced…): 73,484 cycles at 64 waiters. That is inside 52,910–74,328, so the line holds, 844 cycles under the top on one boot. Per waiter: 401 ns, against main's 386 ns in round 3's judged base.
• AMD bench: 22,752 cycles at the head (161 ns per waiter) and 26,460 at the control (196 ns). The self kick barely shows on this bench. This is reported, not gated.
• wake_onto_busy is green on both benches:
  • T14: p99 5 µs, 200 of 200 run off CPU 5, the spinner kept there again 0 times.
  • AMD: p99 25 µs, 200 of 200 run off CPU 6, 0 times.

BLOCKER

• PR body / ToyOSOrg/evidence — The hardware evidence this round rests on is not published, and the body does not cite it. These have to be added:
  1. Round 4 head, T14: the judge command (cargo test --test toyos-build -- --metal --metal-readback …), its exit code, and judge-head.log, whole. Also every boot's boot.txt, verdict.txt, kernel.log and loader.log; the images are not needed. Machine identifiers are scrubbed to build/sourcegate.rs's shapes, as r4/amd/head-reds.txt already is.
  2. Round 4 head, AMD bench: the same, with its non-zero exit code and the 117,795-line judge log published whole, not excerpted.
  3. Round 4 control, both benches: judge-ctl.log with exit 0, and the testcases kernel.log that carries the 64-waiter line.
  4. The STORM and ROW verdicts in the body, with these numbers and links: T14 73,484 against 168,679 cycles; AMD 22,752 against 26,460; per-waiter 401/1,077 ns and 161/196 ns; wake_onto_busy p99 5 µs and 25 µs.
  5. Round 3's metal run, which closes B2: the T14 head judge log (300 of 301, red on wake_storm_cost) and the control's judge log and testcases kernel.log showing wake_onto_busy red with its measured p99.
  6. A head stamp for r4/gates/ci-host-2.log and sim2.log. ci-host-2.exit reads ci host EXIT=0 and names no commit, and run.log names fa3a31a06. The run that produced them needs a head: 4118fbbeb… line, as run.log has.
• kernel/sim/src/hw_impl.rs:247 — runs_on_answers_the_poster_alone tests a one-line accessor a reader checks, self.with(|s| s.poster == Some(cpu)). The body says it was added because sim-runs-on-never turned no scenario red. That means no simulator verdict rests on the model's runs_on, and the test guards nothing behavioural. The "EXIT=101 under it" in the body's mutation table also has no log: r4/mutations/simunit-head.log is the green arm only, and the sim-runs-on-never-*.log files are from fa3a31a06, before the test existed. Delete the test (−16 lines) and its row in the table.

NOTE

• issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md:12 — The claim that "an interrupt that wakes one parked on the CPU it interrupted" waits for that CPU's next pass is false of the tree. Every device handler that posts a wake sets need_resched before its EOI: idt/user_dev.rs:33, idt/isa.rs:12, idt/xhci.rs:10, drivers/hda.rs:170, and idt/kick.rs:7 for the counters' answer. The CPU it interrupted therefore runs a pass on the way out. Only a thread that keeps running is left waiting. Roast 12 in the body says the same thing.
• smp_roster_and_tsc_trail stays red on the AMD bench with no issue on main. Its issue arrives only with Design: the kernel sizes its per-CPU state from the firmware's CPU count #865. If this branch lands first, that red is unrecorded until Design: the kernel sizes its per-CPU state from the firmware's CPU count #865 lands.
• PR body — "Where this stands", "What metal still owes", "Metal: requested", roast 2 ("a waker's storm walk 2.5× slower on the T14") and "Whether the kick pays for itself" are false at this head. The T14 head reads 401 ns per waiter, against 386 ns on main.

SEND BACK

[Japabu]

Review of 0ab013e77 against .claude/agents/reviewer.md on origin/main (b680a973b). This is round 5. Round 4's review (comment of 2026-10-13 00:29) was at 4118fbbeb. This round reviews 0dcb56e27, 3e17f573e, d398c8543 and the merge 0ab013e77.

Net lines (git diff --shortstat origin/main...0ab013e77): 42 files, +1,704 −393. Since round 4, the branch's own side changed by −16 test lines (kernel/sim/src/hw_impl.rs), one issue edited (+6 −5) and one issue added (+27). git diff fa3a31a06 d398c8543 touches only those two issues. So the branch's kernel, simulator and loom sources are the same as at fa3a31a06.

The merge is clean. git merge-tree --write-tree d398c8543 b680a973b gives bb3c5981f, which is 0ab013e77^{tree}, so the merge resolved nothing by hand. The two sides share two files, kernel/loom/Cargo.toml and tests/toyos.rs, and #873's hunks in both apply untouched.

Earlier BLOCKERs

• Round 4 B1, the unpublished hardware evidence: CLOSED.
  • ToyOSOrg/evidence pr-874 is at d170207, whose parent is 4a6b534.
  • Items 1 and 2. r5/metal/r4/{t14-head,amd-head}/ each hold 20 boots, and each boot has boot.txt, verdict.txt, kernel.log and loader.log. The T14 judge log ends [metal] 300 passed, 0 failed, 20 boot(s). The AMD judge log is 117,795 lines, whole, and ends 286 passed, 14 failed with cargo's exit status: 1.
  • Item 3. r5/metal/r4/{t14,amd}-control/ hold judge-ctl.log (1 passed, 0 failed) and the testcases kernel.log.
  • Item 4. The body's STORM and ROW numbers match the logs:
    • wake_storm_cost: 64 waiters reads 73484 and 168679 cycles on the T14, and 22752 and 26460 on the AMD bench.
    • wake_onto_busy reads p99=5us on the T14 and p99=25us on the AMD bench.
  • Item 5, round 3. The T14 head reads 300 passed, 1 failed. Both controls carry FAIL wake_onto_busy: a round trip's p99 is 10001 µs, each with exit status 1.
  • Item 6, the head stamp. It is superseded by r5/gates/run.log: head: 0ab013e77… (0 uncommitted changes) before and after, --ci host EXIT=0 (Host: 76 step(s), all green), kernel-sim EXIT=0, --build-only EXIT=0.
  • Masking. Across r5/:
    • no home path appears;
    • no six-octet MAC appears;
    • every serial number reads "<serial>";
    • no personal-hostname shape appears;
    • the bench host is <host> or <amd-bench>.
  • The body cites every one of these files.
  • Round 1's B2 (a row that reds without the kick) and B4 (metal verdicts for every row) were open in the record at round 4, and close with these files.
• Round 4 B2, runs_on_answers_the_poster_alone: CLOSED.
  • 0dcb56e27 deletes it (−16), and git grep runs_on_answers_the_poster_alone 0ab013e77 finds nothing.
  • The mutation table's sim-runs-on-never row now records the surviving mutation ("every scenario EXIT=0 (at fa3a31a06)") and no longer claims a red.

Earlier NOTEs

• The issue's sentence about a device interrupt: closed.
  • issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md now says a device interrupt that posts a wake runs its CPU's pass on the way out, and that a thread which keeps running is what is left.
  • On x86-64 the tree bears this out: idt/user_dev.rs:33, idt/isa.rs:12, idt/xhci.rs:10, drivers/hda.rs:170 and idt/kick.rs:7 each set need_resched before apic::eoi().
• smp_roster_and_tsc_trail unrecorded: closed.
  • issues/the-kernel-brings-up-at-most-eight-cpus.md at 0ab013e77 and at Design: the kernel sizes its per-CPU state from the firmware's CPU count #865's head 2469667f0 both hash to 46ac1922…. It is byte for byte the same file, so whichever lands second adds an identical file and merges cleanly.
  • It has status: assigned, the orchestrator as owner, and an exit a grep and a metal row can read.
  • Its three MAX_CPUS sites and MAX_LOG_SHARDS are where it says.
• The body's stale sections: closed for the ones named. "What metal still owes" and "Metal: requested" are gone, roast 2 reads 401 against 386 ns, and "Whether the kick pays for itself" is replaced. Others remain; they are under NOTE below.

The guest suite on the merge. The merge queue's own run is enough here, and I raise no BLOCKER:

• The branch's side of the kernel is byte-identical to fa3a31a06, where the whole guest suite ran green, 62 of 62 (log).
• What the merge adds is x86-64 kernel leaves are global under a declared CR4.PGE, and only a kernel-half change drops them #873 as it landed on main, where guest / suite was green in its own merge group.
• The merge applied without a manual resolution, and the two sides share no production source file.
• x86-64 kernel leaves are global under a declared CR4.PGE, and only a kernel-half change drops them #873's three new guest binaries are in RUST_SKIP as metal rows, so the merge adds no guest suite member.
• What no run has measured is the combination, the busy kick's IPIs beside x86-64 kernel leaves are global under a declared CR4.PGE, and only a kernel-half change drops them #873's global leaves and INVPCID discharges, at this exact tree.
  • guest / suite is a required check (required_status_checks: host, guest / suite). Under merge_group it runs under KVM on the group's commit, whose tree contains this one. The queue's grouping_strategy is ALLGREEN.
  • That run is that measurement, on a CPU model that keeps global translations, which a TCG guest does not. A red keeps the branch off main and sends it back.
• The same reasoning covers metal. x86-64 kernel leaves are global under a declared CR4.PGE, and only a kernel-half change drops them #873's TLB change was judged on its own metal run. This branch's rows hold with wide margins (p99 5–25 µs against a 1,000 µs bound).

BLOCKER

None.

NOTE

• PR body, lines 27, 67, 117, 217 and 220 — The body says decision 1's change is "Proposed for acceptance" and is "mine, not the accepted design's", and that the kept probe is "not yet accepted". Both are false of the record:
  • decision 1's change was accepted in the comment of 2026-10-12 23:35;
  • the kept probe's provisional acceptance was upheld by round 4.
• PR body, line 3 — It says [`ba98b45`] holds this round, but ba98b45 is round 4's evidence, and this round's is d170207 and 4a6b534.
• issues/an-ordinary-wake-posted-on-its-own-cpu-waits-for-that-cpus-next-pass.md:11 — "sets need_resched before its EOI" holds on x86-64 only. On AArch64, trap.rs:242-247 sets it after irqchip::end, or returns true from EL0. The pass on the way out, which is what the sentence relies on, holds on both architectures.

LAND