The scheduler simulator runs at real machine shapes and holds work conservation as an invariant

[Japabu]

Stage S0 of the scheduler design in #872, accepted in #872 (comment). The simulator runs every scenario at the shapes of real machines, up to 768 CPUs, and reports what each run did per level of the cache tree. Work conservation becomes invariant I15, checked after every step and bounded in protocol rounds. No kernel source changes, and nothing here is in the image.

What changed, per decision

1. The machine is a shape (kernel/sim/src/topology.rs).
   • A Shape is each dense CPU's x2APIC id plus the SMT, LLC and package widths, as a CPU reads them from leaf 0xB and the deterministic cache leaf. Two CPUs share a level when their keys at it are equal.
   • Scenario::topology replaces Scenario::cpus as the machine's one description; cpus() derives from it.
   • topology::shapes() holds four machines:
     • the T14 as its scout read it: 4 cores × 2 threads under one L3, x2APIC ids 0,2,4,6,1,3,5,7, so SMT siblings are four ids apart;
     • the AMD bench as its scout read it: 2 CCXs × 4 cores × 2 threads, one L3 per CCX, APIC ids 0..15, siblings adjacent;
     • 4x8x2, listed under a seeded permutation;
     • 2x24x8x2: 768 CPUs, two packages of 24 eight-core LLCs, also permuted. The design writes it as 48 LLCs × 8 × 2. The two packages are the research note's two-socket example, and they give the machine level something to count.
   • A scenario that names no shape runs on Shape::flat(n): n single-thread cores in one LLC.
   • Departure from the design's text. §6 sends the keys through Topology::from_keys. That is S1's and does not exist yet, so S0 compares keys in the simulator. The debt is recorded in the track S1 builds, issues/placement-is-blind-to-caches-and-topology.md: the stage that lands Topology::from_keys and the kernel's CPUID decoder moves Shape::level and the oracle test's scouted decoding onto them in the same change.
2. Counts per level. Every run reports four counts by the lowest level the two CPUs share (same CPU, core, LLC, package, machine): migrations, wakes placed, failed steals and IPIs. It also reports the most rounds a CPU spent idle beside stealable work and the most rounds a ready task waited to run. The trace is drained once a step, and I14 reads the same events.
   • A wake is a task taken out of its CPU's parked set, by a message, its deadline or a retire, including one parked in the same step. It is placed where the waking pass left it, read off its state word. Only the acting CPU's step changes its parked set, so the count snapshots that one CPU's set before the step.
3. Protocol rounds (Vm::round). A round opens on every CPU that owes a pass: awake, or halted with an IPI on its way. It completes when each has passed once. Rounds are counted in passes, not time, so a CPU whose pass the explorer defers holds its round open, and a CPU rung before a round opens holds that round open until its pass.
4. I15, work conservation, in its bounded form. No CPU spends more than cpus + 2 rounds idle while another CPU holds a task a probe would take (fair_len >= PUSH_THRESHOLD), under the shipped push.
   • Why the bounded form. The design's safety form turns on whether a victim's push rang a parked CPU. A ring on a doorbell whose edge is already raised looks exactly like no ring from outside the core. The outcome is visible, and every failure the safety form names leaves it standing: an idle CPU nobody rings, one that is rung and declines to probe, a probe nobody answers.
   • Nothing pauses the count. A ring or a task on its way lands after an interrupt's latency, which the rest of the machine could fill with rounds that take no time. Decision 3 answers that: the next round waits on the rung CPU, so an answer in flight costs at most the round it was sent in. A CPU that is rung and declines to probe accrues every round it stays idle, however often it is rung.
   • The bound. cpus + 2 is derived for one victim in work_conservation_bound's doc:
     • cpus rounds until the idle CPU is rung: the round open when it began counting, and cpus − 1 passes of the victim's cursor;
     • one round for the idle CPU's probe pass, which the ring holds the next round open for;
     • one round for the victim's answer, whose post holds the round after it open until the idle CPU has taken the task.
   • Several victims. Nothing derives a bound for them. That is filed as issues/the-push-bounds-an-idle-cpus-wait-only-for-one-victim.md, whose exit is S2.
   • Other policies. Pull, None and the bounded re-arm carry no bound. They are the controls policy.rs prices the push against, and they claim no work conservation.
   • Departure: no per-level patience. Every level's patience is zero until S3 sets the cross-LLC one from R5, so today it would be a parameter with one value. It joins with the rule that uses it.
5. A model gap the shapes exposed. An awake CPU with nothing loaded could have its idle pass deferred by the explorer while the clock ran.
   • At every shape wider than one CPU, rt_saturated_retire failed I14: 16 of 100 seeds at the T14, 24 of 100 at the AMD bench, 5 of 20 at 4x8x2 and 1 of 2 at 768 CPUs. M3 below reproduces exactly these at the head. In the run dumped, the zombie of the killed worker stayed unreleased on a CPU whose idle pass PCT deferred for 59 ms. At one CPU the real-time thread and the corpse share the CPU, which hid it.
   • Such a CPU now owes its pass, like an unserved need_resched. This applies under the new protocol only: the old protocol's idle loop is its steal. Removing that condition turns old_steal_port_is_caught red, EXIT=101, with {"I1": 154, "I6": 154} and no I8 (log), so the condition stays.
   • The fix renumbers old_migrate_kept_the_corpse_i14.trace, which is re-shrunk to the same violation and the same forward.
6. Wasted Cores' group imbalance, staged. Mutation M2 (below) left the whole suite green, so per the mutation rule it became a test. scenarios::group_imbalance places the AMD bench's tasks with a new PlacementShape::Listed adversary: cpu8 idle, the rest of its CCX each one task deep, and cpu0 two deep in the other CCX.
7. A declined probe, staged. SurplusShape::Withheld overwrites each pass's published surplus with zero after its push rang a sleeper on the real one, so the CPU it rang wakes, reads nothing worth a probe and sleeps again. It is the defect two spellings of one inequality write (best_victim's own doc), staged from outside the core, and the committed form of the review's run.
8. Cost. The probe-gap instrument walks the machine once a step instead of once per CPU, which makes 768 CPUs affordable.

Checks

Independent oracle: real hardware. a_benchs_shape_is_what_its_scout_read decodes the scouts' recorded CPUID words, committed verbatim as kernel/sim/tests/scout/*.log from #865's measurement (#865 (comment), staged from 042015035).

• From each CPU's words it reads the x2APIC id, leaf 0xB's SMT and package widths, and how many ids share the highest data or unified cache.
• It holds both bench shapes to them, CPU by CPU.
• The AMD bench's eight CPUs that never came up are read from its MADT, under the widths the eight that ran reported.

Negative gates, committed.

• a_push_that_waits_for_more_than_a_probe_takes_is_caught sets the push one above the probe's threshold (PUSH_THRESHOLD + 1) on three threads spawned onto cpu0. I15 alone catches it in 4 of 20 schedules at each bench. The shipped threshold on the same machine holds on all 20.
• a_cpu_that_is_rung_and_declines_to_probe_is_caught runs the same machine under SurplusShape::Withheld at Shape::flat(2), the T14 and the AMD bench. Every one of the 20 schedules at each fails, on I15 alone. The published surplus on the same machines holds on all 20.

The review's case (logs). The review's best_victim filter (.filter(|&cpu| cpu.0 > self.cpu.id.0)), on lopsided_placement(2, 3, 60 ms) at Shape::flat(2), over the gate's 20 seeds:

• at 6d8c741e4, where the count paused for a kick in flight: green on 20 of 20, worst 1 round (log). The shipped push gives the same (log). I15 was blind there.
• at 26a61ebaa: EXIT=101, red on 20 of 20, each I15: cpu1 has been idle for 5 rounds while cpu0 holds 2 queued fair task(s) (bound 4 rounds) (log). Without the filter: EXIT=0, green on 20 of 20 (log).
• The filter over the whole suite: EXIT=101 (log). A CPU may then probe only higher-numbered CPUs, so policy.rs, the fairness storms and all four I15 cases also red.

Mutations. Each is applied as a checked patch, run, reversed, and the tree shown as it was (patches, driver, summary). The head column is 26a61ebaa; 8b597a5a5 adds only #867's merge, which touches nothing under kernel/pure/ or kernel/sim/. The whole change reverted is the base 131970ac8, the branch's merge-base in round 1, under the same patches.

mutation	at the head 26a61ebaa	at the base 131970ac8, the whole change reverted
M1: a locality rule in best_victim that probes only inside a CPU's group of four ids	EXIT=101: I15 in the_push_keeps_an_idle_cpu_beside_work_inside_its_bound and an_idle_cpu_whose_ccx_looks_busy_is_reached, plus policy.rs's three lopsided recoveries	EXIT=101: policy.rs's three lopsided recoveries (log)
M2: Wasted Cores' group average, which probes the other half of the machine only if its average surplus is the higher	EXIT=101: an_idle_cpu_whose_ccx_looks_busy_is_reached alone, I15	EXIT=0 with no staging, and EXIT=0 with the same machine staged: unseen (logs)
M3: an awake idle CPU owes no pass (decision 5 reverted)	EXIT=101: I14 at the wider shapes (46 failures), the re-shrunk corpus trace, and the declined-probe gate, undetected in a schedule at the T14	—
M4: a CPU an IPI is on its way to holds no round open (decision 3's second half reverted)	EXIT=101: I15 falsely red on 3 of 500 wakeup_storm seeds at two CPUs, 5 rounds against 4	—

Head logs: M1, M2, M3, M4. M2 at 99033d06c, the green that made the staged machine a test: history/.

The wake count, cross-checked against the trace. A throwaway patch asserts, at every step, that the parked-set count equals the trace's own: one TimerFire per deadline wake and one Wake per task that was parked when the step began or parked in it. Over topology, scenarios and policy: EXIT=0 (log, patch).

The level of a pair. a_pair_is_at_the_lowest_level_it_shares reds on the old comparison (left: None, right: Some(Core)).

No new dependency. No guest test is added or changed, and none reaches the simulator.

Gates

At 8b597a5a5, the head:

• cargo test -p kernel-sim --no-fail-fast: EXIT=0. log
• cargo run -- --ci host: EXIT=0, 76 steps all green. log
• cargo run -- --build-only: EXIT=0. No file the image is built from changes. log

At 26a61ebaa, before #867's merge: cargo test -p kernel-sim --no-fail-fast EXIT=0 (log); cargo run -- --ci host EXIT=0, 76 steps all green (log); cargo run -- --build-only EXIT=0 (log); cargo test -p kernel-sim --test topology -- --nocapture --test-threads 1 EXIT=0, 18.37 s user with the host at load 67 (log).

Measurements

What the shipped scheduler did at each shape over every scenario, summed over 100 seeds per scenario at the benches, 20 at 4x8x2 and 2 at 768 CPUs (log). Each column is the lowest level the two CPUs share, so "package" is a crossing between LLCs of one package and "machine" a crossing between packages.

shape	count	same CPU	core	LLC	package	machine
T14	migrations	0	3	45	0	0
	wakes placed	11385	0	0	0	0
	failed steals	0	32	240	0	0
	IPIs	24	482	5281	0	0
AMD bench	wakes placed	12244	0	0	0	0
	IPIs	44	1504	4266	1475	0
4x8x2	wakes placed	2466	0	0	0	0
	IPIs	4	23	200	1275	0
2x24x8x2	wakes placed	246	0	0	0	0
	IPIs	0	0	1	59	71

Rows not shown are zero. Every wake is placed on the CPU it parked on: today's only wake placement elsewhere is the real-time forward, and no run in the table took it.

I15's quantity, the worst rounds idle beside stealable work over 20 schedules (log, from a throwaway test not committed, patch, run at 0a4dc5f56, whose kernel/sim/ and kernel/pure/ the head's are):

shape	lopsided, one victim	storm, many	interactive, many	bound
T14	8	2	3	10
AMD bench	16	4	5	18
4x8x2	—	17	6	66

Growth

git diff --shortstat origin/main...HEAD at 8b597a5a5: 18 files changed, 1101 insertions, 92 deletions.

• No shipped code changes.
• The simulator's sources: +702/−80.
• Its tests, fixtures and corpus: +366/−12, of which the scouts' recorded words are 81.
• The issues: +33: the filed one-victim issue, and S1's debt in the track.
• The topology suite adds about 18 s of CPU to the host gate, measured at load 67.

Unsure

• One victim. I15's bound is derived for one victim only. No schedule crossing it has been found with several, but nothing rules one out (the filed issue).
• Where I15 has teeth. The 768-CPU shape runs 2 seeds per scenario. At the shapes wider than the benches, all()'s workloads give each task a CPU of its own: the worst idle stretch there is 0 rounds. I15's teeth are the bench-shape cases and the two-CPU gate. The lopsided machine at 64 CPUs takes 18 s per 20 seeds and stops at its scenario's step ceiling, so it is not in the suite.
• A wake stolen in its own pass. A task woken and then handed to a thief by the same pass's steal answer is counted as placed on the thief, since that is where the waking pass left it.
• policy.rs's numbers moved in round 1. Closing the model gap changed the explorer's step relation, and policy.rs's printed measurements moved inside their asserted bounds (comparison). One figure its doc tables quote moves: interactive cpus=2 hogs=16's worst per-run runner-up goes from 30 ms to 22 ms. Other rows of those tables already disagree with the base's own output (the push's 748 idle wakes on wakeup_storm(8,64) against the base's 667). This change does not retake them. Round 2 changes no step relation: rounds are an instrument the explorer does not read.

Roast

1. The bound is linear in width. At 768 CPUs, I15 tolerates 770 rounds idle beside work under the shipped push. That is the honest bound of a flat push that rings one sleeper per pass; S2's nearest-first push and two-choice victims are what tighten it.
2. The safety form is not checked step by step. The design states it, and its exemptions are invisible from outside the core. The bounded form catches every failure the safety form names, but rounds later.
3. A round now waits on IPI delivery. A round lasts as long as the slowest rung CPU takes to pass. That is right for a bound in protocol steps, but it means I15's rounds stretch in time whenever rings are in flight, and a bound in rounds says nothing about time; that is metal's.
4. A predicate read from outside the core. I15 reads stealable work as fair_len >= PUSH_THRESHOLD. A wake to another CPU that is busy is drained at its next preemption point and runs within a bound #874 changes what the core counts as stealable. The predicate follows the constant, not a change in what is counted, so A wake to another CPU that is busy is drained at its next preemption point and runs within a bound #874's merge must change this predicate with it.
5. The step relation moved in round 1. The model fix changes it for every new-protocol scenario. One corpus trace was re-shrunk, and policy.rs's numbers moved (above).
6. Two adversaries for one scenario each. PlacementShape::Listed exists for group_imbalance alone, and SurplusShape::Withheld for the declined-probe gate alone. Each is the one way to stage what no workload produces, and without them the group-average mutation and the two-CPU decline land unseen.
7. S1 meets a duplicate. The oracle test decodes four CPUID fields itself and Shape::level compares keys itself. The track now says S1 moves both onto its own decoder and Topology::from_keys.

🤖 Generated with Claude Code

https://claude.ai/code/session_015xhaydXCVRNscoccHYFUWy

[Japabu]

Review of #882 at 6d8c741e4 against .claude/agents/reviewer.md on origin/main (528b9a6c2). This is the first round.

Net size: +1027/−90. No shipped code changes. The simulator's source is +663/−78, its tests, fixtures and corpus +338/−12, and the issue +26.

Controls weighed

• The bench shapes against Design: the kernel sizes its per-CPU state from the firmware's CPU count #865. kernel/sim/tests/scout/t14.log and amd-bench.log are byte-identical to the lines in Design: the kernel sizes its per-CPU state from the firmware's CPU count #865 (comment). I decoded the words by hand:

  • T14: leaf 0xB gives an SMT shift of 1 and a package shift of 4. The leaf 4 L3 (1c03c163) is shared by 16, so its shift is 4. The ids are 0,2,4,6,1,3,5,7.
  • AMD: leaf 0xB gives shifts of 1 and 7. The 0x8000001D L3 (1c163) is shared by 8, so its shift is 3. The ids are 0..15.

  Both match Shape::t14() and Shape::amd_bench(). The second CCX is inferred from the MADT, as Design: the kernel sizes its per-CPU state from the firmware's CPU count #865 marks it (D).

• The committed threshold+1 gate. It is caught in 4 of 20 schedules at each bench, all of them I15, and the shipped threshold holds on all 20. The base has no threshold-drift gate, and the 16 schedules that pass satisfy every other invariant, so I15 alone is what sees the drift.

• M1, the locality rule. It is red at the base too, on policy.rs's three lopsided recoveries. It shows that I15 also fires; it does not show that I15 adds detection.

• M2, the group average. The base, staged and unstaged, exits 0. The head exits 101 on I15 alone (cpu8 idle 19 rounds against a bound of 18). This is S0's control, and it is held as a mutation run, not as a committed gate.

• M3, the model-fix revert. It gives 46 I14 failures (16, 24, 5 and 1 by shape, as the body says) plus the corpus replay. The fix matches kernel/src/sched/driver.rs:678, whose idle loop passes before it halts.

• M4, the pause revert. It gives 3 false reds in 500 two-CPU wakeup_storm seeds. That makes the pause load-bearing, and it is the mechanism of the BLOCKER below.

• Not raised: policy.rs's stale doc-table figure. Under Prose, a count in a source comment is not raised.

BLOCKER

• kernel/sim/src/invariants.rs:913 — I15 is blind on any machine where the idle CPU is the only sleeper and the victim is the only CPU the round waits on, which includes every two-CPU machine.
  • Why. I15 pauses for any kick pending at the step that completes a round. The victim's push pokes in the same pass that completes that round, so the poke is in flight at every round boundary and the idle CPU's count never moves. The doc's claims at :884-886 and :892-893 are therefore false there: a CPU that is rung and declines to probe, or whose probe is never answered, does not "still accrue". No case in the suite runs a decline on such a machine.
  • Patch. In kernel/pure/sched/cpu.rs best_victim, add .filter(|&cpu| cpu.0 > self.cpu.id.0) after the answering filter.
  • Run. scenarios::lopsided_placement(2, 3, 60 * MS) under the shipped push over the gate's 20 seeds, for example with Shape::flat(2) in a_push_that_waits_for_more_than_a_probe_takes_is_caught's shape loop and this patch in place of the threshold.
  • What must happen. I15 must go red. My reading is that it stays green on every seed: cpu1 halts unprobed, and cpu0's quantum-end pass both completes each round and pokes cpu1.
  • If it stays green. Re-derive the pause, for example by counting a round in which the CPU was idle and unrung at any step, or by pausing only for a message in flight and not for a bare poke. Then re-derive the bound, and commit this run as a gate.

NOTE

• kernel/sim/src/vm.rs:1115 — "wakes placed" does not count wakes.
  • It counts every TraceKind::Wake, and handle_retire emits one for a queued task it moves into the dying list (kernel/pure/sched/cpu.rs:1087). That task never parked.
  • A deadline wake traces TimerFire, so it is never counted.
  • So the column, and the body's "8373 at the T14" and "8791 at the AMD bench", count something other than what LevelCounts::wakes says it counts.
• kernel/sim/src/vm.rs:1031 — the Protocol::New exemption rests on an unmeasured claim, that "old_steal_port's I8 needs that freedom". Post cargo test -p kernel-sim --test scenarios old_steal_port with the condition removed. If it is red, the condition stays; if it is green, the condition goes.
• kernel/sim/src/topology.rs:200 — S1's debt is recorded only in the pull request body. Departure 1, together with kernel/sim/tests/topology.rs's own CPUID decoding, leaves S1 owing two replacements: Shape::level's key comparison and scouted's decoding, both to give way to Topology::from_keys and S1's decoder. Only this body's roast 6 records that. Record it where S1's implementer will read it, with that exit, or S1 lands a sibling decoder.

SEND BACK

[Japabu]

Review of #882 at 8b597a5a5 against .claude/agents/reviewer.md on origin/main (867ec23ed). This is round 2.

Scope. It covers 6d8c741e4..0a4dc5f56 and two merges of main, 26a61ebaa and 8b597a5a5. Neither merge has a conflict hunk, and neither touches kernel/sim/, kernel/pure/ or either issue file.

Size. Net +1101/−92 against origin/main. No shipped code changes.

• The simulator's source: +702/−80.
• Its tests, fixtures and corpus: +366/−12.
• The issues: +33.
• Since round 1: +140/−68. The pause this round replaces is deleted.

Round 1's BLOCKER

• kernel/sim/src/invariants.rs:899, I15 blind at two CPUs: CLOSED.

  • The review's case. The review's best_victim filter, on lopsided_placement(2, 3, 60 ms) at Shape::flat(2), over the gate's 20 seeds:

    • At 6d8c741e4: EXIT=0. Green on 20 of 20, worst 1 round, the same seed for seed as the shipped push.
    • At 26a61ebaa: EXIT=101. Red on 20 of 20, each I15: cpu1 has been idle for 5 rounds … (bound 4 rounds). The shipped push is green on 20 of 20.

  • The mechanism, read in the code. open_round (kernel/sim/src/vm.rs:1093) now owes a round to a halted CPU with pending_ipi > 0, and I15 adds every completed round with no pause.

    • At two CPUs, the victim's ringing pass completes round k, and round k+1 waits on the rung CPU's own pass. So every round the CPU stays idle counts against it.
    • M4 (the hold removed) gives false reds: 3 of 500 wakeup_storm seeds in every_scenario_survives_a_seed_sweep. The hold is load-bearing, and it is gated.

  • The committed gate. a_cpu_that_is_rung_and_declines_to_probe_is_caught stages the decline outside the core: publish_surplus(0) runs after leave_pass (vm.rs:1432), so the push still rang on the real surplus.

    • It is green at the head.
    • Under M3 it goes red, with "t14: a declined probe went undetected", so it can fail.
    • My reading is that it also goes red with round 1's pause put back. The ring is delivered within its IPI latency, long before the victim's next quantum-end pass, so every round would complete with a kick pending.

  • The bound, re-derived. I read its three terms against push_on_surplus (kernel/pure/sched/cpu.rs:2105):

    • the cursor moves forward and reaches a sleeping p within cpus − 1 passes;
    • a ring in flight holds the next round open;
    • the answer's post leaves p owed in the round after.

    Measured worst: 8 against 10 at the T14, and 16 against 18 at the AMD bench. the_push_keeps_an_idle_cpu_beside_work_inside_its_bound holds both.

Round 1's NOTEs

• kernel/sim/src/vm.rs:1109, "wakes placed": CLOSED.
  • It now counts tasks leaving the acting CPU's parked set, placed by the state word.
  • The throwaway check against the trace's TimerFire and Wake events: EXIT=0 over topology, scenarios and policy.
  • A parked task that is retired leaves through handle_retire's wake into the dying list, so its state is Ready or Running, and the unreachable! arm holds.
• kernel/sim/src/vm.rs:1037, the Protocol::New condition: CLOSED. With it removed, old_steal_port_is_caught exits 101 with {"I1": 154, "I6": 154}, so it stays.
• S1's debt: CLOSED. It is recorded in issues/placement-is-blind-to-caches-and-topology.md. The exit can be checked: the stage that lands Topology::from_keys and the kernel's decoder moves Shape::level and scouted onto them.
  • Ruling on that edit: allowed. Design: the scheduler places by the machine's cache tree, orders fair work by EEVDF and reserves real time, in stages #872 §4 closes that track at S3, and this branch is its S0. It is the issue the branch takes up, which root CLAUDE.md's Prose lets a pull request edit.
  • The paragraph states a constraint on a later stage. It is not a design or a stage table, so it is within issues/README.md's rule for a track.

Gates at the head (ToyOSOrg/evidence@3589d062, round2/final/; each log opens with 8b597a5a5):

• cargo run -- --ci host: EXIT=0, 76 steps all green, with the new topology tests among them. The reds inside the log are the CI's own controls reaching their verdicts.
• cargo test -p kernel-sim --no-fail-fast: EXIT=0.
• cargo run -- --build-only: EXIT=0.

BLOCKER

None.

NOTE

None.

LAND