The SMMUv3 comes up from the IORT with every stream aborting, and both IOMMU backends read one fault policy, which halts only for a function this kernel drives, and one address window

issues/a-function-the-smmuv3-does-not-route-panics-at-attach.md

@@ -0,0 +1,24 @@
+---
+status: open
+kind: defect
+opened: 2026-10-10
+---
+
+# A function the SMMUv3 does not route panics at attach
+
+`DeviceSpace::create` and `OwnSpace::create` take no function, so on the
+SMMUv3 (`kernel/src/arch/aarch64/smmu/domain.rs`) a domain is made for any
+caller. `attach` then looks the function up in the unit's routes, and
+`Live::stream` panics where it finds none: an enumerated function whose IORT
+route names another unit, names none, or was refused
+(`kernel/src/arch/aarch64/smmu/mod.rs`, `init`'s route loop). A gap in a
+firmware table becomes a kernel panic at `attach`, where it is a refusal at
+`create`. Nothing on QEMU's `virt` reaches it: every function there is routed
+through its one unit.
+
+**Owner.** `issues/toyos-runs-on-arm64.md`'s stage 6, whose claim through an
+SMMUv3 domain is the first caller to hand `create` a function a process names.
+
+**Exit.** `create` takes the function and refuses, by name, one the unit does
+not route, before an id is spent; `attach` has no lookup that can fail; and a
+guest test, or a host test of the refusal, reds with the refusal deleted.

issues/a-late-write-from-a-released-function-halts-the-machine.md

@@ -0,0 +1,34 @@
+---
+status: open
+kind: defect
+opened: 2026-10-10
+---
+
+# A late write from a released function halts the machine
+
+`kernel/src/pcidev/mod.rs`'s `tear_down` clears the function's ownership
+(`crate::iommu::note_user_owned(…, None)`) before its reset has quiesced what
+it already issued: Bus Master Enable is cleared first, so it starts nothing
+new, and its grants stay mapped until it is quiet, so nothing it issued lands
+in a page handed on. A transaction already in flight that faults after the
+ownership is cleared — at an address a grant never covered, or one an
+`unmap` before the release took back — is read by the fault handler as a
+function's no process drives: `owner=kernel`, and `crate::iommu::fault::conclude`
+halts the machine. One process's bug then takes the machine down, which is the
+thing a claim exists to prevent.
+
+This holds on x86-64 under VT-d today, and on AArch64 it is reachable once a
+function can be claimed through an SMMUv3 domain. No test makes a write
+outlive its function's release; QEMU's devices finish a DMA inside the access
+that starts it, so no guest of this suite can.
+
+**Owner.** `issues/toyos-runs-on-arm64.md`'s stage 6, its claim of a function
+through an SMMUv3 domain (the stage's exit: netd claims its NIC), which makes
+the AArch64 half reachable and lands before it.
+
+**Exit.** A fault on a function between its release and its reset's quiet is
+handed to the released claim's record and not to the kernel, the ownership
+cleared only once the reset is quiet; a test that makes such a write — a
+claimed function's grant unmapped while it is aimed there, its claim released
+before the write lands — sees the record and a machine still running, and
+reds with the ownership cleared where it is today.

issues/the-smmuv3-and-its-crates-declare-what-the-aarch64-kernel-also-declares.md

@@ -28,20 +28,27 @@ the two registers' offsets are `kernel/src/arch/aarch64/irqchip.rs`'s: one
 register's fields have two crates. `toyos_its::TRANSLATION_FRAME` and
 `toyos_gicv3::FRAME` are both the GIC's 64 KiB frame.
 
-**Owner.** `issues/toyos-runs-on-arm64.md`'s stage 6: the SMMUv3 unit, which
-writes these tables from `kernel/src/arch/aarch64/` beside `paging.rs`, owns
-the descriptor; the ITS, which edits `irqchip.rs` and reads `GICR_TYPER` for
-every CPU, owns the redistributor.
+**The distributor.** `toyos-its/src/lpi.rs`'s `Layout::new` decodes
+`GICD_TYPER`'s `LPIS`, `IDbits` and `num_LPIs`, and
+`kernel/src/arch/aarch64/irqchip.rs`'s `route_iommu_events` decodes its
+`ITLinesNumber`: one register's fields have two homes.
+
+**Owner.** `issues/toyos-runs-on-arm64.md`. The descriptor is stage 4's, its
+owed break-before-make ordering of a live entry's replacement, which rewrites
+how `kernel/src/arch/aarch64/paging.rs` writes an entry. The redistributor and
+the distributor are stage 6's ITS work, which edits `irqchip.rs` and reads
+`GICR_TYPER` for every CPU and `GICD_TYPER` for its LPIs.
 
 **Exit.** Each bit and the level index has one declaration that the kernel's
 tables, the loader's and the unit's are all written from, and each
-redistributor register's fields have one crate:
+redistributor or distributor register's fields have one home:
 
 ```
 rg -n 'const (VALID|TABLE|TABLE_OR_PAGE|AP_\w+|NOT_GLOBAL|INNER_SHAREABLE|OUTER_SHAREABLE|AF|PXN|UXN): u64' \
     kernel/src/arch/aarch64/paging.rs toyos-smmu/src/table.rs
 rg -n 'fn index' kernel/src/arch/aarch64/paging.rs toyos-smmu/src/table.rs
 ```
 
-print nothing, and `rg -l 'GICR_TYPER' toyos-its kernel/gicv3` names one
-directory.
+print nothing, `rg -l 'GICR_TYPER' toyos-its kernel/gicv3` names one
+directory, and `rg -l 'ITLinesNumber|IDbits|num_LPIs' toyos-its/src kernel/src
+kernel/gicv3/src` names files of one crate.

issues/the-smmuv3s-global-errors-are-read-and-never-delivered.md

@@ -0,0 +1,32 @@
+---
+status: open
+kind: defect
+opened: 2026-10-10
+---
+
+# The SMMUv3's global errors are read and never delivered
+
+`kernel/src/arch/aarch64/smmu/` reads `SMMU_GERROR` against `SMMU_GERRORN`
+only where something else brings it there: in every bounded wait on a register
+or the command queue, and after each drain of the event queue. Its own wired
+interrupt, the IORT's `GERR` GSIV, is routed nowhere and
+`SMMU_IRQ_CTRL.GERROR_IRQEN` stays clear, so an error the unit raises between
+those reads goes unseen until the next one, and one it raises on a quiet
+machine is never seen (IHI 0070 H.a §7.5 lists them):
+
+- `SFM_ERR`: the unit entered Service Failure Mode, terminating every client
+  transaction and no longer accessing its queues (§12.3).
+- `EVENTQ_ABT_ERR`: a write to the event queue aborted, and events were lost
+  with no record of them; read only after a drain, which a lost record does
+  not start.
+- `CMDQ_ERR` outside a wait: none can arise there today, since every command
+  is issued under a wait for its `CMD_SYNC`.
+
+**Owner.** `issues/toyos-runs-on-arm64.md`'s stage 6, its ITS work: it edits
+`kernel/src/arch/aarch64/irqchip.rs` and `trap.rs`, where a second SPI is
+routed and taken.
+
+**Exit.** `SMMU_IRQ_CTRL.GERROR_IRQEN` is set and the `GERR` SPI reaches a
+handler that names each active error and acknowledges it in `SMMU_GERRORN`;
+`virt_smmu`, or a test beside it, raises one — a `CMD_SYNC` behind an illegal
+command is QEMU's `CMDQ_ERR` — and reds with the routing deleted.

kernel/Cargo.toml

@@ -493,6 +493,8 @@ toyos-gicv3 = { path = "gicv3" }
 toyos-gpt = { path = "../toyos-gpt" }
 toyos-hda = { path = "../toyos-hda" }
 toyos-pci = { path = "pci" }
+toyos-phys = { path = "../toyos-phys" }
+toyos-smmu = { path = "../toyos-smmu" }
 toyos-tco = { path = "../toyos-tco" }
 toyos-tsc = { path = "../toyos-tsc" }
 toyos-ps2 = { path = "ps2" }

kernel/src/actuator.rs

@@ -157,6 +157,14 @@ actuators! {
     #[allow(dead_code)]
     psci_withheld = "psci-withheld";
 
+    /// Give the `iommu-testdev` off bus 0 no route, and once the SMMUv3 is
+    /// armed have it and the one on bus 0 write on no domain, then the one on
+    /// bus 0 on a domain at an address it maps, and there again once it is
+    /// taken back. Judged by `virt_smmu`.
+    // The SMMUv3 is AArch64's alone, so x86-64 builds an accessor it never reads.
+    #[allow(dead_code)]
+    smmu_selftest = "smmu-selftest";
+
     /// Time the same read loop on every CPU, either side of the `mov cr0` that enables caching.
     control_regs_bench = "control-regs-bench";
 

kernel/src/arch/aarch64/irqchip.rs

@@ -4,8 +4,9 @@
 //! timer's EL1 virtual timer (Arm ARM K.a, chapter D12), whose PPI the GTDT
 //! names.
 //!
-//! **What this kernel takes is SGIs and the timer's PPI, and nothing else.**
-//! No SPI is routed and no LPI exists: a device's interrupt is a message the
+//! **What this kernel takes is SGIs, the timer's PPI and the IOMMU's event
+//! SPI ([`route_iommu_events`]), and nothing else.** No other SPI is routed
+//! and no LPI exists: a device's interrupt is a message the
 //! GICv3 ITS translates, and every device that would take one is either a
 //! driver the small-kernel track moves out of the kernel or a claimed function,
 //! which the SMMUv3 of the port's stage 6 must translate first
@@ -16,7 +17,7 @@
 //! zero by the entry from EL2 the two count alike. It is level-triggered, so a
 //! handler that neither re-arms nor stops it takes it again at once.
 
-use core::sync::atomic::{AtomicBool, AtomicU32, Ordering::Relaxed};
+use core::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering::Relaxed};
 
 use alloc::vec::Vec;
 
@@ -62,6 +63,21 @@ const CTLR_GRP1: u32 = 1 << 1;
 const RWP: u32 = 1 << 31;
 /// `GICD_PIDR2.ArchRev`, bits 7:4: 3 is GICv3, 4 is GICv4.
 const GICD_PIDR2: u64 = 0xFFE8;
+/// `GICD_TYPER.ITLinesNumber`, bits 4:0: the distributor takes INTIDs below
+/// `32 * (ITLinesNumber + 1)`, and none from 1020.
+const GICD_TYPER: u64 = 0x0004;
+/// An SPI's bit in each of the word-per-32 registers, its byte in
+/// `GICD_IPRIORITYR`, its two bits in `GICD_ICFGR` and its doubleword in
+/// `GICD_IROUTER`.
+const GICD_IGROUPR: u64 = 0x0080;
+const GICD_ISENABLER: u64 = 0x0100;
+const GICD_ICENABLER: u64 = 0x0180;
+const GICD_ISPENDR: u64 = 0x0200;
+const GICD_ICPENDR: u64 = 0x0280;
+const GICD_IPRIORITYR: u64 = 0x0400;
+const GICD_ICFGR: u64 = 0x0C00;
+const GICD_IGRPMODR: u64 = 0x0D00;
+const GICD_IROUTER: u64 = 0x6000;
 
 /// A redistributor's frames: `RD_base`, then `SGI_base` 64 KiB above it.
 const GICR_CTLR: u64 = 0x0000;
@@ -92,6 +108,11 @@ const SPURIOUS: u32 = 1023;
 static TIMER_INTID: AtomicU32 = AtomicU32::new(0);
 static TIMER_EDGE: AtomicBool = AtomicBool::new(false);
 
+/// The distributor's physical base, from [`init`].
+static DISTRIBUTOR: AtomicU64 = AtomicU64::new(0);
+/// The IOMMU's event SPI; zero until [`route_iommu_events`].
+static IOMMU_EVENTS: AtomicU32 = AtomicU32::new(0);
+
 /// Wait until `done`, for at most `1 / per_second` of a second counted at the
 /// rate firmware states: the boot has no calibrated clock yet.
 fn settles(per_second: u64, what: &str, done: impl Fn() -> bool) {
@@ -159,6 +180,7 @@ pub fn init(rsdp_addr: u64) -> Gic {
     }
     let gicd = gicd.expect("GIC: the MADT names no distributor");
     let distributor = crate::mm::paging::map_mmio(gicd, FRAME, MmioPolicy::Uncacheable);
+    DISTRIBUTOR.store(gicd, Relaxed);
     let revision = distributor.read_u32(GICD_PIDR2) >> 4 & 0xF;
     assert!(revision >= 3, "GIC: GICD_PIDR2.ArchRev is {revision}, and this kernel drives a GICv3 or later");
 
@@ -277,6 +299,49 @@ pub(super) fn timer_intid() -> u32 {
     TIMER_INTID.load(Relaxed)
 }
 
+/// Take SPI `intid` on this CPU at [`PRIORITY`], edge-triggered, as the
+/// IOMMU's event interrupt: an SMMUv3's wired interrupts are edge-triggered
+/// (IHI 0070 H.a §3.18.2, §12.4), and the IORT names no trigger. Refused
+/// with the distributor's limit where it takes no such SPI.
+pub(super) fn route_iommu_events(intid: u32) -> Result<(), u32> {
+    let gicd = Mmio::new(DirectMap::from_phys(DISTRIBUTOR.load(Relaxed)), FRAME);
+    let limit = (32 * ((gicd.read_u32(GICD_TYPER) & 0x1F) + 1)).min(1020);
+    if !(32..limit).contains(&intid) {
+        return Err(limit);
+    }
+    let (word, bit) = (u64::from(intid / 32) * 4, 1 << (intid % 32));
+    gicd.write_u32(GICD_ICENABLER + word, bit);
+    settles(100, "GICD_CTLR after disabling an SPI", || gicd.read_u32(GICD_CTLR) & RWP == 0);
+    gicd.write_u32(GICD_IGROUPR + word, gicd.read_u32(GICD_IGROUPR + word) | bit);
+    gicd.write_u32(GICD_IGRPMODR + word, gicd.read_u32(GICD_IGRPMODR + word) & !bit);
+    let priority = GICD_IPRIORITYR + u64::from(intid & !3);
+    let shift = 8 * (intid % 4);
+    gicd.write_u32(priority, gicd.read_u32(priority) & !(0xFF << shift) | u32::from(PRIORITY) << shift);
+    let config = GICD_ICFGR + u64::from(intid / 16) * 4;
+    gicd.write_u32(config, gicd.read_u32(config) | 0b10 << (2 * (intid % 16)));
+    gicd.write_u64(GICD_IROUTER + 8 * u64::from(intid), toyos_gicv3::unpacked_affinity(cpu::hardware_id()));
+    gicd.write_u32(GICD_ICPENDR + word, bit);
+    IOMMU_EVENTS.store(intid, Relaxed);
+    gicd.write_u32(GICD_ISENABLER + word, bit);
+    Ok(())
+}
+
+/// Pend the IOMMU's event SPI again, for records its handler left behind: the
+/// unit raises none for them. Taken once the handler's `end` deactivates it:
+/// the write makes an active SPI active and pending, which is never signalled,
+/// and deactivation leaves it pending (IHI 0069D §8.9.16, §4.1.2, §4.1.1).
+pub(super) fn pend_iommu_events() {
+    let intid = IOMMU_EVENTS.load(Relaxed);
+    assert!(intid != 0, "GIC: the IOMMU's event SPI is pended before it is routed");
+    let gicd = Mmio::new(DirectMap::from_phys(DISTRIBUTOR.load(Relaxed)), FRAME);
+    gicd.write_u32(GICD_ISPENDR + u64::from(intid / 32) * 4, 1 << (intid % 32));
+}
+
+/// The IOMMU's event SPI, or zero where [`route_iommu_events`] routed none.
+pub(super) fn iommu_events() -> u32 {
+    IOMMU_EVENTS.load(Relaxed)
+}
+
 /// Write `ICC_SGI1R_EL1` whole: the SGI it names is raised, after every store
 /// before it. Only a `DSB` orders a system register write after stores to
 /// Normal memory, so without it a target could take the SGI before the store

kernel/src/arch/aarch64/mod.rs

@@ -23,7 +23,6 @@ pub mod entropy;
 pub mod entry;
 pub mod fpu;
 pub mod hw;
-pub mod iommu_unit;
 pub mod irqchip;
 pub mod keyboard_controller;
 pub mod paging;
@@ -33,12 +32,15 @@ pub mod pmu;
 pub mod power;
 pub mod psci;
 pub mod rtc;
+pub mod smmu;
 pub mod smp;
 pub mod switch;
 pub mod tlb;
 pub mod trap;
 pub mod watchdog;
 
+pub use smmu as iommu_unit;
+
 /// The machine every program image this kernel loads must be built for.
 pub const ELF_MACHINE: toyos_elf::Machine = toyos_elf::Machine::Aarch64;