Observed
Advisory job gpu-quick (67-C1, PR #3095 inside the 0.67 train #3127), run 34639757080 on gx10-eph (GB10 sm_121, unified memory), step "aprender-gpu cuda unit tests (--features cuda --lib --release)":
test driver::memory_fuzz_tests::adversarial::test_alloc_oversize_100gb ... FAILED
thread 'driver::memory_fuzz_tests::adversarial::test_alloc_oversize_100gb' panicked at crates/aprender-gpu/src/driver/memory_fuzz_tests/adversarial.rs:36:13
test result: FAILED. 2603 passed; 1 failed; 12 ignored
2603/2604 cuda-featured tests pass on the GB10 — the one red is the oversize-allocation adversarial case.
Why (measured earlier, memory feedback_oversize_alloc_on_unified_memory_ooms_the_host)
On the GB10 the host and device share one 119 GB unified pool: a cuMemAlloc of 100 GB does NOT fail the way it does on a discrete 24 GB card — the driver backs it with host pages (and at ~2× physical it OOMs the host). The test asserts the discrete-GPU behaviour (allocation refused), so it can only pass on a discrete card.
Ask
Make the assertion device-aware: read the device's memory model (unified vs discrete, e.g. cuDeviceGetAttribute(CU_DEVICE_ATTRIBUTE_INTEGRATED) / total memory) and on a unified-memory device assert the safe property instead (the allocation is refused OR bounded below a fraction of physical, never attempted at 100 GB). Also mark it so gpu-quick on gx10 stays informative: this is the only red in the job, and it hides any second regression.
Milestone 0.68.0 (gpu-quick is advisory in 0.67). Refs #3095, #3062.
Observed
Advisory job
gpu-quick(67-C1, PR #3095 inside the 0.67 train #3127), run 34639757080 ongx10-eph(GB10 sm_121, unified memory), step "aprender-gpu cuda unit tests (--features cuda --lib --release)":2603/2604 cuda-featured tests pass on the GB10 — the one red is the oversize-allocation adversarial case.
Why (measured earlier, memory
feedback_oversize_alloc_on_unified_memory_ooms_the_host)On the GB10 the host and device share one 119 GB unified pool: a
cuMemAllocof 100 GB does NOT fail the way it does on a discrete 24 GB card — the driver backs it with host pages (and at ~2× physical it OOMs the host). The test asserts the discrete-GPU behaviour (allocation refused), so it can only pass on a discrete card.Ask
Make the assertion device-aware: read the device's memory model (unified vs discrete, e.g.
cuDeviceGetAttribute(CU_DEVICE_ATTRIBUTE_INTEGRATED)/ total memory) and on a unified-memory device assert the safe property instead (the allocation is refused OR bounded below a fraction of physical, never attempted at 100 GB). Also mark it sogpu-quickon gx10 stays informative: this is the only red in the job, and it hides any second regression.Milestone 0.68.0 (gpu-quick is advisory in 0.67). Refs #3095, #3062.