Repository navigation
hosted_clang_hello: ToyOS's own clang and ld.lld compile and link a C program in a guest, byte-identical to the host's, on demand - #830
Conversation
…loader refuses both at spawn M2a of issues/toyos-builds-itself.md. A machine test stages the clang and ld.lld that --hosted-clang builds, clang's resource headers, the C sysroot's C headers and libtoyos_c.a, hello.c, and what the host's clang and LLD (same LLVM commit) made of it, on the ROOT of a boot of its own (tests/hostedclangcase). ROOT, because the loader reads it into memory whole and the guest reads it with no disk or server behind it; 265 MB here (0xfd00000), in a 4 GiB guest. The guest job runs clang -c with the host's arguments, ld.lld directly on the line the ToyOS driver gives it (clang -### on the host), runs the result, and compares both products with the host's byte for byte. It is red: the kernel's loader refuses both binaries at spawn, each with a table past one kernel allocation (MAX_HEAP_ALLOC, 2,093,056 bytes): spawn: /system/share/hosted-clang/bin/clang: DT_RELASZ declares 5044488 bytes, past one kernel allocation spawn: /system/share/hosted-clang/bin/ld.lld: DT_STRSZ declares 2813680 bytes, past one kernel allocation issues/the-program-loader-refuses-clang-and-lld-whose-elf-tables-pass-one-kernel-allocation.md records it with what each binary declares. hostedclang::ensure is what dispatch did, so the harness makes the product the way --hosted-clang does; resolve is private now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The host-tools issue takes main's rows with the hosted clang's mentions folded into its git and cc rows; ALL_CONFIGS, RUST_SKIP, MACHINE_TESTS and the dispatch keep both sides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
main took sha2 out of the build crate; the digest is toyos_sha2_hw's, printed as hex the way tests/toyos.rs prints one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Review of #830 at Net lines (
Evidence at the head:
BLOCKER
NOTE
SEND BACK |
…hes it like any other Whole-name matching for ON_DEMAND went beyond the owner's ruling, which asks only that CI's unfiltered run not boot hosted_clang_hello. One way of matching remains: a filter that is a substring of a name runs it, so the dead-filter check is main's again and the `hosted_clang` case goes. The empty-filter case names the excluded test literally rather than reading ON_DEMAND, so emptying the list turns it red. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C
|
--- a/tests/toyos.rs
+++ b/tests/toyos.rs
@@ -6758,7 +6758,7 @@
/// Machine tests a run boots only when a filter names them, so no unfiltered
/// run, CI's included, boots one: each needs a product only
/// `cargo run -- --hosted-clang` builds, which no CI job holds.
-const ON_DEMAND: &[&str] = &["hosted_clang_hello"];
+const ON_DEMAND: &[&str] = &[];
/// Whether a run takes `name`: a filter matches it, or there is none and it is
/// not [`ON_DEMAND`]. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Review of #830 at Net lines ( Earlier BLOCKER
Earlier NOTEs
Evidence
BLOCKERNone. NOTENone. LAND |
This is part of M2 of
issues/toyos-builds-itself.md. ToyOS's own clang andld.lld(cargo run -- --hosted-clang) run inside a guest. They compilehello.cthere and link it, the program runs, and the guest'shello.oandhelloare byte for byte what the host's clang and LLD of the same LLVM commit make from the same input. The test runs on demand only, per the owner's ruling: no unfiltered run boots it, CI's included.The four things that stopped it are on
main. #836 relocates executables in userland, so the kernel reads no ELF table these binaries declare. The other three are #851 (statidentity), #854 (libc'srealpath) and #855 (kernel directory identity).What changed, per decision
hosted_clang_hello, a machine test (tests/common/hostedclang.rs), with a guest job of the same name (tests/toyos-rust-tests, onRUST_SKIPbecause only its own boot stages what it needs).tests/hostedclangcase: logkeeper and test-runner), at/system/share/hosted-clang. It holdsbin/clangandbin/ld.lld, clang's resource headers atlib/clang(clang looks for them beside itself), the C sysroot's C headers andlibtoyos_c.aatsysroot,src/hello.c, and the host's two products athost/. ROOT, not a data disk: the loader reads ROOT into memory whole, so the guest reads these read-only fixtures with no disk and no server behind it. A data disk would need a filesystem written on the host.--target=x86_64-unknown-toyos --sysroot=… -c hello.c -o …, fromhello.c's directory). It then runsld.llddirectly, on the line the ToyOS driver gives it (--sysroot=… -pie --eh-frame-hdr -o … -L…/lib hello.o -ltoyos_c, read off the host clang's-###). It cannot link through the driver: a linking driver starts LLD as a child, and libc starts none. The job runs the result, prints what it said, and compares each product with the host's byte for byte. Where they part, it names the first differing offset and 32 bytes of each side. The host prints each product's SHA-256, so a pass means the guest's products have those hashes.ON_DEMANDintests/toyos.rs.keptisfilters.iter().any(|f| name.contains(f)) || (filters.is_empty() && !ON_DEMAND.contains(&name)). An unfiltered run (--ci guest, the nightly) never boots a test listed there; a filter matches it the way it matches any other test. The test needs the hosted clang, whichcargo run -- --hosted-clangalone builds and no CI job holds. There is one way of matching, so the dead-filter check ismain's, unchanged.a_run_selects_by_filter(tests/checks.rs): the unfiltered run takes every declared test excepthosted_clang_hello, which the case names literally rather than readingON_DEMAND, so emptying the list turns it red;hosted_clang_hellotakes it.hostedclang::ensureis whatdispatchdid: sysroot, fork checkout, LLVM held, product resolved. The harness gets the product the way--hosted-clangdoes.resolveis private now.src/build.rs'sALL_CONFIGSnames the new case.every_shipped_boot_config_is_coveredis red without it.issues/toyos-builds-itself.md. Its M2 paragraph said no guest had run either binary. It now says whathosted_clang_helloshows, and what M2's exit still waits on: the driver's link, which starts LLD as a child, and an install bypkg. The "No linker runs inside ToyOS" paragraph no longer says a ToyOS process cannot link an object. It now says no image ships a linker. Its exit, the owner's, is left as written.issues/ci-runs-the-whole-guest-suite.mdrecords the stopgap: its exit deletesON_DEMAND, because once--ci guestruns a declared list, everything outside it is on demand.the-program-loader-refuses-clang-and-lld-whose-elf-tables-pass-one-kernel-allocation.md. It never reachedmainand nothing cites it. Both halves of its exit hold.hosted_clang_hellogets past both spawns. After Executables relocate themselves, and the kernel reads no executable's dynamic section #836 the kernel allocates nothing a file's dynamic section declares, because it reads none.Why a guest test, and why it costs CI nothing
The claim is that these two binaries run on ToyOS's kernel and libc and write the host's bytes there.
x86_64-unknown-toyosexecutables, and a host can only run the host's clang, which this test already uses as its oracle.So QEMU is the cheapest tier that reaches the claim. Because the test is on demand, its cost is a boot when somebody asks for one.
Gates
At
0e3de9158:cargo run -- --ci hostcargo test --test toyos-checks a_run_selects_by_filterMutation at
0e3de9158, applied as a checked patch, run, and reversed in the same script, which left the tree clean:ON_DEMANDemptied (on-demand-empty.patch):a_run_selects_by_filterexits 101 through the unfiltered case,filter []: took ["hosted_clang_hello"] it should not and left out [](checks-select-mutant.log).At
650613b01, not rerun: the diff since touches onlykept, the dead-filter line (back tomain's), a check case and an issue. The unfiltered selection and thehosted_clang_helloselection are what they were, whicha_run_selects_by_filterholds above; nothing the guest runs changed.cargo test --test toyos-build -- hosted_clang_hellold.lldended 0,hellosaid its line,hello.ois the host's 1,384 bytes (sha25634e76391…5462945b22) andhellothe host's 1,738,664 (sha2562ef17e0e…830c8f26); 14 scargo test --test toyos-build(the whole suite, unfiltered)hosted_clang_hellois not among themcargo run -- --build-onlyEarlier:
hosted_clang_helloat90a594e4bexited 0 with the same products and verdict in 582 s, most of it building the hosted clang and LLD (productab00e1668046a5e3), which nobody on this host had built for this sysroot; the run at650613b01found it in the store. Log in parts 1 2 3 4 5 6.What I am unsure of
-- hostedbootshosted_clang_hello, and on a host that has not built the hosted clang that run first spends about ten minutes building it (582 s measured above). Whether a filter should have to name it whole is the owner's to rule; this branch makes no rule of it.ld.lldruns directly, not through the driver, so its link line is copied fromclang -###. If the driver's line changes, the guest's and host's products still match each other but stop being the driver's.What the reviewer is owed
hosted_clang_hello; the reasons are in the "Why a guest test" section.hosted_clang_helloand it gains one case.git diff --shortstat origin/main...0e3de9158): 11 files, +289 −18.src/: +15 −8;tests/toyos.rs+16 −2, checks +3 −2, case 17, registration 1);🤖 Generated with Claude Code
https://claude.ai/code/session_017cSFvbD35xJ2kGANVdm23C