build: WASM/WASI Nix flake environment and build helper scripts - #151
Open
angerman wants to merge 98 commits into
Open
build: WASM/WASI Nix flake environment and build helper scripts#151angerman wants to merge 98 commits into
angerman wants to merge 98 commits into
Conversation
This change reverts part of !14544, which forces the bootstrap compiler to have ghc-internal. As such it breaks booting with ghc 9.8.4. A better solution would be to make this conditional on the ghc version in the cabal file!
…ernal If the boot compiler doesn't have ghc-internal use "<unavailble>" as the `cGhcInternalUnitId`. This allows booting with older compilers. The subsequent stage2 compilers will have the proper ghc-internal id from their stage1 compiler, that boots them.
Make the first simple optimization pass after desugaring a real CoreToDo pass. This allows CorePlugins to decide whether they want to be executed before or after this pass.
It's more user-friendly to directly print the right thing instead of requiring the user to retry with the additional `-dppr-debug` flag.
mermaid is a common diagram format that can be inlined in markdown files, and e.g. github will even render it. This change adds support for mermaid diagram output to ghc-pkg.
This adds support to ghc-pkg to infer a package-db from a target name.
Add a new optional unitDataDir field to GhcPkg.InstalledPackageInfo and populate it in ghc-pkg's convertPackageInfoToCacheFormat. This is needed by the WASM linker to locate per-package data files (specifically WasmGlobalRegs.S) via GHC.Unit.Database.unitDataDir at link time, without requiring a file-system search.
By mistake we tried to use deriveConstant without passing `--gcc-flag -fcommon` (which Hadrian does) and it failed. This patch adds deriveConstant support for constants stored in the .bss section so that deriveConstant works without passing `-fcommon` to the C compiler.
Apple's LLVM toolchain uses `arm64` as the canonical architecture name for AArch64 on Apple platforms, while GNU config.sub normalises to `aarch64`. This mismatch causes `--target=aarch64-apple-darwin` to be passed to clang, which conflicts with toolchain wrappers (e.g. nix cc-wrapper) that expect `arm64-apple-darwin`. The result is thousands of test failures on aarch64-darwin because the cc-wrapper warning pollutes compiler output and the target flag interaction breaks compilation. Fix by adding normaliseLlvmTarget that rewrites `aarch64-apple-*` to `arm64-apple-*` for the LLVM target triple, matching Apple conventions and the existing llvm-targets file which already uses arm64-apple-darwin.
Add AC_ARG_WITH([compiler]) to allow specifying the Haskell compiler via the --with-compiler flag, consistent with standard autoconf practices for tool configuration. This provides an alternative to setting the GHC environment variable, making the build system more flexible and consistent with other configure scripts that accept --with-* options for tools.
Add entries to prevent AI agent config files from being accidentally committed. These files contain project-specific instructions for various AI coding assistants and should remain local. Covers: Claude Code, GitHub Copilot, Cursor, Gemini CLI/Jules, OpenAI Codex, and JetBrains Junie. See: https://agents.md/ for the AGENTS.md standard
Fixes #26434 In detail, this does a number of things: * Makes GHC aware of 'extra-libraries-static' (this changes the package database format). * Adds a switch '-static-external' that will honour 'extra-libraries-static' to link external system dependencies statically. * Adds a new field to settings/targets: "ld supports verbatim namespace". This field is used by '-static-external' to conditionally use '-l:foo.a' syntax during linking, which is more robust than trying to find the absolute path to an archive on our own. * Adds a switch '-fully-static' that is meant as a high-level interface for e.g. cabal. This also honours 'extra-libraries-static'. This also attempts to clean up the confusion around library search directories. At the moment, we have 3 types of directories in the package database format: * library-dirs * library-dirs-static * dynamic-library-dirs However, we only have two types of linking: dynamic or static. Given the existing logic in 'mungeDynLibFields', this patch assumes that 'library-dirs' is really just nothing but a fallback and always prefers the more specific variants if they exist and are non-empty. Conceptually, we should be ok with even just one search dirs variant. Haskell libraries are named differently depending on whether they're static or dynamic, so GHC can conveniently pick the right one depending on the linking needs. That means we don't really need to play tricks with search paths to convince the compiler to do linking as we want it. For system C libraries, the convention has been anyway to place static and dynamic libs next to each other, so we need to deal with that issue anyway and it is outside of our control. But this is out of the scope of this patch. This patch is backwards compatible with cabal. Cabal should however be patched to use the new '-fully-static' switch.
"Executable" seems more appropriate.
This patch teaches GHC how to build the external interpreter program when it is missing. As long as we have the `ghci` library, doing this is trivial so most of this patch is refactoring for doing it sanely. (cherry picked from commit 55eab80)
The comment still referenced the old `linkBinary` name after the rename to `linkExecutable` in 55ff022.
GHC and ghc-iserv load Haskell shared libraries dynamically for Template Haskell and GHCi. These libraries reference RTS symbols (e.g., stg_INTLIKE_closure) that are linked into the executable. Without special linker flags, those symbols aren't visible to dlopen'd libraries. This commit adds platform-specific linker flags to export these symbols: - Linux/FreeBSD: -rdynamic (passes --export-dynamic to ld) - macOS: -flat_namespace (makes all symbols visible across namespaces) - Windows: Cannot use --export-all-symbols due to 65535 symbol limit See Note [ghc-iserv and dynamic symbol export] in ghc-iserv.cabal.in for detailed explanation of the approach and alternatives considered.
Replace legacy __sync_fetch_and_* builtins with their modern __atomic_fetch_*
equivalents. This simplifies the code significantly, particularly for the nand
operation which previously required extensive workarounds for compiler
compatibility issues.
Changes:
- Replace __sync_fetch_and_{add,sub,and,or,xor} with __atomic_fetch_*
- Replace __sync_fetch_and_nand with __atomic_fetch_nand
- Remove CAS-based fallback for nand operations
- Remove compiler-specific warning suppressions for -Wsync-nand
- Remove volatile qualifiers (not needed with __atomic builtins)
- Update comments to reflect modern atomics usage
All operations maintain __ATOMIC_SEQ_CST memory ordering for sequential
consistency, matching the original behavior.
Co-authored-by: Andrea Bedini <andrea@andreabedini.com>
This commit adds build system support for creating dynamic GHC builds, including Makefile targets, bindist generation, and utility configurations. Key changes: 1. Makefile enhancements - Add DYNAMIC=1 build variable support - Create dylib symlinks for macOS dynamic builds - Use concrete file target for testsuite-timeout - Include ghc-iserv-dyn in tarballs for all targets - Proper bindist generation for dynamic builds 2. Utility cabal files (hp2ps.cabal, unlit.cabal) - Configure for dynamic linking support - Ensure utilities work with dynamic GHC 3. ghc-iserv infrastructure (iservmain.c) - Updates for dynamic interpreter server - Proper initialization for dynamic linking context 4. Test expectations for Stable Haskell - Update bug report URL in test expectations Usage: make DYNAMIC=1 _build/bindist # Build dynamic GHC bindist
This commit adds infrastructure for RTS sublibrary loading in dynamic builds, enabling the split RTS architecture to work with shared library linking. Key changes: 1. RTS sublibrary infrastructure (rts/rts.cabal) - Define separate sublibraries for RTS components - Add proper library dependencies and visibility - Configure shared library generation for RTS parts 2. Configure support for dynamic builds (rts/configure.ac) - Detect platform-specific dynamic linking requirements - Set appropriate linker flags for each sublibrary - Handle symbol visibility for exported functions 3. API updates for sublibrary boundaries (rts/include/RtsAPI.h) - Adjust exported symbol declarations - Ensure proper visibility across sublibrary boundaries 4. AutoApply support for interpreter (rts/AutoApply*.cmm) - Add AutoApply.cmm and vector variants (V16, V32, V64) - Required for dynamic bytecode interpreter operation 5. Cabal project configuration - cabal.project.stage1: Add no-ghc-internal flag for stage1 builds - cabal.project.stage2: Configure full RTS with all sublibraries 6. Thread infrastructure (rts/Threads.h) - Updates for sublibrary thread handling
This commit updates the testsuite to handle the split RTS architecture and dynamic GHC build configuration. Key changes: 1. testlib.py improvements - More robust test driver for dynamic builds - Better handling of shared library paths - Improved error detection and reporting 2. Test infrastructure (boilerplate.mk) - Configure tests for dynamic linking environment - Set proper library paths for test execution 3. Test adjustments for RTS split - T18072debug: Update grep to match cabal-based RTS naming - T23142.hs: Revert module name to fix -Di debug output test - keep-cafs-fail.stdout: Update expected output 4. Dynamic linking test updates - ghci/linking/dyn/all.T: Adjust for dynamic GHC - T2228: Restore expect_broken(7298) for dynamic builds - T11531.stderr: Update expected error messages 5. Platform-specific adjustments - T10458: Skip on musl with dynamic GHC - T11223 tests: Update stderr expectations for Windows 6. Test configuration - .gitignore: Add patterns for dynamic test artifacts - dynlibs/Makefile: Update for dynamic build testing - perf/size/all.T: Adjust size expectations
Add "Stable Haskell Edition" branding to user-visible output while maintaining drop-in compatibility with upstream GHC: - ghc --version: Append "(Stable Haskell Edition)" suffix - ghc -v2 banner: Add edition to verbose compiler banner - GHCi welcome: Add edition and update URL to GitHub repo - ghc --info: Add new "Edition" field (keeps "Project name" unchanged) - Bug reports: Redirect all URLs to github.com/stable-haskell/ghc/issues All internal identifiers (cProjectVersion, unit IDs, etc.) remain unchanged to preserve ABI and tool compatibility.
The branding commit changed the bug report URL from haskell.org/ghc/reportabug to github.com/stable-haskell/ghc/issues. Update test expectation files to match the new URL output. Fixes CI failures in T11223_link_order_a_b_2_fail and T11223_simple_duplicate_lib tests across all platforms.
This commit extends the CI/CD pipeline to build and test dynamic GHC configurations alongside the existing static builds. Key changes: 1. ci.yml - Main CI workflow - Add DYNAMIC=1 to build matrix - Configure dynamic build jobs for Linux and macOS - Run ghci-ext tests on dynamic builds (require interpreter) - Parallel execution of static and dynamic builds 2. reusable-release.yml - Release workflow - Add dynamic GHC builds to release artifacts - Generate separate bindists for dynamic configuration - Include ghc-iserv-dyn in release tarballs - Re-enable release workflow on pull requests for testing The dynamic build matrix allows testing of: - Template Haskell with dynamic code loading - GHCi interactive features - Dynamic library loading and linking - Interpreter-based test suites (ghci-ext) Build configurations: - Static (default): DYNAMIC=0 or unset - Dynamic: DYNAMIC=1
Add GitHub Actions release workflow with reusable build/test jobs, artifact upload/download, and proper _build/dist output paths.
Add QUIET mode for reduced output, per-phase timing instrumentation, metrics collection scripts, and matplotlib-based build phase plots. Includes macOS fixes and review feedback.
Since we now ignore loading ANY rts dependency.
- Add AR/RANLIB variables for wasm32-wasi-ar and wasm32-wasi-ranlib (explicit tool paths for cross-compilation) - Add --disable-libffi-adjustors to GHC_TOOLCHAIN_ARGS (wasm32 has no native adjustors, so ghc-toolchain defaults to libffi, but WASI cannot support ffi_closure_alloc which requires W^X memory; note: +use-system-libffi for general FFI is separate and correct) - Add -fno-exceptions to CXX_OPTS (wasi-sdk libc++ has no exception support)
* hsc2hs: batch cross-compilation for massive speedup Replace hackage hsc2hs-0.68.10 with stable-haskell/hsc2hs fork that includes batch cross-compilation support. This reduces C compiler invocations from hundreds/thousands per .hsc file to just 1-2 total by batching all constant-like directives (#const, #size, #alignment, step with graceful fallback to per-directive compilation. Key changes in the fork (bf966e8): - Batch collection of all batchable directives with conditional stack tracking - Single C file generation with all constants as global variables - Assembly parsing via ATTParser to extract all values at once - Graceful try/catch around ATT.parse for compilers producing non-AT&T assembly (e.g. emcc for WebAssembly), falling through to per-directive path - --no-batch flag for debugging/fallback Performance impact per .hsc file: - Before (non-via-asm): ~28 compilations per directive (e.g. 4089 for Flags.hsc) - Before (via-asm): 1 compilation per directive (e.g. 147 for Flags.hsc) - After (batch): 2 compilations total per file (validity check + batch) Fork: https://github.com/stable-haskell/hsc2hs/tree/feat/batch-cross-compilation * fix: update hsc2hs fork with Windows CodeView debug section fix Update hsc2hs fork reference to include fix for ATT parser crash on Windows where Clang emits CodeView .debug$S sections even with -g0. The parser now filters out debug sections and handles unrecognized instruction patterns gracefully.
.nix-wasm-bin/ — symlink directory created by flake.nix pointing
into the Nix store for wasi-sdk tools; local only
.nixos-remote-build.conf — per-machine config for the remote Linux build
helper script; not part of the source tree
angerman
force-pushed
the
stable-ghc-9.14
branch
from
March 6, 2026 01:06
05e1a0c to
7968d7c
Compare
Switch from manual shell.nix management to using the ghc-wasm-meta flake which provides a properly integrated wasi-sdk with libffi-wasm. Benefits: - wasi-sdk from ghc-wasm-meta includes libffi-wasm pre-integrated - Maintained by the GHC WASM team - Reproducible builds via flake.lock - No manual download/patching required The wasi-sdk package from ghc-wasm-meta is used directly, eliminating the need for manual libffi-wasm installation scripts. See: https://gitlab.haskell.org/haskell-wasm/ghc-wasm-meta
…ride When wasi-sdk is in buildInputs, nix's mkShell automatically sets CC and LD to the WASM cross-compiler tools. This breaks any native compilation during the build (e.g. stage1 deriveConstants, configure scripts). The wasi-sdk is now accessed exclusively via WASI_SDK_DIR in shellHook, and the cross-compiler tools are only available via named wrappers (wasm32-wasi-clang, wasm-ld, etc.) in PATH.
GHC's WASM LLVM backend uses llc/opt/llvm-as to compile LLVM IR to WASM assembly. Using nixpkgs LLVM 18 tools with wasi-sdk's clang 21 assembler causes version mismatch errors: 'error: .size directive ignored for function symbols' Fix by creating wrappers that use wasi-sdk's LLVM 21 versions of llc, opt, and llvm-as, which are compatible with wasi-sdk's clang 21.
- Export WASM_EXTRA_LIB_DIRS and WASM_EXTRA_INCLUDE_DIRS pointing to the wasi-sysroot, so the Makefile can pass --extra-lib-dirs and --extra-include-dirs to cabal for WASM RTS compilation. - Fix help text: the Makefile default CABAL=_build/stage0/bin/cabal is correct; no CABAL= override is needed. Just run: make stage2 / stage3-... - Keep ghc-wasm-meta unpinned (latest): with TNTC disabled (see previous commit), info table symbols are pure DATA, not WASM FUNCTION types. The new LLVM's .size restriction on function symbols does not apply. Using the latest ghc-wasm-meta is safe and provides the full wasi-sdk with wasi-sysroot already cached from previous builds.
build-wasm-make.sh: - Switch from nix-shell to nix develop (flake-based environment) - Remove explicit CABAL= override; Makefile default _build/stage0/bin/cabal is correct and stage0 cabal is auto-built from the repo's pinned source build-wasm-on-linux0.sh: - Fix rsync to exclude both '.git' file (git worktree reference) and '.git/' directory; prevents broken worktree path from being synced - After rsync, re-initialize a proper git repo on the remote with git init + git add -A + git commit; required for nix develop to evaluate flake.nix via git+file:// URL - Update prerequisites: only Nix with flake support needed (no ghcup)
- Document switch from nixpkgs pkgsCross.wasi32 to wasi-sdk 24.0 - Explain nixpkgs LLD bug that prevented linking - Document libffi incompatibility with WASI (Emscripten-only) - Clarify wrapper scripts approach for LD_LIBRARY_PATH propagation - Add reference to libffi source showing Emscripten requirement
angerman
force-pushed
the
build/wasm-nix-environment
branch
from
March 6, 2026 01:12
1d99ecb to
2e88edb
Compare
angerman
force-pushed
the
stable-ghc-9.14
branch
2 times, most recently
from
August 30, 2026 01:45
8bdf55f to
93c3dc1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Nix flake development environment and remote build helper scripts
needed to build the WASM/WASI cross-compiler (
wasm32-unknown-wasi-ghc)on a Linux machine from a macOS workstation.
This is the build infrastructure companion to PR #142 (
fix/wasm-remove-pic-guards),kept separate so the compiler/RTS fixes can be reviewed independently.
What's included
flake.nix/flake.lock: Nix flake that pulls inghc-wasm-metaforwasi-sdk(clang, wasm-ld, wasi-sysroot) and exportsWASM_EXTRA_LIB_DIRS/WASM_EXTRA_INCLUDE_DIRSfor the build.Uses wasi-sdk's LLVM 21 tools (
llc,opt,llvm-as) so versions match.Does not add
wasi-sdktobuildInputs(would overrideCC/LDand break the native stage2 build).
build-wasm-make.sh: wrapper that entersnix developand runsmake stage3-wasm32-unknown-wasiwith the correct environment.build-wasm-on-linux0.sh: syncs the local source tree to a remoteLinux machine via rsync (excluding
_build/,.git, generated autoconffiles) and triggers a full build there. Uses
--no-deleteto preserveautoconf-generated files on the remote that are absent from a git worktree.
USAGE.md: step-by-step build instructions covering prerequisites,build invocation, and how to test the resulting
wasm32-unknown-wasi-ghc.Test plan
nix develop .opens a shell withwasm32-unknown-wasi-clangon PATHnix develop . -c make stage3-wasm32-unknown-wasicompletes on x86_64-linuxwasm32-unknown-wasi-ghc /tmp/hello.hs -o /tmp/hello.wasm && wasmtime /tmp/hello.wasmprintsHello WASM