ci: re-enable ubuntu-22-rocm release build with larger ccache - #91
Open
jimw567 wants to merge 5 commits into
Open
ci: re-enable ubuntu-22-rocm release build with larger ccache#91jimw567 wants to merge 5 commits into
jimw567 wants to merge 5 commits into
Conversation
added 5 commits
August 18, 2026 15:50
Revert ggml-org#26969 (which disabled the ROCm Ubuntu release job) and address the underlying cause rather than just turning the job back on. Root cause of the 2h+ builds: the ROCm 7.14 switch (ggml-org#25775) doubled the Linux GPU target count from 11 to 22. Each HIP translation unit is compiled once per target arch, so both build time and ccache size roughly doubled. The ggml-org/ccache-action defaults to max-size 500MB, which was enough for the 11 targets of ROCm 7.2.1 but too small for 22: CI logs from the 7.14 era show the cache pinned at the 500MB cap (78% full, 10 evictions mid-build, only ~45% hit rate), so it never went warm and every run took ~90-135 min. Fix: re-enable the ubuntu-22-rocm job and its ccache steps, and raise the ccache max-size to 2G so the 22-target cache fits and builds stay warm across runs. Also restores the job in the release job's needs: list and the release-notes download link.
Standalone workflow_dispatch job that runs only the ubuntu-22-rocm build (22 targets, ROCm 7.14, ccache max-size 2G) so build time and cold->warm ccache behavior can be measured on stock GitHub runners without triggering the full release matrix or publishing a release. Prints ccache -s before/after and BUILD_SECONDS to the job summary. Remove before merging the parent change.
Add a push trigger scoped to this branch so the timing job runs without needing workflow_dispatch registration on the default branch.
The ROCm toolchain is pip-installed fresh on every run, so the clang binary's mtime changes each time. With ccache's default compiler_check=mtime that invalidates the whole cache and warm builds only reached ~70% hits. Hash the compiler contents instead so the cache survives toolchain reinstalls. Also applied to the temp timing workflow for measurement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Re-enables the ROCm Ubuntu release build that upstream disabled in ggml-org#26969, and fixes the underlying cause instead of just turning the job back on.
Root cause
The ROCm 7.14 switch (ggml-org#25775) doubled the Linux GPU target count from 11 to 22. Each HIP translation unit is compiled once per target arch, so both build time and ccache footprint roughly doubled.
ggml-org/ccache-actiondefaults tomax-size: 500MB. That was sufficient for the 11 targets of ROCm 7.2.1 (builds were ~11 min warm), but too small for 22 targets. CI logs from the 7.14 era (before the cache was disabled) show the cache pinned at the 500MB cap:Cleanups) during a single buildSo the cache self-evicted every run and never converged. Upstream disabled the cache (ggml-org#26962), then the whole job (ggml-org#26969), citing "ccache does not work and build takes 2hrs+".
Fix
ubuntu-22-rocmjob (byte-identical to upstream's pre-ci : disable ubuntu-rocm ggml-org/llama.cpp#26969 job) and its ccache save/restore steps.max-sizeto2Gso the 22-target cache fits and builds stay warm across runs.releasejob'sneeds:list and the release-notes download link.Evaluation
Local reproduction (22 targets, ROCm 7.14, uncapped ccache):
This confirms a 2G cap is more than enough. The Release workflow triggers on push-to-master / manual dispatch (not on PRs), so this needs a manual
workflow_dispatchrun on the branch to validate build time on stock GitHub runners.Test plan