[9.3.0] Clone files into the disk cache where the filesystem supports it (htt… - #30901
Open
iancha1992 wants to merge 2 commits into
Open
[9.3.0] Clone files into the disk cache where the filesystem supports it (htt…#30901iancha1992 wants to merge 2 commits into
iancha1992 wants to merge 2 commits into
Conversation
…elbuild#30776) - `DiskCacheClient.uploadFile` now copies through `FileSystemUtils#copyFile`, which reaches `Files#copy` and lets the filesystem serve the copy as a copy-on-write clone: `clonefile` on macOS, `copy_file_range` on Linux with a supporting filesystem. The cache entry then shares its blocks with the output it came from. The previous code streamed the bytes into the temporary file. - Where no clone is possible (different filesystem, or no kernel support), `Files#copy` falls back to a byte copy, so this is never slower than the stream it replaces. - The temporary-file-then-rename scaffolding moved into a private `save()` helper, shared by the stream path and the new file path. - A clone inherits the source's permissions and mtime, so both are reset: outputs are typically `0555` and can be older than the upload, an entry has to stay readable for every user of a shared cache, and the garbage collector reads the mtime to find the least recently used entries. - The cloned temporary file is fsynced before the rename, keeping the durability the streamed write had. Materializing outputs from the disk cache already clones, since ec90e05 ("Optimize file copies by using NIO methods"). The upload direction still wrote a second physical copy of a file that was already on disk. Measured on macOS/APFS: one genrule producing a 1 GiB output, fresh disk cache, real disk consumed via `df`. | | before | after | |---|---|---| | 1 GiB output plus its cache entry | 2058 MiB | 1028 MiB | Physical extent mapping (`fcntl(F_LOG2PHYS_EXT)`) on the same pair agrees: 0/3 sampled extents shared before, 3/3 after. On a large iOS app build, the disk cache held 13.1 GiB duplicating content that was also present in the output tree. Two consequences worth knowing: - An entry sharing blocks with an output is no longer freed by deleting either one alone. - `DiskCacheGarbageCollector` sizes entries by their logical length, so it over-counts what deletion reclaims. No - [x] I have added tests for the new use cases (if any). - [ ] I have updated the documentation (if applicable). RELNOTES: Files uploaded to the disk cache are now copied with a copy-on-write clone where the filesystem supports it, so a cache entry can share its blocks with the output it was uploaded from. Closes bazelbuild#30776. PiperOrigin-RevId: 968392369 Change-Id: I89bfa402106cd59964045353854336fa36098d8b
iancha1992
enabled auto-merge
August 28, 2026 20:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ps://github.com//pull/30776)
DiskCacheClient.uploadFilenow copies throughFileSystemUtils#copyFile, which reachesFiles#copyand lets the filesystem serve the copy as a copy-on-write clone:clonefileon macOS,copy_file_rangeon Linux with a supporting filesystem. The cache entry then shares its blocks with the output it came from. The previous code streamed the bytes into the temporary file.Files#copyfalls back to a byte copy, so this is never slower than the stream it replaces.save()helper, shared by the stream path and the new file path.0555and can be older than the upload, an entry has to stay readable for every user of a shared cache, and the garbage collector reads the mtime to find the least recently used entries.Materializing outputs from the disk cache already clones, since ec90e05 ("Optimize file copies by using NIO methods"). The upload direction still wrote a second physical copy of a file that was already on disk.
Measured on macOS/APFS: one genrule producing a 1 GiB output, fresh disk cache, real disk consumed via
df.Physical extent mapping (
fcntl(F_LOG2PHYS_EXT)) on the same pair agrees: 0/3 sampled extents shared before, 3/3 after. On a large iOS app build, the disk cache held 13.1 GiB duplicating content that was also present in the output tree.Two consequences worth knowing:
DiskCacheGarbageCollectorsizes entries by their logical length, so it over-counts what deletion reclaims.No
RELNOTES: Files uploaded to the disk cache are now copied with a copy-on-write clone where the filesystem supports it, so a cache entry can share its blocks with the output it was uploaded from.
Closes #30776.
PiperOrigin-RevId: 968392369
Change-Id: I89bfa402106cd59964045353854336fa36098d8b
Description
Motivation
Build API Changes
No
Checklist
Release Notes
RELNOTES: None
Commit a341ee5