Skip to content

perf!: avoid unnecessary metadata during deletion - #397

Merged
Byron merged 1 commit into
mainfrom
speed-up-deletions
Sep 7, 2026
Merged

perf!: avoid unnecessary metadata during deletion#397
Byron merged 1 commit into
mainfrom
speed-up-deletions

Conversation

@Byron

@Byron Byron commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Tasks

This section is for Byron only. Models continuing this PR must not add, remove, check, uncheck, rename, or reorder checkboxes here.

  • refackiew

Everything below this line was generated by Codex.

Created by Codex on behalf of Byron. Byron will review before this is ready to merge.

Deletion previously collected sizes, timestamps, allocation information, and file identities for every entry. It now requests only entry types and stops storing per-entry byte counts.

dua-core::Options::skip_metadata selects standard directory enumeration on macOS and Windows and skips per-entry metadata calls and metadata jobs on other platforms. Explicit Windows roots query only attributes and reparse tags. Symlinks remain untraversed, and Windows long/verbatim paths retain their existing handling. Directory symlinks and junctions use a non-recursive remove_dir fallback on Windows; the expanded CI coverage exposed the previous use of the wrong removal API.

Breaking change: Entry::metadata is now Option<io::Result<Metadata>>, and Options literals need skip_metadata. Default options still collect full metadata. Successful deletion notifications use the existing scanned total; partially failed deletions report ? bytes with their entry/error counts.

Performance evaluation

Release-mode comparison against 115f7bb5 on macOS 26.6.2, arm64, local APFS. Times are medians of seven measured runs after one warmup, with baseline/candidate order alternating. Fixture creation was outside the timer. The harness used each revision's deletion routine and dua-core walker.

Fixture Threads Walk before → after (ms) Delete before → after (ms)
10,000 files in one directory 1 11.04 → 8.81 244.36 → 216.32
10,000 files in one directory 8 5.92 → 3.53 219.66 → 211.74
8,192 files across 272 directories 1 13.59 → 10.13 189.04 → 182.52
8,192 files across 272 directories 8 3.79 → 4.16 168.29 → 165.98
2,000 empty child directories 1 19.32 → 18.62 64.29 → 61.73
2,000 empty child directories 8 14.53 → 14.16 61.96 → 62.04

These fixtures show up to 11.5% less deletion time and up to 40% less traversal time, with several cases effectively unchanged. File removal dominates total time. Linux and Windows performance was not measured locally.

Validation

  • Regression coverage for omitted metadata, both traversal orders, multiple roots, restart, symlinks, and readable macOS directories without metadata access.
  • Windows path and symlink checks exercise both metadata modes. Deletion tests cover explicit symlink roots and symlinks inside deleted directories, retaining nonempty targets outside the deleted tree.
  • make check-pre-push passed.
  • cargo clippy --workspace --all-targets --all-features -- -D warnings passed.
  • cargo check -p dua-core --all-targets passed for aarch64-pc-windows-msvc and aarch64-unknown-linux-gnu.
  • Codex review ran once for each commit (3f9652fa, d13908b4) and reported no actionable findings.
  • The Windows CI fix also passed targeted deletion tests and cargo check --workspace --all-targets --target aarch64-pc-windows-msvc locally.

Reported issue

Evaluate if deletions can be faster if they only obtain the information that's actually required, like the type of the entry. While on some platforms like linux it might make no difference, on others they may be more choice, like Windows and macOS.

It's fine to make it an officially breaking change in the dua-core library.

@Byron
Byron force-pushed the speed-up-deletions branch from d13908b to fdfd9b2 Compare September 7, 2026 08:13
<!-- agent -->
Deletion needs entry types, but every walk collected full sizes, timestamps,
allocation information, and file identities. Add `Options::skip_metadata` and
use directory-entry types without extra metadata lookups when available.
macOS and Windows use standard enumeration for this mode; Windows roots
query only attribute/tag information. Keep the native metadata readers for
ordinary scans and preserve Windows verbatim-path handling.

Drop per-entry deletion byte accounting. Successful removals use the existing
scanned total; partial failures retain entry/error counts and display unknown
bytes instead of collecting metadata solely for that notification.

Local macOS/APFS release medians showed 0-11.5% lower deletion time, with
wide-directory traversal up to 40% shorter; gains vary with tree shape.

BREAKING CHANGE: dua-core `Entry::metadata` is now
Option<io::Result<Metadata>>, and Options literals require `skip_metadata`.
Default options continue to collect full metadata.

Assisted-by: GPT 6.0
Co-authored-by: GPT 6.0 <codex@openai.com>
@Byron
Byron force-pushed the speed-up-deletions branch from fdfd9b2 to f643c74 Compare September 7, 2026 08:48
@Byron
Byron marked this pull request as ready for review September 7, 2026 08:49
@Byron
Byron merged commit 97211e6 into main Sep 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant