Skip to content

Sch: Restore per-processor pressure in task cost estimation - #744

Merged
jpsamaroo merged 1 commit into
masterfrom
jps/fix-663
Aug 27, 2026
Merged

Sch: Restore per-processor pressure in task cost estimation#744
jpsamaroo merged 1 commit into
masterfrom
jps/fix-663

Conversation

@jpsamaroo

Copy link
Copy Markdown
Member

estimate_task_costs! dropped the est_business term (each candidate processor's currently-reserved compute pressure) in b16afad, leaving only a fixed per-task cross-worker transfer cost in the comparison. That fixed cost always makes the scheduler's own worker look cheapest, so under the default scope every task piled onto worker 1 regardless of how busy it became (#663).

Restore the pressure term, reading it from worker_time_pressure's current LockedObject/atomic-counter representation, so busier processors are considered costlier and work spreads across workers again.

Fixes #663

Written by Claude Sonnet

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Dagger benchmarks: dirty vs master

Median time

master dirty master / dirty
array/dagger/N=1024 (block 512)/add (X + X) 5 ± 0.27 ms 4.56 ± 1.2 ms 1.1 ± 0.3
array/dagger/N=1024 (block 512)/alloc (rand) 1.67 ± 0.018 ms 2.3 ± 0.55 ms 0.728 ± 0.18
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 2.29 ± 0.61 ms 1.9 ± 0.65 ms 1.2 ± 0.52
array/dagger/N=1024 (block 512)/map (sin.(X)) 5.61 ± 0.51 ms 5.12 ± 1.5 ms 1.1 ± 0.33
array/dagger/N=1024 (block 512)/norm 1.19 ± 0.1 ms 1.15 ± 0.024 ms 1.03 ± 0.091
array/dagger/N=1024 (block 512)/reduce (sum) 2.46 ± 1.4 ms 2.38 ± 0.97 ms 1.03 ± 0.72
array/dagger/N=1024 (block 512)/transpose (permutedims) 4.69 ± 0.62 ms 4.62 ± 0.59 ms 1.02 ± 0.19
array/dagger/N=256 (block 256)/add (X + X) 0.813 ± 0.032 ms 0.773 ± 2 ms 1.05 ± 2.7
array/dagger/N=256 (block 256)/alloc (rand) 0.717 ± 0.099 ms 0.657 ± 0.045 ms 1.09 ± 0.17
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.455 ± 0.079 ms 0.442 ± 0.019 ms 1.03 ± 0.18
array/dagger/N=256 (block 256)/map (sin.(X)) 1 ± 0.14 ms 0.946 ± 0.0094 ms 1.06 ± 0.15
array/dagger/N=256 (block 256)/norm 0.477 ± 0.0061 ms 0.419 ± 0.11 ms 1.14 ± 0.29
array/dagger/N=256 (block 256)/reduce (sum) 0.846 ± 0.13 ms 0.847 ± 0.036 ms 0.998 ± 0.16
array/dagger/N=256 (block 256)/transpose (permutedims) 0.792 ± 0.012 ms 0.698 ± 0.054 ms 1.14 ± 0.09
linalg/dagger/N=1024 (block 512)/cholesky 22.5 ± 4.5 ms 19.9 ± 3.2 ms 1.13 ± 0.29
linalg/dagger/N=1024 (block 512)/lu 0.0427 ± 0.00062 s 0.0407 ± 0.0045 s 1.05 ± 0.12
linalg/dagger/N=1024 (block 512)/matmul (A*A) 0.0447 ± 0.026 s 0.052 ± 0.0056 s 0.861 ± 0.51
linalg/dagger/N=1024 (block 512)/matvec (A*x) 3.56 ± 2.4 ms 3.12 ± 1.5 ms 1.14 ± 0.95
linalg/dagger/N=1024 (block 512)/qr 0.112 ± 0.0087 s 0.112 ± 0.0055 s 1 ± 0.093
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.0509 ± 0.0028 s 0.0504 ± 0.0021 s 1.01 ± 0.071
linalg/dagger/N=1024 (block 512)/svd 0.0374 h 0.037 h 1.01
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 0.0325 ± 0.00057 s 0.0332 ± 0.0046 s 0.979 ± 0.14
linalg/dagger/N=256 (block 256)/cholesky 6.16 ± 0.81 ms 4.59 ± 2.5 ms 1.34 ± 0.76
linalg/dagger/N=256 (block 256)/lu 6.6 ± 3.9 ms 4.53 ± 0.5 ms 1.46 ± 0.89
linalg/dagger/N=256 (block 256)/matmul (A*A) 2.33 ± 0.83 ms 2.12 ± 0.74 ms 1.1 ± 0.55
linalg/dagger/N=256 (block 256)/matvec (A*x) 1.98 ± 2.4 ms 1.07 ± 2.9 ms 1.85 ± 5.5
linalg/dagger/N=256 (block 256)/qr 4.97 ± 0.69 ms 5.25 ± 0.97 ms 0.947 ± 0.22
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 10.2 ± 1.2 ms 10.1 ± 10 ms 1.01 ± 1
linalg/dagger/N=256 (block 256)/svd 0.608 ± 0.0021 s 0.571 ± 0.04 s 1.07 ± 0.075
linalg/dagger/N=256 (block 256)/syrk (A'*A) 3.53 ± 0.91 ms 3.76 ± 1.4 ms 0.939 ± 0.43
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 7.69 ± 1.3 ms 6.49 ± 0.68 ms 1.19 ± 0.24
stencil/dagger/N=1024 (block 512)/assign (const) 1.84 ± 1.6 ms 0.893 ± 0.029 ms 2.06 ± 1.8
stencil/dagger/N=1024 (block 512)/multi-expr 4.76 ± 2.5 ms 2.61 ± 2.3 ms 1.83 ± 1.9
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 6.58 ± 0.65 ms 5.1 ± 1.3 ms 1.29 ± 0.35
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 5.85 ± 0.37 ms 5.54 ± 0.14 ms 1.06 ± 0.071
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 6 ± 0.71 ms 5.69 ± 0.67 ms 1.05 ± 0.18
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 6.47 ± 0.57 ms 5.58 ± 0.3 ms 1.16 ± 0.12
stencil/dagger/N=1024 (block 512)/update (+) 1.42 ± 1.7 ms 1.9 ± 1.4 ms 0.751 ± 1.1
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 1.82 ± 0.075 ms 1.63 ± 0.047 ms 1.11 ± 0.056
stencil/dagger/N=256 (block 256)/assign (const) 0.481 ± 0.04 ms 0.551 ± 0.23 ms 0.872 ± 0.38
stencil/dagger/N=256 (block 256)/multi-expr 1.96 ± 1.3 ms 1.43 ± 1.2 ms 1.37 ± 1.5
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.46 ± 0.031 ms 1.45 ± 0.54 ms 1.01 ± 0.37
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.34 ± 0.1 ms 2.45 ± 1.5 ms 0.546 ± 0.34
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.36 ± 0.14 ms 1.17 ± 0.12 ms 1.16 ± 0.17
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.48 ± 0.3 ms 1.28 ± 0.23 ms 1.16 ± 0.31
stencil/dagger/N=256 (block 256)/update (+) 1.1 ± 0.76 ms 0.71 ± 0.23 ms 1.55 ± 1.2
time_to_load 1.06 ± 0.0073 s 1.05 ± 0.0058 s 1 ± 0.0089

Allocations / memory

master dirty master / dirty
array/dagger/N=1024 (block 512)/add (X + X) 5.47 k allocs: 8.21 MB 5.52 k allocs: 8.22 MB 1
array/dagger/N=1024 (block 512)/alloc (rand) 2.74 k allocs: 8.09 MB 2.69 k allocs: 8.09 MB 1
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 2.39 k allocs: 8.08 MB 2.4 k allocs: 8.08 MB 1
array/dagger/N=1024 (block 512)/map (sin.(X)) 2.07 k allocs: 8.07 MB 2.08 k allocs: 8.07 MB 1
array/dagger/N=1024 (block 512)/norm 2.71 k allocs: 0.0906 MB 2.68 k allocs: 0.0891 MB 1.02
array/dagger/N=1024 (block 512)/reduce (sum) 4.11 k allocs: 0.141 MB 3.9 k allocs: 0.133 MB 1.06
array/dagger/N=1024 (block 512)/transpose (permutedims) 3.32 k allocs: 8.14 MB 3.28 k allocs: 8.13 MB 1
array/dagger/N=256 (block 256)/add (X + X) 1.71 k allocs: 0.574 MB 1.7 k allocs: 0.574 MB 1
array/dagger/N=256 (block 256)/alloc (rand) 0.749 k allocs: 0.526 MB 0.749 k allocs: 0.526 MB 1
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.646 k allocs: 0.522 MB 0.635 k allocs: 0.522 MB 1
array/dagger/N=256 (block 256)/map (sin.(X)) 0.569 k allocs: 0.519 MB 0.569 k allocs: 0.519 MB 1
array/dagger/N=256 (block 256)/norm 0.71 k allocs: 24.5 kB 0.702 k allocs: 24.2 kB 1.01
array/dagger/N=256 (block 256)/reduce (sum) 1.29 k allocs: 0.0464 MB 1.3 k allocs: 0.0466 MB 0.995
array/dagger/N=256 (block 256)/transpose (permutedims) 1.1 k allocs: 0.551 MB 1.1 k allocs: 0.551 MB 1
linalg/dagger/N=1024 (block 512)/cholesky 7.81 k allocs: 10.3 MB 7.77 k allocs: 10.3 MB 1
linalg/dagger/N=1024 (block 512)/lu 15.6 k allocs: 14.6 MB 15.5 k allocs: 14.6 MB 1
linalg/dagger/N=1024 (block 512)/matmul (A*A) 7.57 k allocs: 8.28 MB 7.63 k allocs: 8.28 MB 1
linalg/dagger/N=1024 (block 512)/matvec (A*x) 6.14 k allocs: 0.246 MB 5.95 k allocs: 0.238 MB 1.03
linalg/dagger/N=1024 (block 512)/qr 11.4 k allocs: 9.59 MB 11 k allocs: 9.57 MB 1
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 29.3 k allocs: 15.2 MB 29.6 k allocs: 15.2 MB 1
linalg/dagger/N=1024 (block 512)/svd 0.0488 M allocs: 0.197 GB 0.0485 M allocs: 0.197 GB 1
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 8.87 k allocs: 24.4 MB 8.81 k allocs: 24.4 MB 1
linalg/dagger/N=256 (block 256)/cholesky 3.03 k allocs: 0.633 MB 3.02 k allocs: 0.633 MB 1
linalg/dagger/N=256 (block 256)/lu 5.42 k allocs: 1.24 MB 5.35 k allocs: 1.24 MB 1
linalg/dagger/N=256 (block 256)/matmul (A*A) 1.92 k allocs: 0.581 MB 1.91 k allocs: 0.581 MB 1
linalg/dagger/N=256 (block 256)/matvec (A*x) 2.34 k allocs: 0.099 MB 2.33 k allocs: 0.0986 MB 1
linalg/dagger/N=256 (block 256)/qr 3.44 k allocs: 0.779 MB 3.43 k allocs: 0.778 MB 1
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 12 k allocs: 1.53 MB 11.8 k allocs: 1.52 MB 1
linalg/dagger/N=256 (block 256)/svd 15.4 k allocs: 6.71 MB 15.4 k allocs: 6.71 MB 1
linalg/dagger/N=256 (block 256)/syrk (A'*A) 2.97 k allocs: 2.63 MB 2.97 k allocs: 2.63 MB 1
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 4.7 k allocs: 8.2 MB 4.72 k allocs: 8.2 MB 1
stencil/dagger/N=1024 (block 512)/assign (const) 2.1 k allocs: 0.0926 MB 2.15 k allocs: 0.0943 MB 0.982
stencil/dagger/N=1024 (block 512)/multi-expr 4.78 k allocs: 0.211 MB 4.76 k allocs: 0.21 MB 1
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 3.73 k allocs: 0.231 MB 3.75 k allocs: 0.231 MB 0.997
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 3.6 k allocs: 0.227 MB 3.6 k allocs: 0.227 MB 1
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 3.71 k allocs: 0.262 MB 3.71 k allocs: 0.262 MB 1
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 3.56 k allocs: 0.166 MB 3.55 k allocs: 0.166 MB 1
stencil/dagger/N=1024 (block 512)/update (+) 2.59 k allocs: 0.115 MB 2.73 k allocs: 2.12 MB 0.0544
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 1.54 k allocs: 0.574 MB 1.53 k allocs: 0.574 MB 1
stencil/dagger/N=256 (block 256)/assign (const) 0.795 k allocs: 0.0403 MB 0.806 k allocs: 0.0406 MB 0.991
stencil/dagger/N=256 (block 256)/multi-expr 1.72 k allocs: 0.0865 MB 1.71 k allocs: 0.0863 MB 1
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.23 k allocs: 0.0708 MB 1.3 k allocs: 0.0735 MB 0.964
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.2 k allocs: 0.0702 MB 1.19 k allocs: 0.0694 MB 1.01
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.25 k allocs: 0.0797 MB 1.31 k allocs: 0.0818 MB 0.975
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.17 k allocs: 0.0619 MB 1.2 k allocs: 0.0632 MB 0.979
stencil/dagger/N=256 (block 256)/update (+) 0.917 k allocs: 0.0459 MB 0.917 k allocs: 0.0459 MB 1
time_to_load 0.147 k allocs: 10.8 kB 0.147 k allocs: 10.8 kB 1

Plots

⚠️ Regressions (time > 25.0% and outside the reported ±spread; allocs/memory > 25.0%)

  • stencil/dagger/N=1024 (block 512)/update (+) (memory): +1739.3%
  • array/dagger/N=1024 (block 512)/alloc (rand) (time): +37.4%
Within noise (9 metric(s) past threshold but inside the ±spread; not counted)
  • stencil/dagger/N=256 (block 256)/neighbors (Pad) (time): 83.2%
  • stencil/dagger/N=1024 (block 512)/update (+) (time): 33.1%
  • linalg/dagger/N=256 (block 256)/cholesky (time): -25.4%
  • stencil/dagger/N=256 (block 256)/multi-expr (time): -26.8%
  • linalg/dagger/N=256 (block 256)/lu (time): -31.3%
  • stencil/dagger/N=256 (block 256)/update (+) (time): -35.4%
  • stencil/dagger/N=1024 (block 512)/multi-expr (time): -45.3%
  • linalg/dagger/N=256 (block 256)/matvec (A*x) (time): -45.8%
  • stencil/dagger/N=1024 (block 512)/assign (const) (time): -51.4%

Full results and plots (download the benchmark-results artifact).

estimate_task_costs! dropped the est_business term (each candidate
processor's currently-reserved compute pressure) in b16afad, leaving
only a fixed per-task cross-worker transfer cost in the comparison.
That fixed cost always makes the scheduler's own worker look cheapest,
so under the default scope every task piled onto worker 1 regardless
of how busy it became (#663).

Restore the pressure term, reading it from worker_time_pressure's
current LockedObject/atomic-counter representation, so busier
processors are considered costlier and work spreads across workers
again.

Fixes #663

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jpsamaroo
jpsamaroo merged commit bd8f56b into master Aug 27, 2026
2 of 4 checks passed
@jpsamaroo
jpsamaroo deleted the jps/fix-663 branch August 27, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default scope ignores distributed workers with SlurmClusterManager

1 participant