Skip to content

Fix stencils over-transfer of neighbor data - #737

Draft
jpsamaroo wants to merge 7 commits into
masterfrom
jps/mpi-bad-scale-stencil
Draft

Fix stencils over-transfer of neighbor data#737
jpsamaroo wants to merge 7 commits into
masterfrom
jps/mpi-bad-scale-stencil

Conversation

@jpsamaroo

@jpsamaroo jpsamaroo commented Aug 13, 2026

Copy link
Copy Markdown
Member

This is a regression introduced by prior stencil optimizations, which improved multi-threaded execution but significantly regressed multi-process and multi-rank.

Written by Claude Opus

@github-actions

github-actions Bot commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Dagger benchmarks: dirty vs master

Multi-threaded benchmarks (4 threads)

Dagger benchmarks: dirty vs master

Median time

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 19.6 ± 0.89 ms 20 ± 1.5 ms 0.98 ± 0.086
array/dagger/N=1024 (block 128)/alloc (rand) 13.1 ± 0.58 ms 13.4 ± 1.3 ms 0.976 ± 0.1
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 11.8 ± 0.76 ms 14.8 ± 1.8 ms 0.796 ± 0.11
array/dagger/N=1024 (block 128)/map (sin.(X)) 11.1 ± 0.93 ms 10.7 ± 1.4 ms 1.03 ± 0.16
array/dagger/N=1024 (block 128)/norm 13.5 ± 1 ms 12 ± 0.87 ms 1.13 ± 0.12
array/dagger/N=1024 (block 128)/reduce (sum) 26.2 ± 0.96 ms 27.1 ± 0.24 ms 0.965 ± 0.036
array/dagger/N=1024 (block 128)/transpose (permutedims) 13.1 ± 1.1 ms 15.1 ± 1 ms 0.871 ± 0.094
array/dagger/N=1024 (block 512)/add (X + X) 2.71 ± 0.74 ms 3.98 ± 0.94 ms 0.682 ± 0.25
array/dagger/N=1024 (block 512)/alloc (rand) 2.29 ± 0.73 ms 1.7 ± 0.86 ms 1.35 ± 0.81
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 2.1 ± 1.1 ms 2.12 ± 0.63 ms 0.988 ± 0.6
array/dagger/N=1024 (block 512)/map (sin.(X)) 6.47 ± 1.8 ms 6.55 ± 0.35 ms 0.989 ± 0.27
array/dagger/N=1024 (block 512)/norm 1.07 ± 0.036 ms 2.77 ± 1.9 ms 0.386 ± 0.27
array/dagger/N=1024 (block 512)/reduce (sum) 1.91 ± 0.11 ms 2.96 ± 1.8 ms 0.644 ± 0.4
array/dagger/N=1024 (block 512)/transpose (permutedims) 4.97 ± 0.49 ms 4 ± 0.56 ms 1.24 ± 0.21
array/dagger/N=256 (block 128)/add (X + X) 1.78 ± 1.3 ms 1.72 ± 1.2 ms 1.03 ± 1
array/dagger/N=256 (block 128)/alloc (rand) 1.85 ± 1 ms 1.55 ± 0.56 ms 1.19 ± 0.79
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 0.897 ± 0.073 ms 1.03 ± 0.033 ms 0.869 ± 0.076
array/dagger/N=256 (block 128)/map (sin.(X)) 1.18 ± 0.13 ms 1.25 ± 0.62 ms 0.944 ± 0.48
array/dagger/N=256 (block 128)/norm 1.07 ± 0.31 ms 1.01 ± 0.2 ms 1.06 ± 0.37
array/dagger/N=256 (block 128)/reduce (sum) 2.13 ± 0.51 ms 2.53 ± 0.35 ms 0.841 ± 0.23
array/dagger/N=256 (block 128)/transpose (permutedims) 1.58 ± 0.046 ms 2.09 ± 0.17 ms 0.754 ± 0.066
array/dagger/N=256 (block 256)/add (X + X) 0.955 ± 0.47 ms 1.14 ± 0.037 ms 0.841 ± 0.41
array/dagger/N=256 (block 256)/alloc (rand) 0.686 ± 0.0069 ms 0.728 ± 0.024 ms 0.943 ± 0.033
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.398 ± 0.038 ms 0.449 ± 0.42 ms 0.887 ± 0.82
array/dagger/N=256 (block 256)/map (sin.(X)) 1.06 ± 0.027 ms 1.06 ± 0.042 ms 1 ± 0.047
array/dagger/N=256 (block 256)/norm 0.424 ± 0.12 ms 0.441 ± 0.081 ms 0.962 ± 0.33
array/dagger/N=256 (block 256)/reduce (sum) 0.597 ± 0.026 ms 0.651 ± 0.51 ms 0.917 ± 0.72
array/dagger/N=256 (block 256)/transpose (permutedims) 0.733 ± 0.026 ms 0.769 ± 0.037 ms 0.953 ± 0.056
linalg/dagger/N=1024 (block 128)/cholesky 0.0612 ± 0.0053 s 0.0687 ± 0.012 s 0.892 ± 0.18
linalg/dagger/N=1024 (block 128)/lu 0.132 ± 0.019 s 0.133 ± 0.012 s 0.993 ± 0.17
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.166 ± 0.029 s 0.154 ± 0.024 s 1.07 ± 0.25
linalg/dagger/N=1024 (block 128)/matvec (A*x) 15.7 ± 1.3 ms 16.1 ± 2 ms 0.975 ± 0.14
linalg/dagger/N=1024 (block 128)/qr 0.152 ± 0.02 s 0.149 ± 0.017 s 1.02 ± 0.18
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.217 ± 0.019 s 0.2 ± 0.019 s 1.08 ± 0.14
linalg/dagger/N=1024 (block 128)/svd 28.7 s 27.9 s 1.03
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.0872 ± 0.03 s 0.0882 ± 0.015 s 0.988 ± 0.38
linalg/dagger/N=1024 (block 512)/cholesky 18.7 ± 4 ms 19 ± 7.5 ms 0.985 ± 0.44
linalg/dagger/N=1024 (block 512)/lu 0.0517 ± 0.0031 s 0.042 ± 0.0042 s 1.23 ± 0.14
linalg/dagger/N=1024 (block 512)/matmul (A*A) 0.0706 ± 0.025 s 0.0545 ± 0.0032 s 1.29 ± 0.46
linalg/dagger/N=1024 (block 512)/matvec (A*x) 2.98 ± 1.4 ms 2.14 ± 0.33 ms 1.4 ± 0.71
linalg/dagger/N=1024 (block 512)/qr 0.109 ± 0.0014 s 0.109 ± 0.0022 s 0.996 ± 0.024
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.0539 ± 0.0029 s 0.0564 ± 0.0021 s 0.957 ± 0.063
linalg/dagger/N=1024 (block 512)/svd 0.0391 h 0.0371 h 1.05
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 0.0373 ± 0.0026 s 0.0378 ± 0.00086 s 0.987 ± 0.071
linalg/dagger/N=256 (block 128)/cholesky 7.98 ± 4.8 ms 7.24 ± 3.1 ms 1.1 ± 0.81
linalg/dagger/N=256 (block 128)/lu 12.2 ± 1.8 ms 11.2 ± 4.1 ms 1.09 ± 0.43
linalg/dagger/N=256 (block 128)/matmul (A*A) 6.83 ± 2.7 ms 9.97 ± 6.2 ms 0.685 ± 0.5
linalg/dagger/N=256 (block 128)/matvec (A*x) 5 ± 1.4 ms 3.12 ± 0.88 ms 1.6 ± 0.63
linalg/dagger/N=256 (block 128)/qr 8.98 ± 3.5 ms 8.9 ± 2.1 ms 1.01 ± 0.46
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 20.1 ± 1.5 ms 27.5 ± 12 ms 0.73 ± 0.33
linalg/dagger/N=256 (block 128)/svd 0.737 ± 0.08 s 0.391 ± 0.093 s 1.89 ± 0.49
linalg/dagger/N=256 (block 128)/syrk (A'*A) 6.39 ± 1.9 ms 9.39 ± 7.4 ms 0.681 ± 0.57
linalg/dagger/N=256 (block 256)/cholesky 4.71 ± 3.3 ms 5.91 ± 2.3 ms 0.796 ± 0.64
linalg/dagger/N=256 (block 256)/lu 6.94 ± 2.1 ms 3.49 ± 0.15 ms 1.99 ± 0.6
linalg/dagger/N=256 (block 256)/matmul (A*A) 2.21 ± 0.61 ms 2.32 ± 0.88 ms 0.956 ± 0.45
linalg/dagger/N=256 (block 256)/matvec (A*x) 1.31 ± 0.44 ms 1.35 ± 0.47 ms 0.97 ± 0.47
linalg/dagger/N=256 (block 256)/qr 4.67 ± 0.69 ms 5.11 ± 2.5 ms 0.913 ± 0.47
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 9.42 ± 9.6 ms 10.3 ± 1.4 ms 0.918 ± 0.94
linalg/dagger/N=256 (block 256)/svd 0.71 ± 0.014 s 0.276 ± 0.017 s 2.57 ± 0.17
linalg/dagger/N=256 (block 256)/syrk (A'*A) 2.81 ± 1.1 ms 3.32 ± 2.7 ms 0.848 ± 0.76
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 0.819 ± 0.019 s 0.808 ± 0.02 s 1.01 ± 0.034
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 0.458 ± 0.017 s 0.464 ± 0.0064 s 0.987 ± 0.04
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.049 ± 0.0067 s 0.0536 ± 0.0087 s 0.915 ± 0.19
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 0.808 ± 0.03 s 0.798 ± 0.012 s 1.01 ± 0.041
sparse/dagger/N=256 (block 16)/spgemm (S*S) 0.46 ± 0.014 s 0.456 ± 0.013 s 1.01 ± 0.042
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.0495 ± 0.006 s 0.0474 ± 0.00062 s 1.04 ± 0.13
stencil/dagger/N=1024 (block 128)/alloc (neighbors Wrap) 14.8 ± 1.2 ms 16.4 ± 0.85 ms 0.907 ± 0.085
stencil/dagger/N=1024 (block 128)/assign (const) 5.88 ± 0.58 ms 7.43 ± 2.9 ms 0.791 ± 0.31
stencil/dagger/N=1024 (block 128)/multi-expr 12.7 ± 2.3 ms 16.7 ± 3.9 ms 0.759 ± 0.23
stencil/dagger/N=1024 (block 128)/neighbors (Clamp) 11.8 ± 0.094 ms 12 ± 0.34 ms 0.982 ± 0.029
stencil/dagger/N=1024 (block 128)/neighbors (Pad) 12.2 ± 0.9 ms 12.6 ± 0.36 ms 0.969 ± 0.077
stencil/dagger/N=1024 (block 128)/neighbors (Reflect) 11.7 ± 0.87 ms 12.8 ± 2.5 ms 0.914 ± 0.19
stencil/dagger/N=1024 (block 128)/neighbors (Wrap) 12.3 ± 0.94 ms 12.9 ± 1.6 ms 0.95 ± 0.14
stencil/dagger/N=1024 (block 128)/update (+) 6.94 ± 1.2 ms 7.27 ± 1.3 ms 0.954 ± 0.23
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 6.47 ± 0.94 ms 6.69 ± 0.21 ms 0.967 ± 0.14
stencil/dagger/N=1024 (block 512)/assign (const) 0.802 ± 1.9 ms 0.813 ± 1 ms 0.987 ± 2.6
stencil/dagger/N=1024 (block 512)/multi-expr 3.89 ± 1.4 ms 2.28 ± 0.62 ms 1.7 ± 0.77
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 6.18 ± 1.6 ms 5.74 ± 1.5 ms 1.08 ± 0.4
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 6.26 ± 0.55 ms 5.95 ± 1.7 ms 1.05 ± 0.31
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 5.97 ± 0.94 ms 6.06 ± 0.064 ms 0.986 ± 0.16
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 6.03 ± 2 ms 6.62 ± 2.8 ms 0.911 ± 0.49
stencil/dagger/N=1024 (block 512)/update (+) 1.37 ± 1.7 ms 3.53 ± 1.8 ms 0.388 ± 0.52
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap) 3.76 ± 3.3 ms 2.13 ± 0.11 ms 1.76 ± 1.5
stencil/dagger/N=256 (block 128)/assign (const) 1.37 ± 0.75 ms 1.34 ± 0.29 ms 1.02 ± 0.6
stencil/dagger/N=256 (block 128)/multi-expr 1.95 ± 1.1 ms 2.27 ± 2.1 ms 0.857 ± 0.93
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 1.39 ± 2.7 ms 2.2 ± 0.53 ms 0.633 ± 1.2
stencil/dagger/N=256 (block 128)/neighbors (Pad) 3.04 ± 1.6 ms 2.25 ± 1.4 ms 1.35 ± 1.1
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 3.61 ± 0.74 ms 2.03 ± 3.2 ms 1.78 ± 2.8
stencil/dagger/N=256 (block 128)/neighbors (Wrap) 1.39 ± 0.3 ms 2.12 ± 0.66 ms 0.659 ± 0.25
stencil/dagger/N=256 (block 128)/update (+) 1.2 ± 0.41 ms 0.864 ± 3.3 ms 1.39 ± 5.3
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 2.25 ± 1.2 ms 3.41 ± 1.5 ms 0.659 ± 0.45
stencil/dagger/N=256 (block 256)/assign (const) 0.509 ± 0.16 ms 0.504 ± 0.026 ms 1.01 ± 0.32
stencil/dagger/N=256 (block 256)/multi-expr 0.906 ± 0.085 ms 2.58 ± 1.6 ms 0.351 ± 0.23
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.24 ± 0.33 ms 1.14 ± 0.0093 ms 1.08 ± 0.29
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.61 ± 0.21 ms 2.15 ± 0.36 ms 0.748 ± 0.16
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.37 ± 2 ms 1.62 ± 0.01 ms 0.845 ± 1.2
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.16 ± 0.0099 ms 1.75 ± 0.48 ms 0.664 ± 0.18
stencil/dagger/N=256 (block 256)/update (+) 1.37 ± 0.94 ms 0.845 ± 0.5 ms 1.62 ± 1.5
time_to_load 1.08 ± 0.0034 s 1.08 ± 0.0028 s 0.995 ± 0.0041

Allocations / memory

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 0.0784 M allocs: 11 MB 0.0788 M allocs: 11 MB 0.998
array/dagger/N=1024 (block 128)/alloc (rand) 0.043 M allocs: 9.44 MB 0.0431 M allocs: 9.44 MB 1
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 0.0382 M allocs: 9.3 MB 0.0376 M allocs: 9.28 MB 1
array/dagger/N=1024 (block 128)/map (sin.(X)) 0.0331 M allocs: 9.11 MB 0.0329 M allocs: 9.1 MB 1
array/dagger/N=1024 (block 128)/norm 0.0426 M allocs: 1.41 MB 0.0425 M allocs: 1.41 MB 1
array/dagger/N=1024 (block 128)/reduce (sum) 0.0857 M allocs: 2.91 MB 0.0854 M allocs: 2.91 MB 1
array/dagger/N=1024 (block 128)/transpose (permutedims) 0.0446 M allocs: 9.73 MB 0.0447 M allocs: 9.73 MB 1
array/dagger/N=1024 (block 512)/add (X + X) 5.58 k allocs: 8.22 MB 5.51 k allocs: 8.22 MB 1
array/dagger/N=1024 (block 512)/alloc (rand) 2.76 k allocs: 8.09 MB 2.75 k allocs: 8.09 MB 1
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 2.4 k allocs: 8.08 MB 2.4 k allocs: 8.08 MB 1
array/dagger/N=1024 (block 512)/map (sin.(X)) 2.08 k allocs: 8.07 MB 2.1 k allocs: 8.07 MB 1
array/dagger/N=1024 (block 512)/norm 2.7 k allocs: 0.0897 MB 2.71 k allocs: 0.0902 MB 0.995
array/dagger/N=1024 (block 512)/reduce (sum) 4.13 k allocs: 0.142 MB 4.57 k allocs: 0.158 MB 0.898
array/dagger/N=1024 (block 512)/transpose (permutedims) 3.33 k allocs: 8.14 MB 3.31 k allocs: 8.14 MB 1
array/dagger/N=256 (block 128)/add (X + X) 5.57 k allocs: 0.717 MB 5.51 k allocs: 0.715 MB 1
array/dagger/N=256 (block 128)/alloc (rand) 2.77 k allocs: 0.593 MB 2.71 k allocs: 0.591 MB 1
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 2.43 k allocs: 0.583 MB 2.43 k allocs: 0.583 MB 1
array/dagger/N=256 (block 128)/map (sin.(X)) 2.12 k allocs: 0.571 MB 2.1 k allocs: 0.571 MB 1
array/dagger/N=256 (block 128)/norm 2.7 k allocs: 0.0903 MB 2.67 k allocs: 0.089 MB 1.01
array/dagger/N=256 (block 128)/reduce (sum) 4.57 k allocs: 0.157 MB 4.57 k allocs: 0.157 MB 1
array/dagger/N=256 (block 128)/transpose (permutedims) 3.35 k allocs: 0.637 MB 3.37 k allocs: 0.637 MB 0.999
array/dagger/N=256 (block 256)/add (X + X) 1.73 k allocs: 0.575 MB 1.73 k allocs: 0.575 MB 0.999
array/dagger/N=256 (block 256)/alloc (rand) 0.754 k allocs: 0.526 MB 0.754 k allocs: 0.526 MB 1
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.651 k allocs: 0.522 MB 0.651 k allocs: 0.522 MB 1
array/dagger/N=256 (block 256)/map (sin.(X)) 0.574 k allocs: 0.52 MB 0.574 k allocs: 0.52 MB 1
array/dagger/N=256 (block 256)/norm 0.715 k allocs: 24.6 kB 0.715 k allocs: 24.6 kB 1
array/dagger/N=256 (block 256)/reduce (sum) 0.816 k allocs: 30.5 kB 0.816 k allocs: 30.5 kB 1
array/dagger/N=256 (block 256)/transpose (permutedims) 1.11 k allocs: 0.551 MB 1.11 k allocs: 0.551 MB 1
linalg/dagger/N=1024 (block 128)/cholesky 0.122 M allocs: 15.8 MB 0.128 M allocs: 16.1 MB 0.987
linalg/dagger/N=1024 (block 128)/lu 0.323 M allocs: 24 MB 0.319 M allocs: 23.9 MB 1
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.239 M allocs: 15.9 MB 0.239 M allocs: 15.9 MB 1
linalg/dagger/N=1024 (block 128)/matvec (A*x) 0.0663 M allocs: 2.46 MB 0.067 M allocs: 2.49 MB 0.99
linalg/dagger/N=1024 (block 128)/qr 0.243 M allocs: 25.1 MB 0.233 M allocs: 24.8 MB 1.01
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.445 M allocs: 28.7 MB 0.425 M allocs: 27.8 MB 1.03
linalg/dagger/N=1024 (block 128)/svd 3.76 M allocs: 2.32 GB 3.79 M allocs: 2.32 GB 1
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.172 M allocs: 17 MB 0.173 M allocs: 17.1 MB 0.999
linalg/dagger/N=1024 (block 512)/cholesky 7.81 k allocs: 10.3 MB 7.79 k allocs: 10.3 MB 1
linalg/dagger/N=1024 (block 512)/lu 15.6 k allocs: 14.6 MB 15.5 k allocs: 14.6 MB 1
linalg/dagger/N=1024 (block 512)/matmul (A*A) 7.72 k allocs: 8.29 MB 7.69 k allocs: 8.28 MB 1
linalg/dagger/N=1024 (block 512)/matvec (A*x) 6.26 k allocs: 0.249 MB 6.32 k allocs: 0.251 MB 0.991
linalg/dagger/N=1024 (block 512)/qr 11.9 k allocs: 9.61 MB 11.2 k allocs: 9.58 MB 1
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 30.3 k allocs: 15.2 MB 29.5 k allocs: 15.2 MB 1
linalg/dagger/N=1024 (block 512)/svd 0.0498 M allocs: 0.197 GB 0.0492 M allocs: 0.197 GB 1
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 9.09 k allocs: 20.4 MB 9.13 k allocs: 20.4 MB 1
linalg/dagger/N=256 (block 128)/cholesky 7.85 k allocs: 0.936 MB 7.84 k allocs: 0.936 MB 1
linalg/dagger/N=256 (block 128)/lu 15.6 k allocs: 1.49 MB 15.6 k allocs: 1.49 MB 1
linalg/dagger/N=256 (block 128)/matmul (A*A) 7.79 k allocs: 0.788 MB 7.79 k allocs: 0.788 MB 1
linalg/dagger/N=256 (block 128)/matvec (A*x) 6.33 k allocs: 0.245 MB 6.46 k allocs: 0.251 MB 0.976
linalg/dagger/N=256 (block 128)/qr 11.4 k allocs: 1.23 MB 11.2 k allocs: 1.23 MB 1
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 30.3 k allocs: 2.09 MB 29.9 k allocs: 2.08 MB 1.01
linalg/dagger/N=256 (block 128)/svd 0.0487 M allocs: 14.6 MB 0.0488 M allocs: 14.6 MB 1
linalg/dagger/N=256 (block 128)/syrk (A'*A) 9.13 k allocs: 1.61 MB 9.09 k allocs: 1.61 MB 1
linalg/dagger/N=256 (block 256)/cholesky 3.04 k allocs: 0.634 MB 3.03 k allocs: 0.634 MB 1
linalg/dagger/N=256 (block 256)/lu 5.42 k allocs: 1.24 MB 5.4 k allocs: 1.24 MB 1
linalg/dagger/N=256 (block 256)/matmul (A*A) 1.92 k allocs: 0.581 MB 1.94 k allocs: 0.582 MB 0.999
linalg/dagger/N=256 (block 256)/matvec (A*x) 2.35 k allocs: 0.0994 MB 2.36 k allocs: 0.0995 MB 0.999
linalg/dagger/N=256 (block 256)/qr 3.45 k allocs: 0.779 MB 3.45 k allocs: 0.78 MB 1
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 11.9 k allocs: 1.53 MB 11.9 k allocs: 1.53 MB 1
linalg/dagger/N=256 (block 256)/svd 15.5 k allocs: 6.71 MB 15.5 k allocs: 6.71 MB 1
linalg/dagger/N=256 (block 256)/syrk (A'*A) 3.05 k allocs: 2.14 MB 3.06 k allocs: 2.14 MB 1
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 3.01 M allocs: 0.111 GB 2.96 M allocs: 0.11 GB 1.02
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 3.93 M allocs: 0.175 GB 3.94 M allocs: 0.175 GB 0.999
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.182 M allocs: 6.92 MB 0.182 M allocs: 6.94 MB 0.997
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 2.99 M allocs: 0.111 GB 2.95 M allocs: 0.109 GB 1.01
sparse/dagger/N=256 (block 16)/spgemm (S*S) 3.92 M allocs: 0.139 GB 3.78 M allocs: 0.133 GB 1.05
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.191 M allocs: 7.24 MB 0.172 M allocs: 7.4 MB 0.978
stencil/dagger/N=1024 (block 128)/alloc (neighbors Wrap) 0.0647 M allocs: 10.7 MB 0.067 M allocs: 10.8 MB 0.992
stencil/dagger/N=1024 (block 128)/assign (const) 24.7 k allocs: 1.05 MB 25.3 k allocs: 1.04 MB 1
stencil/dagger/N=1024 (block 128)/multi-expr 0.058 M allocs: 2.47 MB 0.0582 M allocs: 2.49 MB 0.992
stencil/dagger/N=1024 (block 128)/neighbors (Clamp) 0.0489 M allocs: 2.4 MB 0.0504 M allocs: 2.56 MB 0.938
stencil/dagger/N=1024 (block 128)/neighbors (Pad) 0.0479 M allocs: 2.35 MB 0.0497 M allocs: 2.42 MB 0.973
stencil/dagger/N=1024 (block 128)/neighbors (Reflect) 0.0485 M allocs: 2.41 MB 0.05 M allocs: 2.57 MB 0.937
stencil/dagger/N=1024 (block 128)/neighbors (Wrap) 0.0472 M allocs: 2.12 MB 0.0492 M allocs: 2.33 MB 0.91
stencil/dagger/N=1024 (block 128)/update (+) 0.0328 M allocs: 1.42 MB 0.0329 M allocs: 1.42 MB 1
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 4.76 k allocs: 8.2 MB 4.96 k allocs: 8.21 MB 0.999
stencil/dagger/N=1024 (block 512)/assign (const) 2.16 k allocs: 0.0948 MB 2.15 k allocs: 0.0943 MB 1.01
stencil/dagger/N=1024 (block 512)/multi-expr 4.82 k allocs: 0.212 MB 4.88 k allocs: 2.21 MB 0.0956
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 3.82 k allocs: 0.232 MB 4.02 k allocs: 2.24 MB 0.104
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 3.7 k allocs: 0.228 MB 3.78 k allocs: 0.23 MB 0.992
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 3.92 k allocs: 2.27 MB 3.92 k allocs: 0.265 MB 8.55
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 3.68 k allocs: 0.169 MB 3.79 k allocs: 0.174 MB 0.971
stencil/dagger/N=1024 (block 512)/update (+) 2.58 k allocs: 0.114 MB 2.62 k allocs: 0.116 MB 0.986
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap) 4.88 k allocs: 0.706 MB 4.98 k allocs: 0.712 MB 0.993
stencil/dagger/N=256 (block 128)/assign (const) 2.16 k allocs: 0.0951 MB 2.14 k allocs: 0.0942 MB 1.01
stencil/dagger/N=256 (block 128)/multi-expr 4.83 k allocs: 0.213 MB 4.78 k allocs: 0.211 MB 1.01
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 3.83 k allocs: 0.187 MB 3.92 k allocs: 0.188 MB 0.99
stencil/dagger/N=256 (block 128)/neighbors (Pad) 3.74 k allocs: 0.185 MB 3.78 k allocs: 0.184 MB 1.01
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 3.83 k allocs: 0.195 MB 3.89 k allocs: 0.196 MB 0.995
stencil/dagger/N=256 (block 128)/neighbors (Wrap) 3.69 k allocs: 0.294 MB 3.81 k allocs: 0.174 MB 1.69
stencil/dagger/N=256 (block 128)/update (+) 2.65 k allocs: 0.118 MB 2.63 k allocs: 0.116 MB 1.01
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 1.54 k allocs: 0.574 MB 1.59 k allocs: 0.576 MB 0.997
stencil/dagger/N=256 (block 256)/assign (const) 0.811 k allocs: 0.041 MB 0.811 k allocs: 0.041 MB 1
stencil/dagger/N=256 (block 256)/multi-expr 1.75 k allocs: 0.0877 MB 1.73 k allocs: 0.0873 MB 1
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.26 k allocs: 0.0715 MB 1.3 k allocs: 0.0729 MB 0.981
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.21 k allocs: 0.07 MB 1.24 k allocs: 0.0708 MB 0.989
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.27 k allocs: 0.0796 MB 1.28 k allocs: 0.0798 MB 0.997
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.2 k allocs: 0.063 MB 1.23 k allocs: 0.064 MB 0.984
stencil/dagger/N=256 (block 256)/update (+) 0.902 k allocs: 0.0456 MB 0.916 k allocs: 0.046 MB 0.991
time_to_load 0.147 k allocs: 10.8 kB 0.147 k allocs: 10.8 kB 1

Plots

⚠️ Regressions (time > 25.0% and outside the reported ±spread; allocs/memory > 25.0%)

  • stencil/dagger/N=1024 (block 512)/multi-expr (memory): +946.5%
  • stencil/dagger/N=1024 (block 512)/neighbors (Clamp) (memory): +864.2%
  • stencil/dagger/N=256 (block 256)/neighbors (Wrap) (time): +50.7%
  • array/dagger/N=256 (block 128)/transpose (permutedims) (time): +32.6%
  • array/dagger/N=1024 (block 128)/broadcast (X .+ 1) (time): +25.7%

Improvements

  • stencil/dagger/N=1024 (block 512)/neighbors (Reflect) (memory): -88.3%
  • linalg/dagger/N=256 (block 256)/svd (time): -61.1%
  • linalg/dagger/N=256 (block 256)/lu (time): -49.7%
  • linalg/dagger/N=256 (block 128)/svd (time): -47.0%
  • stencil/dagger/N=256 (block 128)/neighbors (Wrap) (memory): -40.8%
Within noise (24 metric(s) past threshold but inside the ±spread; not counted)
  • stencil/dagger/N=256 (block 256)/multi-expr (time): 184.7%
  • array/dagger/N=1024 (block 512)/norm (time): 159.0%
  • stencil/dagger/N=1024 (block 512)/update (+) (time): 157.9%
  • stencil/dagger/N=256 (block 128)/neighbors (Clamp) (time): 58.1%
  • array/dagger/N=1024 (block 512)/reduce (sum) (time): 55.2%
  • stencil/dagger/N=256 (block 128)/neighbors (Wrap) (time): 51.9%
  • stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) (time): 51.7%
  • linalg/dagger/N=256 (block 128)/syrk (A'*A) (time): 46.9%
  • array/dagger/N=1024 (block 512)/add (X + X) (time): 46.7%
  • linalg/dagger/N=256 (block 128)/matmul (A*A) (time): 46.0%
  • linalg/dagger/N=256 (block 128)/solve (A\b via lu) (time): 37.0%
  • stencil/dagger/N=256 (block 256)/neighbors (Pad) (time): 33.6%
  • stencil/dagger/N=1024 (block 128)/multi-expr (time): 31.8%
  • stencil/dagger/N=1024 (block 128)/assign (const) (time): 26.4%
  • linalg/dagger/N=256 (block 256)/cholesky (time): 25.6%
  • array/dagger/N=1024 (block 512)/alloc (rand) (time): -25.7%
  • stencil/dagger/N=256 (block 128)/neighbors (Pad) (time): -26.0%
  • stencil/dagger/N=256 (block 128)/update (+) (time): -28.2%
  • linalg/dagger/N=1024 (block 512)/matvec (A*x) (time): -28.3%
  • linalg/dagger/N=256 (block 128)/matvec (A*x) (time): -37.6%
  • stencil/dagger/N=256 (block 256)/update (+) (time): -38.3%
  • stencil/dagger/N=1024 (block 512)/multi-expr (time): -41.3%
  • stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap) (time): -43.3%
  • stencil/dagger/N=256 (block 128)/neighbors (Reflect) (time): -43.9%
Distributed benchmarks (4 processes)

Dagger benchmarks: dirty vs master

Median time

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 0.245 ± 0.039 s 0.0808 ± 0.027 s 3.03 ± 1.1
array/dagger/N=1024 (block 128)/alloc (rand) 23.9 ± 4.2 ms 20.4 ± 0.97 ms 1.17 ± 0.21
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 29.7 ± 1.7 ms 28.3 ± 1.5 ms 1.05 ± 0.081
array/dagger/N=1024 (block 128)/map (sin.(X)) 0.0386 ± 0.0015 s 0.0324 ± 0.0016 s 1.19 ± 0.074
array/dagger/N=1024 (block 128)/norm 23.6 ± 1.7 ms 18.3 ± 2.7 ms 1.29 ± 0.21
array/dagger/N=1024 (block 128)/reduce (sum) 0.0445 ± 0.28 s 0.0402 ± 0.0027 s 1.11 ± 6.9
array/dagger/N=1024 (block 128)/transpose (permutedims) 0.0577 ± 0.016 s 26.1 ± 1.1 ms 2.21 ± 0.61
array/dagger/N=1024 (block 512)/add (X + X) 19.5 ± 91 ms 11.2 ± 3.3 ms 1.73 ± 8.1
array/dagger/N=1024 (block 512)/alloc (rand) 2.23 ± 0.031 ms 2.4 ± 0.15 ms 0.929 ± 0.059
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 3.99 ± 0.24 ms 4.31 ± 0.15 ms 0.926 ± 0.064
array/dagger/N=1024 (block 512)/map (sin.(X)) 8.08 ± 0.73 ms 7.26 ± 0.22 ms 1.11 ± 0.11
array/dagger/N=1024 (block 512)/norm 2.58 ± 0.31 ms 2.56 ± 0.1 ms 1.01 ± 0.13
array/dagger/N=1024 (block 512)/reduce (sum) 3.49 ± 0.58 ms 3.04 ± 0.72 ms 1.15 ± 0.33
array/dagger/N=1024 (block 512)/transpose (permutedims) 7.93 ± 0.49 ms 7.83 ± 36 ms 1.01 ± 4.7
array/dagger/N=256 (block 128)/add (X + X) 1.7 ± 0.094 ms 1.63 ± 0.33 ms 1.05 ± 0.22
array/dagger/N=256 (block 128)/alloc (rand) 1.31 ± 0.027 ms 1.27 ± 0.029 ms 1.03 ± 0.032
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 2.7 ± 0.32 ms 2.67 ± 0.098 ms 1.01 ± 0.13
array/dagger/N=256 (block 128)/map (sin.(X)) 1.77 ± 0.61 ms 1.61 ± 0.62 ms 1.1 ± 0.57
array/dagger/N=256 (block 128)/norm 0.961 ± 0.65 ms 0.868 ± 0.61 ms 1.11 ± 1.1
array/dagger/N=256 (block 128)/reduce (sum) 0.0764 ± 0.037 s 4.01 ± 34 ms 19 ± 1.6e+02
array/dagger/N=256 (block 128)/transpose (permutedims) 1.25 ± 0.057 ms 1.18 ± 0.01 ms 1.05 ± 0.049
array/dagger/N=256 (block 256)/add (X + X) 0.905 ± 0.011 ms 0.889 ± 0.076 ms 1.02 ± 0.088
array/dagger/N=256 (block 256)/alloc (rand) 0.825 ± 0.047 ms 0.842 ± 0.009 ms 0.979 ± 0.056
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.494 ± 0.061 ms 0.574 ± 0.068 ms 0.861 ± 0.15
array/dagger/N=256 (block 256)/map (sin.(X)) 0.945 ± 0.019 ms 0.974 ± 0.035 ms 0.97 ± 0.04
array/dagger/N=256 (block 256)/norm 0.426 ± 0.0056 ms 0.402 ± 0.017 ms 1.06 ± 0.046
array/dagger/N=256 (block 256)/reduce (sum) 0.571 ± 0.019 ms 0.609 ± 0.038 ms 0.937 ± 0.067
array/dagger/N=256 (block 256)/transpose (permutedims) 0.745 ± 0.021 ms 0.746 ± 0.013 ms 0.999 ± 0.033
linalg/dagger/N=1024 (block 128)/cholesky 0.599 ± 0.14 s 0.441 ± 0.0064 s 1.36 ± 0.31
linalg/dagger/N=1024 (block 128)/lu 0.684 ± 0.45 s 0.22 ± 0.0045 s 3.1 ± 2.1
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.355 ± 0.057 s 0.383 ± 0.093 s 0.929 ± 0.27
linalg/dagger/N=1024 (block 128)/matvec (A*x) 0.107 ± 0.0044 s 0.0745 ± 0.0091 s 1.43 ± 0.18
linalg/dagger/N=1024 (block 128)/qr 0.378 ± 0.021 s 0.308 ± 0.011 s 1.23 ± 0.081
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.266 ± 0.0093 s 0.255 ± 0.017 s 1.04 ± 0.078
linalg/dagger/N=1024 (block 128)/svd 46.5 s 46 s 1.01
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.286 ± 0.046 s 0.161 ± 0.0029 s 1.78 ± 0.29
linalg/dagger/N=1024 (block 512)/cholesky 0.172 ± 0.1 s 0.0497 ± 0.12 s 3.46 ± 8.5
linalg/dagger/N=1024 (block 512)/lu 0.0329 ± 0.0055 s 0.0344 ± 0.0014 s 0.957 ± 0.16
linalg/dagger/N=1024 (block 512)/matmul (A*A) 0.0501 ± 0.014 s 25.3 ± 5.3 ms 1.98 ± 0.68
linalg/dagger/N=1024 (block 512)/matvec (A*x) 2.61 ± 1.8 ms 2.51 ± 0.036 ms 1.04 ± 0.7
linalg/dagger/N=1024 (block 512)/qr 0.128 ± 0.00018 s 0.126 ± 0.0012 s 1.01 ± 0.01
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.0422 ± 0.001 s 0.0384 ± 0.0014 s 1.1 ± 0.047
linalg/dagger/N=1024 (block 512)/svd 54.5 s 0.0171 h 0.885
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 25.9 ± 4.9 ms 29.8 ± 0.64 ms 0.868 ± 0.17
linalg/dagger/N=256 (block 128)/cholesky 5.32 ± 1.2e+02 ms 24.2 ± 1.3e+02 ms 0.22 ± 5.2
linalg/dagger/N=256 (block 128)/lu 0.641 ± 0.0029 s 14 ± 3e+02 ms 45.6 ± 9.7e+02
linalg/dagger/N=256 (block 128)/matmul (A*A) 5.29 ± 0.29 ms 4.85 ± 0.014 ms 1.09 ± 0.06
linalg/dagger/N=256 (block 128)/matvec (A*x) 2.07 ± 0.086 ms 1.92 ± 1.9 ms 1.08 ± 1.1
linalg/dagger/N=256 (block 128)/qr 8.73 ± 0.98 ms 7.69 ± 0.047 ms 1.14 ± 0.13
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 0.128 ± 0.069 s 0.0433 ± 0.14 s 2.95 ± 9.5
linalg/dagger/N=256 (block 128)/svd 0.534 ± 0.003 s 0.497 ± 0.0016 s 1.07 ± 0.007
linalg/dagger/N=256 (block 128)/syrk (A'*A) 7.27 ± 0.92 ms 6.42 ± 0.73 ms 1.13 ± 0.19
linalg/dagger/N=256 (block 256)/cholesky 2.33 ± 0.072 ms 2.3 ± 0.33 ms 1.01 ± 0.15
linalg/dagger/N=256 (block 256)/lu 3.58 ± 0.23 ms 3.57 ± 0.16 ms 1 ± 0.078
linalg/dagger/N=256 (block 256)/matmul (A*A) 1.9 ± 0.15 ms 1.96 ± 0.012 ms 0.97 ± 0.077
linalg/dagger/N=256 (block 256)/matvec (A*x) 1.04 ± 0.024 ms 1.05 ± 0.0068 ms 0.99 ± 0.023
linalg/dagger/N=256 (block 256)/qr 4.86 ± 0.12 ms 4.88 ± 0.16 ms 0.996 ± 0.041
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 6.16 ± 0.73 ms 5.82 ± 0.064 ms 1.06 ± 0.13
linalg/dagger/N=256 (block 256)/svd 0.346 ± 0.00032 s 0.348 ± 0.00092 s 0.994 ± 0.0028
linalg/dagger/N=256 (block 256)/syrk (A'*A) 3.04 ± 0.086 ms 3.11 ± 0.15 ms 0.977 ± 0.054
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 10.6 s 3.32 s 3.2
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 5.9 s 5.22 s 1.13
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.58 ± 0.022 s 0.443 ± 0.003 s 1.31 ± 0.049
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 8.63 s 9.52 s 0.906
sparse/dagger/N=256 (block 16)/spgemm (S*S) 5.91 s 5.48 s 1.08
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.676 ± 0.034 s 0.393 ± 0.016 s 1.72 ± 0.11
stencil/dagger/N=1024 (block 128)/assign (const) 0.0379 ± 0.0099 s 30.3 ± 4.3 ms 1.25 ± 0.37
stencil/dagger/N=1024 (block 128)/multi-expr 0.295 ± 0.018 s 0.176 ± 0.00058 s 1.68 ± 0.1
stencil/dagger/N=1024 (block 128)/update (+) 0.266 ± 0.0015 s 0.133 ± 0.0073 s 2 ± 0.11
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 11.5 ± 0.89 ms
stencil/dagger/N=1024 (block 512)/assign (const) 1.55 ± 0.096 ms 2.68 ± 0.84 ms 0.581 ± 0.19
stencil/dagger/N=1024 (block 512)/multi-expr 25.9 ± 0.43 ms 23.8 ± 4.3 ms 1.09 ± 0.2
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 0.0341 ± 0.0008 s
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 31.3 ± 2.6 ms
stencil/dagger/N=1024 (block 512)/update (+) 27 ± 0.74 ms 15.3 ± 4.1 ms 1.77 ± 0.48
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap) 2.61 ± 0.0021 ms
stencil/dagger/N=256 (block 128)/assign (const) 0.908 ± 0.074 ms 0.933 ± 0.025 ms 0.973 ± 0.083
stencil/dagger/N=256 (block 128)/multi-expr 2.06 ± 0.049 ms 2.43 ± 0.028 ms 0.848 ± 0.022
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 2.28 ± 0.017 ms 2.2 ± 0.16 ms 1.04 ± 0.076
stencil/dagger/N=256 (block 128)/neighbors (Pad) 2.07 ± 0.022 ms
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 1.99 ± 0.098 ms 2.12 ± 0.083 ms 0.939 ± 0.059
stencil/dagger/N=256 (block 128)/update (+) 1.65 ± 0.24 ms 2 ± 0.22 ms 0.825 ± 0.15
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 1.73 ± 0.0086 ms
stencil/dagger/N=256 (block 256)/assign (const) 0.711 ± 0.027 ms 0.635 ± 0.00023 ms 1.12 ± 0.043
stencil/dagger/N=256 (block 256)/multi-expr 1.32 ± 0.063 ms 1.65 ± 0.006 ms 0.801 ± 0.038
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.47 ± 0.078 ms 1.6 ± 0.022 ms 0.921 ± 0.05
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.58 ± 0.07 ms
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.5 ± 0.077 ms 1.7 ± 0.084 ms 0.886 ± 0.063
stencil/dagger/N=256 (block 256)/update (+) 0.999 ± 0.0078 ms 1.05 ± 0.0016 ms 0.954 ± 0.0076
time_to_load 1.18 ± 0.0053 s 1.18 ± 0.017 s 0.997 ± 0.015

Allocations / memory

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 0.43 M allocs: 29.5 MB 0.19 M allocs: 18.8 MB 1.57
array/dagger/N=1024 (block 128)/alloc (rand) 0.0497 M allocs: 5.83 MB 0.0515 M allocs: 7.37 MB 0.79
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 0.0776 M allocs: 8.24 MB 0.0779 M allocs: 7.95 MB 1.04
array/dagger/N=1024 (block 128)/map (sin.(X)) 0.0917 M allocs: 7.28 MB 0.0713 M allocs: 7.62 MB 0.956
array/dagger/N=1024 (block 128)/norm 0.0639 M allocs: 2.52 MB 0.0529 M allocs: 2.02 MB 1.25
array/dagger/N=1024 (block 128)/reduce (sum) 0.101 M allocs: 3.93 MB 0.0928 M allocs: 3.69 MB 1.07
array/dagger/N=1024 (block 128)/transpose (permutedims) 0.108 M allocs: 10.6 MB 0.0696 M allocs: 8.61 MB 1.23
array/dagger/N=1024 (block 512)/add (X + X) 6.5 k allocs: 8.25 MB 9.04 k allocs: 10.4 MB 0.796
array/dagger/N=1024 (block 512)/alloc (rand) 2.83 k allocs: 8.1 MB 2.99 k allocs: 4.15 MB 1.95
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 4.22 k allocs: 4.18 MB 3.99 k allocs: 4.17 MB 1
array/dagger/N=1024 (block 512)/map (sin.(X)) 3.94 k allocs: 4.17 MB 5.47 k allocs: 2.24 MB 1.86
array/dagger/N=1024 (block 512)/norm 3.61 k allocs: 0.134 MB 4.67 k allocs: 0.202 MB 0.664
array/dagger/N=1024 (block 512)/reduce (sum) 5.59 k allocs: 0.217 MB 5.06 k allocs: 0.198 MB 1.1
array/dagger/N=1024 (block 512)/transpose (permutedims) 3.58 k allocs: 8.15 MB 6.35 k allocs: 4.28 MB 1.9
array/dagger/N=256 (block 128)/add (X + X) 6.21 k allocs: 0.741 MB 6.16 k allocs: 0.741 MB 1
array/dagger/N=256 (block 128)/alloc (rand) 2.83 k allocs: 0.596 MB 2.83 k allocs: 0.595 MB 1
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 4.46 k allocs: 0.355 MB 4.26 k allocs: 0.359 MB 0.989
array/dagger/N=256 (block 128)/map (sin.(X)) 2.34 k allocs: 0.505 MB 2.33 k allocs: 0.54 MB 0.935
array/dagger/N=256 (block 128)/norm 2.7 k allocs: 0.0886 MB 2.7 k allocs: 0.0886 MB 1
array/dagger/N=256 (block 128)/reduce (sum) 5.4 k allocs: 0.203 MB 5.21 k allocs: 0.2 MB 1.01
array/dagger/N=256 (block 128)/transpose (permutedims) 3.58 k allocs: 0.647 MB 3.52 k allocs: 0.646 MB 1
array/dagger/N=256 (block 256)/add (X + X) 2.04 k allocs: 0.591 MB 2.03 k allocs: 0.59 MB 1
array/dagger/N=256 (block 256)/alloc (rand) 0.853 k allocs: 0.53 MB 0.854 k allocs: 0.53 MB 1
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.649 k allocs: 0.522 MB 0.648 k allocs: 0.522 MB 1
array/dagger/N=256 (block 256)/map (sin.(X)) 0.571 k allocs: 0.519 MB 0.571 k allocs: 0.519 MB 1
array/dagger/N=256 (block 256)/norm 0.712 k allocs: 24.2 kB 0.712 k allocs: 24.2 kB 1
array/dagger/N=256 (block 256)/reduce (sum) 0.814 k allocs: 30.2 kB 0.814 k allocs: 30.2 kB 1
array/dagger/N=256 (block 256)/transpose (permutedims) 1.32 k allocs: 0.563 MB 1.3 k allocs: 0.563 MB 1
linalg/dagger/N=1024 (block 128)/cholesky 0.738 M allocs: 0.0499 GB 0.714 M allocs: 0.0425 GB 1.17
linalg/dagger/N=1024 (block 128)/lu 0.671 M allocs: 0.044 GB 0.6 M allocs: 0.0389 GB 1.13
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.721 M allocs: 0.0429 GB 0.642 M allocs: 0.0327 GB 1.31
linalg/dagger/N=1024 (block 128)/matvec (A*x) 0.22 M allocs: 11.8 MB 0.159 M allocs: 8.24 MB 1.43
linalg/dagger/N=1024 (block 128)/qr 0.449 M allocs: 0.0401 GB 0.385 M allocs: 0.0321 GB 1.25
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.778 M allocs: 0.0437 GB 0.724 M allocs: 0.041 GB 1.06
linalg/dagger/N=1024 (block 128)/svd 22.1 M allocs: 2.75 GB 21.1 M allocs: 2.87 GB 0.959
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.544 M allocs: 0.039 GB 0.436 M allocs: 28.9 MB 1.38
linalg/dagger/N=1024 (block 512)/cholesky 0.0378 M allocs: 13.6 MB 24.6 k allocs: 9.29 MB 1.47
linalg/dagger/N=1024 (block 512)/lu 19.9 k allocs: 14.8 MB 19.4 k allocs: 14.8 MB 1
linalg/dagger/N=1024 (block 512)/matmul (A*A) 11.1 k allocs: 8.4 MB 11.1 k allocs: 8.41 MB 1
linalg/dagger/N=1024 (block 512)/matvec (A*x) 7.13 k allocs: 0.282 MB 7.08 k allocs: 0.281 MB 1
linalg/dagger/N=1024 (block 512)/qr 14.2 k allocs: 9.7 MB 14.1 k allocs: 9.7 MB 1
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.0412 M allocs: 15.7 MB 0.0356 M allocs: 15.5 MB 1.02
linalg/dagger/N=1024 (block 512)/svd 0.221 M allocs: 0.334 GB 0.178 M allocs: 0.313 GB 1.07
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 12 k allocs: 20.5 MB 12 k allocs: 20.5 MB 1
linalg/dagger/N=256 (block 128)/cholesky 9.98 k allocs: 1.01 MB 9.96 k allocs: 1.02 MB 0.999
linalg/dagger/N=256 (block 128)/lu 30 k allocs: 2.09 MB 25.9 k allocs: 1.92 MB 1.09
linalg/dagger/N=256 (block 128)/matmul (A*A) 11.1 k allocs: 0.904 MB 11.1 k allocs: 0.905 MB 0.998
linalg/dagger/N=256 (block 128)/matvec (A*x) 7.13 k allocs: 0.277 MB 7.07 k allocs: 0.275 MB 1
linalg/dagger/N=256 (block 128)/qr 14.1 k allocs: 1.33 MB 14 k allocs: 1.33 MB 1
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 0.0757 M allocs: 4.07 MB 0.0516 M allocs: 2.99 MB 1.36
linalg/dagger/N=256 (block 128)/svd 0.215 M allocs: 30.1 MB 0.173 M allocs: 27.2 MB 1.11
linalg/dagger/N=256 (block 128)/syrk (A'*A) 12 k allocs: 1.71 MB 11.9 k allocs: 1.71 MB 1
linalg/dagger/N=256 (block 256)/cholesky 3.55 k allocs: 0.662 MB 3.52 k allocs: 0.661 MB 1
linalg/dagger/N=256 (block 256)/lu 6.44 k allocs: 1.29 MB 6.4 k allocs: 1.29 MB 1
linalg/dagger/N=256 (block 256)/matmul (A*A) 2.26 k allocs: 0.598 MB 2.25 k allocs: 0.598 MB 1
linalg/dagger/N=256 (block 256)/matvec (A*x) 2.81 k allocs: 0.123 MB 2.8 k allocs: 0.122 MB 1
linalg/dagger/N=256 (block 256)/qr 4.04 k allocs: 0.811 MB 4.01 k allocs: 0.81 MB 1
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 14.2 k allocs: 1.65 MB 14.1 k allocs: 1.65 MB 1
linalg/dagger/N=256 (block 256)/svd 18.3 k allocs: 6.86 MB 17.7 k allocs: 6.84 MB 1
linalg/dagger/N=256 (block 256)/syrk (A'*A) 3.6 k allocs: 2.16 MB 3.57 k allocs: 2.16 MB 1
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 19.9 M allocs: 0.848 GB 8.86 M allocs: 0.348 GB 2.44
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 12.3 M allocs: 0.557 GB 10.4 M allocs: 0.496 GB 1.12
sparse/dagger/N=1024 (block 64)/spmv (S*x) 1.15 M allocs: 0.0483 GB 0.995 M allocs: 0.0399 GB 1.21
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 16.4 M allocs: 0.69 GB 18.1 M allocs: 0.808 GB 0.854
sparse/dagger/N=256 (block 16)/spgemm (S*S) 12.7 M allocs: 0.532 GB 10.7 M allocs: 0.498 GB 1.07
sparse/dagger/N=256 (block 16)/spmv (S*x) 1.3 M allocs: 0.0535 GB 0.869 M allocs: 0.0343 GB 1.56
stencil/dagger/N=1024 (block 128)/assign (const) 0.0687 M allocs: 3.09 MB 0.0625 M allocs: 2.94 MB 1.05
stencil/dagger/N=1024 (block 128)/multi-expr 0.497 M allocs: 27.9 MB 0.341 M allocs: 15.5 MB 1.8
stencil/dagger/N=1024 (block 128)/update (+) 0.437 M allocs: 31 MB 0.234 M allocs: 11.1 MB 2.79
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 6.32 k allocs: 8.26 MB
stencil/dagger/N=1024 (block 512)/assign (const) 2.38 k allocs: 0.105 MB 2.99 k allocs: 0.134 MB 0.78
stencil/dagger/N=1024 (block 512)/multi-expr 28.9 k allocs: 19.2 MB 27.6 k allocs: 1.26 MB 15.3
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 29.4 k allocs: 19.3 MB
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 26.5 k allocs: 19.2 MB
stencil/dagger/N=1024 (block 512)/update (+) 26.1 k allocs: 19.1 MB 13.1 k allocs: 0.599 MB 31.9
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap) 6.3 k allocs: 0.764 MB
stencil/dagger/N=256 (block 128)/assign (const) 2.51 k allocs: 0.11 MB 2.49 k allocs: 0.112 MB 0.985
stencil/dagger/N=256 (block 128)/multi-expr 6.46 k allocs: 0.278 MB 7.79 k allocs: 0.331 MB 0.84
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 5.37 k allocs: 0.25 MB 5.2 k allocs: 0.242 MB 1.03
stencil/dagger/N=256 (block 128)/neighbors (Pad) 5.05 k allocs: 0.237 MB
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 5.15 k allocs: 0.248 MB 5.17 k allocs: 0.25 MB 0.995
stencil/dagger/N=256 (block 128)/update (+) 3.7 k allocs: 0.164 MB 4.39 k allocs: 0.19 MB 0.86
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 2.13 k allocs: 0.601 MB
stencil/dagger/N=256 (block 256)/assign (const) 1.09 k allocs: 0.056 MB 1.07 k allocs: 0.056 MB 0.999
stencil/dagger/N=256 (block 256)/multi-expr 2.59 k allocs: 0.628 MB 2.92 k allocs: 0.641 MB 0.98
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.83 k allocs: 0.598 MB 1.84 k allocs: 0.598 MB 0.999
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.82 k allocs: 0.598 MB
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.77 k allocs: 0.603 MB 1.87 k allocs: 0.608 MB 0.992
stencil/dagger/N=256 (block 256)/update (+) 1.36 k allocs: 0.567 MB 1.54 k allocs: 0.574 MB 0.987
time_to_load 0.147 k allocs: 10.8 kB 0.147 k allocs: 10.8 kB 1

⚠️ Regressions (time > 35.0% and outside the reported ±spread; allocs/memory > 25.0%)

  • array/dagger/N=1024 (block 512)/transpose (permutedims) (allocs): +77.4%
  • stencil/dagger/N=1024 (block 512)/assign (const) (time): +72.2%
  • array/dagger/N=1024 (block 512)/norm (memory): +50.5%
  • array/dagger/N=1024 (block 512)/add (X + X) (allocs): +39.1%
  • array/dagger/N=1024 (block 512)/map (sin.(X)) (allocs): +38.7%
  • array/dagger/N=1024 (block 512)/norm (allocs): +29.2%
  • stencil/dagger/N=1024 (block 512)/assign (const) (memory): +28.3%
  • array/dagger/N=1024 (block 128)/alloc (rand) (memory): +26.5%
  • array/dagger/N=1024 (block 512)/add (X + X) (memory): +25.6%
  • stencil/dagger/N=1024 (block 512)/assign (const) (allocs): +25.4%

Improvements

  • linalg/dagger/N=256 (block 128)/lu (time): -97.8%
  • stencil/dagger/N=1024 (block 512)/update (+) (memory): -96.9%
  • array/dagger/N=256 (block 128)/reduce (sum) (time): -94.7%
  • stencil/dagger/N=1024 (block 512)/multi-expr (memory): -93.5%
  • sparse/dagger/N=1024 (block 64)/cg solve (laplacian) (time): -68.7%
  • linalg/dagger/N=1024 (block 128)/lu (time): -67.8%
  • array/dagger/N=1024 (block 128)/add (X + X) (time): -67.0%
  • stencil/dagger/N=1024 (block 128)/update (+) (memory): -64.1%
  • sparse/dagger/N=1024 (block 64)/cg solve (laplacian) (memory): -59.0%
  • array/dagger/N=1024 (block 128)/add (X + X) (allocs): -55.8%
  • sparse/dagger/N=1024 (block 64)/cg solve (laplacian) (allocs): -55.5%
  • array/dagger/N=1024 (block 128)/transpose (permutedims) (time): -54.8%
  • stencil/dagger/N=1024 (block 128)/update (+) (time): -49.9%
  • stencil/dagger/N=1024 (block 512)/update (+) (allocs): -49.6%
  • linalg/dagger/N=1024 (block 512)/matmul (A*A) (time): -49.4%
  • array/dagger/N=1024 (block 512)/alloc (rand) (memory): -48.7%
  • array/dagger/N=1024 (block 512)/transpose (permutedims) (memory): -47.4%
  • stencil/dagger/N=1024 (block 128)/update (+) (allocs): -46.5%
  • array/dagger/N=1024 (block 512)/map (sin.(X)) (memory): -46.1%
  • stencil/dagger/N=1024 (block 128)/multi-expr (memory): -44.5%
  • linalg/dagger/N=1024 (block 128)/syrk (A'*A) (time): -43.9%
  • stencil/dagger/N=1024 (block 512)/update (+) (time): -43.4%
  • sparse/dagger/N=256 (block 16)/spmv (S*x) (time): -41.9%
  • stencil/dagger/N=1024 (block 128)/multi-expr (time): -40.4%
  • array/dagger/N=1024 (block 128)/add (X + X) (memory): -36.2%
  • sparse/dagger/N=256 (block 16)/spmv (S*x) (memory): -35.9%
  • array/dagger/N=1024 (block 128)/transpose (permutedims) (allocs): -35.5%
  • linalg/dagger/N=1024 (block 512)/cholesky (allocs): -34.8%
  • sparse/dagger/N=256 (block 16)/spmv (S*x) (allocs): -33.3%
  • linalg/dagger/N=1024 (block 512)/cholesky (memory): -31.9%
  • linalg/dagger/N=256 (block 128)/solve (A\b via lu) (allocs): -31.8%
  • stencil/dagger/N=1024 (block 128)/multi-expr (allocs): -31.4%
  • linalg/dagger/N=1024 (block 128)/matvec (A*x) (memory): -30.3%
  • linalg/dagger/N=1024 (block 128)/matvec (A*x) (allocs): -27.8%
  • linalg/dagger/N=1024 (block 128)/syrk (A'*A) (memory): -27.6%
  • linalg/dagger/N=256 (block 128)/solve (A\b via lu) (memory): -26.5%
Within noise (4 metric(s) past threshold but inside the ±spread; not counted)
  • linalg/dagger/N=256 (block 128)/cholesky (time): 355.4%
  • array/dagger/N=1024 (block 512)/add (X + X) (time): -42.3%
  • linalg/dagger/N=256 (block 128)/solve (A\b via lu) (time): -66.1%
  • linalg/dagger/N=1024 (block 512)/cholesky (time): -71.1%
MPI benchmarks (4 ranks)

Results unavailable (job did not produce a report).

Full results and plots (download the benchmark-results-* artifacts).

@jpsamaroo
jpsamaroo force-pushed the jps/mpi-bad-scale-stencil branch from 3e7eedb to 3c0b231 Compare August 17, 2026 22:36
@jpsamaroo
jpsamaroo force-pushed the jps/mpi-bad-scale-stencil branch 2 times, most recently from c4cf41e to 03a74d4 Compare September 4, 2026 23:47
@jpsamaroo
jpsamaroo marked this pull request as draft September 4, 2026 23:50
jpsamaroo and others added 7 commits September 5, 2026 10:46
@jpsamaroo
DArray/stencil: Avoid xfer of full chunks for boundary across differe…
1ec0bd0 
…nt spaces
@jpsamaroo
CI/benchmarks: Add Distributed and MPI benchmarks
4c7e6e1
@jpsamaroo
datadeps: Allocate cross-space slots instead of transferring them
febf021 
A Datadeps slot's contents are never assumed to be current. `generate_slot!`
says so outright -- it deliberately does not sync with the owner -- and
`compute_remainder_for_arg!` decides what to copy purely from `arg_history` /
`arg_owner`, never from the buffer. A slot in a space that does not yet appear
in the argument's history is therefore always filled by a copy-to (`FullCopy`,
or a span-exact `MultiRemainderAliasing` once part of it is current) before any
task can read it.

So for a dense, isbits payload crossing a space boundary, `move_rewrap` was
sending bytes that were guaranteed to be overwritten. It now allocates the
destination buffer and sends nothing: only the element type and dimensions
travel (broadcast like a header under MPI, closure arguments under Distributed).
The allocation is Libc-backed straight from `alloc_libc_array` rather than
allocated and then copied into Libc memory by `libc_backed`, since there is
nothing to copy. Under MPI that also fixes a latent no-op -- `libc_backed` was
being applied to a `Chunk`, which hits the identity method.

`from_space != to_space` is the load-bearing guard. When the two coincide the
leaf transfer is the identity, so the "slot" *is* the original data and
`compute_remainder_for_arg!` returns `NoAliasing()`; handing back a fresh buffer
there would silently discard the argument. Allocation also stays routed through
`aliased_object!`, so an already-present parent is still reused rather than
shadowed by a detached buffer -- which is what keeps an array and its view
sharing one destination allocation.

N.B. This depends on the preceding free-syncdeps fix. Transferring the slot was
masking a use-after-free: a recycled Libc block was immediately overwritten with
real data, so a racing read returned plausible values. Without the transfer the
same read hits uninitialized memory, which is how that bug first became visible
(NaN out of a distributed Cholesky).

Measured on 4 MPI ranks, a 1024x1024 `@stencil` sweep with `@neighbors`:
689ms/sweep at 128x128 tiles (872ms before) and 316ms at 512x512 tiles (509ms
before). `test/mpi.jl` 398/403/388/376 passing; Distributed datadeps 1272
passing.

Test contract updated accordingly: cross-space slots are no longer asserted to
hold a copy of the source, only to have the right shape, type, and parent
sharing; same-space slots must still be the real data. Two new testsets cover
the allocation itself and the partial-currency invariant it rests on -- a task
reading spans never written in its own space.
@jpsamaroo
CI: Give the 4-rank MPI CPU job more timeout headroom
268a100 
test/mpi.jl at 4 ranks passes cleanly (398/398) in ~12 minutes on an idle
workstation, matching the ~12m benchmark in c13465a's commit message --
no reproducible deadlock. CI reportedly runs ~40 minutes and occasionally
exceeds the 60-minute timeout, consistent with slower/shared runners eating
the margin rather than a hang. Bump 4-rank's budget to 90 minutes; 2-rank
stays at 60 since it isn't reported as tight.
@jpsamaroo @claude
MPIExt: Broadcast move_rewrap's header via the dispatch-decoupled relay
3daea47 
move_rewrap runs from generate_slot! inside the per-task planning loop,
and each task is handed to eager_launch! (and so starts executing,
concurrently) before the loop moves on to plan the next task's header
broadcast. That means this broadcast is not at the "sequential,
non-overlapping point" the raw bcast_yield tree assumes -- it can race
an already-dispatched task's own execute!/poolget activity, closing the
same forwarder wait-cycle documented above bcast_meta_yield.

Confirmed: a 4-rank mpiexec run of test/mpi.jl hung almost immediately
with the old bcast_yield header broadcast under load from a concurrent
MPI job on the same machine, and passed cleanly (16m44s, 398/398, no
warnings) after switching to bcast_meta_yield.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jpsamaroo @claude
MPIExt: Drop the blocking barrier that deadlocks the 4-rank MPI job
786cf6b 
The 4-rank CPU MPI job has been running out its CI timeout on Julia 1.10 and
1.11 with nothing in the log to point at. It reproduces once the run is given
CI's CPU budget rather than a workstation's: pinned to 4 CPUs (what
ubuntu-latest provides), Julia 1.10 at HEAD deadlocks outright, and SIGUSR1
backtrace dumps catch all four ranks inside `MPI_Barrier`, called from
`check_uniform`. Nothing here is version-specific -- it is a race whose odds
turn on GC and scheduling timing, which is why 1.12 slips through.

Three defects, in decreasing order of severity:

  * `check_uniform` ended with `MPI.Barrier`. That is a blocking `ccall`, so it
    parks the OS thread inside MPI: the thread runs no other Julia task --
    notably not `bcast_relay_loop`, which other ranks depend on to have a
    broadcast forwarded -- and it never reaches a GC safepoint, so any other
    thread requesting a collection stalls the process until the barrier
    returns. Rank r parks here waiting for rank s while s waits on a broadcast
    only r's relay can forward, and neither can move. The barrier bought
    nothing either: `compare_all` directly above it is already an arrival
    barrier, since it cannot return until every rank has entered it and sent.

  * `bcast_slot_wait` kept a one-shot slot keyed by tag alone. A tag does not
    identify a single broadcast -- `to_tag()` returns the planning task's thunk
    id, so every `move_rewrap` header broadcast issued while generating that
    task's slots shares it, one per wrapper level and one per argument. The
    second delivery overwrote the first before its consumer read it; that
    consumer then took the wrong value and the next one waited forever on a
    payload that had already arrived. Now a FIFO keyed by (root, tag), drained
    in the rank-uniform order MPI's non-overtaking guarantee establishes.

  * The relay busy-spun on `MPI_Improbe`, taking 107 of one thread's 127
    profile samples -- a core per rank, which a 4-vCPU runner hosting 4 ranks
    cannot spare. It now spins hot through a burst and backs off after it,
    keeping the added latency off all but the first message of the next burst.

Both waits that had no deadlock detection now have it, so a future regression
here reports itself instead of silently spending the CI budget.

Verified on Julia 1.10, 4 ranks x 2 threads pinned to 4 CPUs. HEAD deadlocks in
the Stencils testset after ~35 minutes: ranks 1-3 time out on a `compare_all`
recv from rank 0 on tag 1073741823 (`MPI.tag_ub()`) while rank 0, sitting in
the undetected barrier, reports nothing at all, and `mpiexec` never exits. With
this change the suite passes on all four ranks twice over, in 18m46s and
18m19s, with the expected 398/403/376/388 test counts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jpsamaroo @claude
datadeps: Compute free syncdeps from the buffer's own recorded aliasing
d744b14 
`gather_free_syncdeps!`'s uniform-execution fallback synced on the object
cache's `key` ainfo, guarded on that key being tracked:

    haskey(state.ainfos_overlaps, wrapped) &&
        get_write_deps!(state, space, wrapped, write_num, syncdeps)

That guard is close to never true. `ainfos_overlaps` is populated by
`populate_ainfo!` only for ainfos tracked as a direct task dependency, and the
buffers reaching this branch are exactly the ones that are not -- they merely
underlie wrapper arguments (the parent array shared by several `view`s, whose
tracked slots are the views rather than the buffer). Worse, the key ainfo
describes the original object in its *source* space: every span carries its
space, and `intersect` skips candidates sharing no space index
(memory-spaces.jl:428), so a source-space ainfo cannot overlap a
destination-space one even when it is tracked. The branch therefore emitted an
`unsafe_free!` with an *empty* syncdep set, racing every task still reading the
buffer.

What the free loop actually needs to know is which tasks touch memory *inside*
the buffer, and that is the tracked ainfos in the buffer's own space overlapping
its extent -- which requires the buffer's *destination-space* aliasing. That
cannot be recomputed here: `aliasing` is a collective under uniform (SPMD)
execution, so reaching it from the free loop costs one broadcast per freed
buffer at a sequential point.

It need not be recomputed, though. `set_stored!` already computes exactly this
ainfo, on every rank, when it allocates the buffer -- it needs it to register
the buffer in `derived` -- and then discards it. Retain it in a new
`value_ainfos` field and hand it to `gather_free_syncdeps!`, which now runs one
exact, space-filtered overlap search for both execution modes. The
uniform/non-uniform split disappears, as does the non-uniform path's own
free-time `aliasing` call.

The result is exact rather than conservative. Syncing against every tracked
ainfo in the space instead would hold every buffer until that space's work had
drained -- the opposite of what deferring frees is meant to buy.

A missing record is now an error rather than a quiet `return`: degrading to an
empty syncdep set is precisely the use-after-free this function exists to
prevent.

Also adopts, from 8469a61 on jps/datadeps-region-async (which closes the same
hole by taking the collective instead), the parts independent of that choice:
`gather_overlap_syncdeps!` as a shared helper, and
`DATADEPS_ASSERT_FREE_SYNCDEPS`, a debug-gated invariant check that re-derives
the answer by linear scan + `will_alias` rather than the interval tree, so a bug
in the fast path cannot also hide from the assertion meant to catch it.

Distributed datadeps suite (4 workers) 2068 passing, 2 broken, no failures --
including the new end-to-end regression test, which asserts every `unsafe_free!`
a shared-parent-views region emits has non-empty syncdeps and runs with the
invariant check forced on. 4-rank `test/mpi.jl` 443/470/455/466 passing, no
failures or errors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Pp6hFZNHR44pLhDrR6aXq
@jpsamaroo
jpsamaroo force-pushed the jps/mpi-bad-scale-stencil branch from 03a74d4 to d744b14 Compare September 5, 2026 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

performance stencils

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpsamaroo