Skip to content

Optimizations to multi-process and multi-rank Datadeps - #739

Open
jpsamaroo wants to merge 11 commits into
masterfrom
jps/hierarchical-mp-mr
Open

Optimizations to multi-process and multi-rank Datadeps#739
jpsamaroo wants to merge 11 commits into
masterfrom
jps/hierarchical-mp-mr

Conversation

@jpsamaroo

@jpsamaroo jpsamaroo commented Aug 18, 2026

Copy link
Copy Markdown
Member

Written by Claude Opus

@github-actions

github-actions Bot commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Dagger benchmarks: dirty vs master

Multi-threaded benchmarks (4 threads)

Dagger benchmarks: dirty vs master

Summary by job

Job Regressions Improvements Within noise
linalg/dagger ✅ 0 1 1
array/dagger 0 0 4
sparse/dagger 0 0 0
stencil/dagger 0 0 3

Median time

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 30.7 ± 0.27 ms 30 ± 0.32 ms 1.02 ± 0.014
array/dagger/N=1024 (block 128)/alloc (rand) 13.8 ± 0.2 ms 14.9 ± 0.59 ms 0.923 ± 0.039
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 13.2 ± 0.93 ms 13 ± 0.083 ms 1.01 ± 0.072
array/dagger/N=1024 (block 128)/map (sin.(X)) 11.7 ± 0.53 ms 12.7 ± 1 ms 0.922 ± 0.086
array/dagger/N=1024 (block 128)/norm 14 ± 1.5 ms 13.5 ± 2 ms 1.03 ± 0.19
array/dagger/N=1024 (block 128)/reduce (sum) 27.5 ± 1.9 ms 28.1 ± 0.81 ms 0.979 ± 0.074
array/dagger/N=1024 (block 128)/transpose (permutedims) 14.4 ± 0.19 ms 16.4 ± 1.9 ms 0.876 ± 0.1
array/dagger/N=1024 (block 512)/add (X + X) 4.65 ± 1.3 ms 4.49 ± 1.2 ms 1.04 ± 0.4
array/dagger/N=1024 (block 512)/alloc (rand) 1.56 ± 0.32 ms 1.66 ± 0.23 ms 0.941 ± 0.23
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 1.97 ± 0.3 ms 2.05 ± 0.52 ms 0.96 ± 0.28
array/dagger/N=1024 (block 512)/map (sin.(X)) 5.53 ± 1.6 ms 6.08 ± 0.96 ms 0.909 ± 0.3
array/dagger/N=1024 (block 512)/norm 1.13 ± 0.03 ms 1.86 ± 1.8 ms 0.607 ± 0.57
array/dagger/N=1024 (block 512)/reduce (sum) 2.5 ± 0.67 ms 1.97 ± 0.93 ms 1.27 ± 0.69
array/dagger/N=1024 (block 512)/transpose (permutedims) 4.98 ± 0.84 ms 5.01 ± 1.7 ms 0.994 ± 0.37
array/dagger/N=256 (block 128)/add (X + X) 3.44 ± 1.9 ms 2.86 ± 1.1 ms 1.2 ± 0.79
array/dagger/N=256 (block 128)/alloc (rand) 1.36 ± 0.82 ms 1.51 ± 0.63 ms 0.902 ± 0.66
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 1.29 ± 0.67 ms 1.06 ± 0.15 ms 1.22 ± 0.65
array/dagger/N=256 (block 128)/map (sin.(X)) 1.08 ± 1 ms 2.12 ± 0.96 ms 0.512 ± 0.55
array/dagger/N=256 (block 128)/norm 1.21 ± 0.54 ms 1.07 ± 0.057 ms 1.13 ± 0.51
array/dagger/N=256 (block 128)/reduce (sum) 2.4 ± 0.14 ms 3.79 ± 1.8 ms 0.633 ± 0.31
array/dagger/N=256 (block 128)/transpose (permutedims) 1.2 ± 0.24 ms 1.67 ± 0.57 ms 0.715 ± 0.28
array/dagger/N=256 (block 256)/add (X + X) 0.94 ± 0.079 ms 0.885 ± 0.079 ms 1.06 ± 0.13
array/dagger/N=256 (block 256)/alloc (rand) 0.634 ± 0.042 ms 0.606 ± 0.013 ms 1.05 ± 0.073
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.463 ± 0.0081 ms 0.393 ± 0.025 ms 1.18 ± 0.078
array/dagger/N=256 (block 256)/map (sin.(X)) 0.976 ± 0.13 ms 1.04 ± 0.1 ms 0.942 ± 0.15
array/dagger/N=256 (block 256)/norm 0.445 ± 0.046 ms 0.432 ± 0.064 ms 1.03 ± 0.19
array/dagger/N=256 (block 256)/reduce (sum) 0.602 ± 0.0068 ms 0.608 ± 0.16 ms 0.99 ± 0.26
array/dagger/N=256 (block 256)/transpose (permutedims) 0.842 ± 0.26 ms 0.716 ± 0.14 ms 1.18 ± 0.43
linalg/dagger/N=1024 (block 128)/cholesky 0.0718 ± 0.0047 s 0.0676 ± 0.0037 s 1.06 ± 0.091
linalg/dagger/N=1024 (block 128)/lu 0.131 ± 0.0083 s 0.15 ± 0.006 s 0.879 ± 0.066
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.173 ± 0.014 s 0.186 ± 0.014 s 0.931 ± 0.11
linalg/dagger/N=1024 (block 128)/matvec (A*x) 13.8 ± 2.3 ms 14 ± 5.2 ms 0.99 ± 0.4
linalg/dagger/N=1024 (block 128)/qr 0.179 ± 0.024 s 0.189 ± 0.034 s 0.948 ± 0.21
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.202 ± 0.022 s 0.21 ± 0.02 s 0.965 ± 0.14
linalg/dagger/N=1024 (block 128)/svd 23.5 s 24.8 s 0.947
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.126 ± 0.013 s 0.118 ± 0.022 s 1.07 ± 0.23
linalg/dagger/N=1024 (block 512)/cholesky 18.7 ± 2.1 ms 19.1 ± 3.6 ms 0.98 ± 0.22
linalg/dagger/N=1024 (block 512)/lu 0.0392 ± 0.0047 s 0.0421 ± 0.0014 s 0.932 ± 0.12
linalg/dagger/N=1024 (block 512)/matmul (A*A) 0.0461 ± 0.0026 s 0.0454 ± 0.0089 s 1.02 ± 0.21
linalg/dagger/N=1024 (block 512)/matvec (A*x) 2.68 ± 2.3 ms 1.76 ± 0.69 ms 1.52 ± 1.4
linalg/dagger/N=1024 (block 512)/qr 0.108 ± 0.0052 s 0.113 ± 0.0024 s 0.957 ± 0.05
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.048 ± 0.0063 s 0.0504 ± 0.0024 s 0.952 ± 0.13
linalg/dagger/N=1024 (block 512)/svd 0.0303 h 0.0339 h 0.894
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 0.0359 ± 0.0029 s 0.0363 ± 0.00028 s 0.99 ± 0.079
linalg/dagger/N=256 (block 128)/cholesky 6.85 ± 1.5 ms 8.3 ± 3.5 ms 0.826 ± 0.39
linalg/dagger/N=256 (block 128)/lu 9.04 ± 0.19 ms 9.3 ± 1.2 ms 0.972 ± 0.13
linalg/dagger/N=256 (block 128)/matmul (A*A) 7.72 ± 1.1 ms 3.76 ± 2 ms 2.05 ± 1.1
linalg/dagger/N=256 (block 128)/matvec (A*x) 2.19 ± 0.48 ms 1.69 ± 0.47 ms 1.3 ± 0.46
linalg/dagger/N=256 (block 128)/qr 9.33 ± 2.1 ms 9.47 ± 2.3 ms 0.985 ± 0.32
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 19.5 ± 4.8 ms 19.3 ± 3.3 ms 1.01 ± 0.3
linalg/dagger/N=256 (block 128)/svd 0.513 ± 0.011 s 0.517 ± 0.012 s 0.992 ± 0.031
linalg/dagger/N=256 (block 128)/syrk (A'*A) 7.99 ± 1.9 ms 7.72 ± 3.7 ms 1.03 ± 0.55
linalg/dagger/N=256 (block 256)/cholesky 2.62 ± 2 ms 2.07 ± 0.061 ms 1.27 ± 0.97
linalg/dagger/N=256 (block 256)/lu 3.21 ± 0.15 ms 3.37 ± 0.26 ms 0.952 ± 0.085
linalg/dagger/N=256 (block 256)/matmul (A*A) 2.08 ± 0.52 ms 2.34 ± 0.095 ms 0.889 ± 0.23
linalg/dagger/N=256 (block 256)/matvec (A*x) 1.04 ± 0.039 ms 0.784 ± 0.7 ms 1.33 ± 1.2
linalg/dagger/N=256 (block 256)/qr 4.4 ± 0.21 ms 4.45 ± 0.46 ms 0.99 ± 0.11
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 6.03 ± 1.5 ms 6.39 ± 0.82 ms 0.943 ± 0.26
linalg/dagger/N=256 (block 256)/svd 0.5 ± 0.008 s 0.435 ± 0.018 s 1.15 ± 0.052
linalg/dagger/N=256 (block 256)/syrk (A'*A) 2.77 ± 0.081 ms 3.11 ± 0.94 ms 0.89 ± 0.27
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 0.485 ± 0.025 s 0.485 ± 0.013 s 1 ± 0.057
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 0.437 ± 0.0092 s 0.447 ± 0.0034 s 0.979 ± 0.022
sparse/dagger/N=1024 (block 64)/spmv (S*x) 30.8 ± 2.8 ms 27.8 ± 0.42 ms 1.11 ± 0.1
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 0.485 ± 0.0075 s 0.489 ± 0.013 s 0.991 ± 0.031
sparse/dagger/N=256 (block 16)/spgemm (S*S) 0.429 ± 0.0098 s 0.435 ± 0.0046 s 0.985 ± 0.025
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.0317 ± 0.011 s 29.3 ± 3 ms 1.08 ± 0.38
stencil/dagger/N=1024 (block 128)/alloc (neighbors Wrap) 0.037 ± 0.0023 s 0.0381 ± 0.0016 s 0.971 ± 0.072
stencil/dagger/N=1024 (block 128)/assign (const) 13.8 ± 0.71 ms 14.4 ± 0.25 ms 0.953 ± 0.052
stencil/dagger/N=1024 (block 128)/multi-expr 28.9 ± 1.9 ms 29.6 ± 0.65 ms 0.977 ± 0.068
stencil/dagger/N=1024 (block 128)/neighbors (Clamp) 23.6 ± 0.98 ms 21 ± 0.064 ms 1.13 ± 0.047
stencil/dagger/N=1024 (block 128)/neighbors (Pad) 22 ± 2.2 ms 24.1 ± 0.64 ms 0.914 ± 0.095
stencil/dagger/N=1024 (block 128)/neighbors (Reflect) 22.7 ± 0.37 ms 21 ± 0.58 ms 1.08 ± 0.034
stencil/dagger/N=1024 (block 128)/neighbors (Wrap) 23.4 ± 2.3 ms 24.2 ± 0.71 ms 0.966 ± 0.1
stencil/dagger/N=1024 (block 128)/update (+) 15.2 ± 0.16 ms 15.8 ± 0.0053 ms 0.965 ± 0.0099
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 9.23 ± 1.8 ms 8.32 ± 2.8 ms 1.11 ± 0.43
stencil/dagger/N=1024 (block 512)/assign (const) 1.35 ± 0.16 ms 1.74 ± 0.58 ms 0.774 ± 0.27
stencil/dagger/N=1024 (block 512)/multi-expr 2.93 ± 0.27 ms 3.25 ± 0.41 ms 0.902 ± 0.14
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 7.81 ± 1.8 ms 6.59 ± 0.19 ms 1.19 ± 0.27
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 6.14 ± 0.22 ms 6.44 ± 0.37 ms 0.955 ± 0.065
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 6.29 ± 0.28 ms 6.69 ± 1.4 ms 0.939 ± 0.19
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 6.43 ± 0.22 ms 6.39 ± 0.026 ms 1.01 ± 0.035
stencil/dagger/N=1024 (block 512)/update (+) 1.44 ± 0.075 ms 1.52 ± 0.074 ms 0.948 ± 0.067
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap) 3.27 ± 0.62 ms 3.13 ± 0.39 ms 1.04 ± 0.24
stencil/dagger/N=256 (block 128)/assign (const) 1.27 ± 0.26 ms 1.79 ± 2.2 ms 0.71 ± 0.89
stencil/dagger/N=256 (block 128)/multi-expr 2.98 ± 2.3 ms 2.79 ± 1.9 ms 1.07 ± 1.1
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 1.99 ± 0.026 ms 1.97 ± 0.079 ms 1.01 ± 0.043
stencil/dagger/N=256 (block 128)/neighbors (Pad) 1.81 ± 0.065 ms 2.04 ± 0.65 ms 0.885 ± 0.28
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 1.8 ± 0.39 ms 1.89 ± 0.059 ms 0.951 ± 0.21
stencil/dagger/N=256 (block 128)/neighbors (Wrap) 1.92 ± 0.12 ms 3.09 ± 2.3 ms 0.622 ± 0.47
stencil/dagger/N=256 (block 128)/update (+) 1.67 ± 1.3 ms 1.33 ± 0.19 ms 1.25 ± 1
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 1.69 ± 0.074 ms 1.72 ± 0.056 ms 0.979 ± 0.054
stencil/dagger/N=256 (block 256)/assign (const) 0.532 ± 0.028 ms 0.605 ± 0.052 ms 0.88 ± 0.088
stencil/dagger/N=256 (block 256)/multi-expr 0.997 ± 0.065 ms 1.03 ± 0.013 ms 0.971 ± 0.064
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.31 ± 0.073 ms 1.43 ± 0.034 ms 0.919 ± 0.056
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.26 ± 0.035 ms 1.4 ± 0.074 ms 0.903 ± 0.054
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.35 ± 0.33 ms 1.52 ± 0.12 ms 0.888 ± 0.23
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.29 ± 0.13 ms 1.43 ± 0.28 ms 0.909 ± 0.2
stencil/dagger/N=256 (block 256)/update (+) 0.55 ± 0.039 ms 0.635 ± 0.023 ms 0.866 ± 0.069
time_to_load 0.994 ± 0.014 s 0.988 ± 0.0096 s 1.01 ± 0.017

Allocations / memory

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 0.0775 M allocs: 11.1 MB 0.0787 M allocs: 11.1 MB 0.997
array/dagger/N=1024 (block 128)/alloc (rand) 30.8 k allocs: 9.07 MB 30.6 k allocs: 9.06 MB 1
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 28.8 k allocs: 9.02 MB 28.8 k allocs: 9.02 MB 1
array/dagger/N=1024 (block 128)/map (sin.(X)) 26 k allocs: 8.9 MB 26.2 k allocs: 8.9 MB 1
array/dagger/N=1024 (block 128)/norm 29.9 k allocs: 1.03 MB 29.9 k allocs: 1.03 MB 0.996
array/dagger/N=1024 (block 128)/reduce (sum) 0.0617 M allocs: 2.19 MB 0.0619 M allocs: 2.19 MB 1
array/dagger/N=1024 (block 128)/transpose (permutedims) 0.0359 M allocs: 9.47 MB 0.0362 M allocs: 9.49 MB 0.998
array/dagger/N=1024 (block 512)/add (X + X) 5.43 k allocs: 8.22 MB 5.51 k allocs: 8.22 MB 1
array/dagger/N=1024 (block 512)/alloc (rand) 1.93 k allocs: 8.07 MB 1.95 k allocs: 8.07 MB 1
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 1.81 k allocs: 8.06 MB 1.84 k allocs: 8.07 MB 1
array/dagger/N=1024 (block 512)/map (sin.(X)) 1.69 k allocs: 8.06 MB 1.67 k allocs: 8.06 MB 1
array/dagger/N=1024 (block 512)/norm 1.91 k allocs: 0.0662 MB 1.89 k allocs: 0.0654 MB 1.01
array/dagger/N=1024 (block 512)/reduce (sum) 3 k allocs: 0.105 MB 3.24 k allocs: 0.115 MB 0.915
array/dagger/N=1024 (block 512)/transpose (permutedims) 2.74 k allocs: 8.12 MB 2.83 k allocs: 8.12 MB 1
array/dagger/N=256 (block 128)/add (X + X) 5.5 k allocs: 0.724 MB 5.5 k allocs: 0.723 MB 1
array/dagger/N=256 (block 128)/alloc (rand) 1.96 k allocs: 0.569 MB 1.98 k allocs: 0.569 MB 0.999
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 1.77 k allocs: 0.563 MB 1.81 k allocs: 0.564 MB 0.997
array/dagger/N=256 (block 128)/map (sin.(X)) 1.69 k allocs: 0.559 MB 1.71 k allocs: 0.559 MB 0.999
array/dagger/N=256 (block 128)/norm 1.91 k allocs: 0.0663 MB 1.9 k allocs: 0.0659 MB 1.01
array/dagger/N=256 (block 128)/reduce (sum) 3.21 k allocs: 0.114 MB 3.19 k allocs: 0.113 MB 1.01
array/dagger/N=256 (block 128)/transpose (permutedims) 2.8 k allocs: 0.621 MB 2.85 k allocs: 0.623 MB 0.997
array/dagger/N=256 (block 256)/add (X + X) 1.72 k allocs: 0.577 MB 1.74 k allocs: 0.578 MB 0.998
array/dagger/N=256 (block 256)/alloc (rand) 0.552 k allocs: 0.52 MB 0.549 k allocs: 0.519 MB 1
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.492 k allocs: 0.518 MB 0.492 k allocs: 0.518 MB 1
array/dagger/N=256 (block 256)/map (sin.(X)) 0.472 k allocs: 0.517 MB 0.472 k allocs: 0.517 MB 1
array/dagger/N=256 (block 256)/norm 0.508 k allocs: 18.2 kB 0.508 k allocs: 18.2 kB 1
array/dagger/N=256 (block 256)/reduce (sum) 0.677 k allocs: 26.2 kB 0.671 k allocs: 26 kB 1.01
array/dagger/N=256 (block 256)/transpose (permutedims) 0.965 k allocs: 0.547 MB 1 k allocs: 0.549 MB 0.997
linalg/dagger/N=1024 (block 128)/cholesky 0.105 M allocs: 15.4 MB 0.102 M allocs: 15.3 MB 1.01
linalg/dagger/N=1024 (block 128)/lu 0.287 M allocs: 23 MB 0.293 M allocs: 23.2 MB 0.99
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.23 M allocs: 15.8 MB 0.224 M allocs: 15.6 MB 1.01
linalg/dagger/N=1024 (block 128)/matvec (A*x) 0.0435 M allocs: 1.75 MB 0.0445 M allocs: 1.78 MB 0.982
linalg/dagger/N=1024 (block 128)/qr 0.227 M allocs: 25 MB 0.225 M allocs: 24.9 MB 1
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.361 M allocs: 26 MB 0.366 M allocs: 26.2 MB 0.992
linalg/dagger/N=1024 (block 128)/svd 2.33 M allocs: 2.27 GB 2.4 M allocs: 2.27 GB 0.999
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.17 M allocs: 17.2 MB 0.168 M allocs: 17.1 MB 1.01
linalg/dagger/N=1024 (block 512)/cholesky 6.89 k allocs: 10.3 MB 7 k allocs: 10.3 MB 1
linalg/dagger/N=1024 (block 512)/lu 14.5 k allocs: 14.6 MB 14.5 k allocs: 14.6 MB 1
linalg/dagger/N=1024 (block 512)/matmul (A*A) 7.35 k allocs: 8.28 MB 7.15 k allocs: 8.28 MB 1
linalg/dagger/N=1024 (block 512)/matvec (A*x) 4.18 k allocs: 0.182 MB 4.23 k allocs: 0.184 MB 0.989
linalg/dagger/N=1024 (block 512)/qr 11.7 k allocs: 9.62 MB 11.9 k allocs: 9.62 MB 1
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 25 k allocs: 15.1 MB 25 k allocs: 15.1 MB 1
linalg/dagger/N=1024 (block 512)/svd 0.0466 M allocs: 0.197 GB 0.0475 M allocs: 0.197 GB 1
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 8.62 k allocs: 20.4 MB 8.53 k allocs: 20.4 MB 1
linalg/dagger/N=256 (block 128)/cholesky 6.81 k allocs: 0.909 MB 6.96 k allocs: 0.914 MB 0.995
linalg/dagger/N=256 (block 128)/lu 14.4 k allocs: 1.47 MB 14.6 k allocs: 1.48 MB 0.994
linalg/dagger/N=256 (block 128)/matmul (A*A) 7.21 k allocs: 0.779 MB 7.29 k allocs: 0.782 MB 0.996
linalg/dagger/N=256 (block 128)/matvec (A*x) 4.29 k allocs: 0.181 MB 4.35 k allocs: 0.182 MB 0.994
linalg/dagger/N=256 (block 128)/qr 11.8 k allocs: 1.26 MB 11.8 k allocs: 1.26 MB 0.998
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 24.2 k allocs: 1.9 MB 24.5 k allocs: 1.92 MB 0.988
linalg/dagger/N=256 (block 128)/svd 0.0461 M allocs: 14.6 MB 0.0469 M allocs: 14.6 MB 0.998
linalg/dagger/N=256 (block 128)/syrk (A'*A) 8.46 k allocs: 1.6 MB 8.6 k allocs: 1.6 MB 0.997
linalg/dagger/N=256 (block 256)/cholesky 2.62 k allocs: 0.622 MB 2.68 k allocs: 0.625 MB 0.996
linalg/dagger/N=256 (block 256)/lu 4.89 k allocs: 1.23 MB 4.98 k allocs: 1.23 MB 0.997
linalg/dagger/N=256 (block 256)/matmul (A*A) 1.79 k allocs: 0.58 MB 1.83 k allocs: 0.581 MB 0.998
linalg/dagger/N=256 (block 256)/matvec (A*x) 1.66 k allocs: 0.0773 MB 1.68 k allocs: 0.0778 MB 0.994
linalg/dagger/N=256 (block 256)/qr 3.48 k allocs: 0.785 MB 3.56 k allocs: 0.788 MB 0.997
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 9.99 k allocs: 1.47 MB 10.2 k allocs: 1.48 MB 0.993
linalg/dagger/N=256 (block 256)/svd 14.5 k allocs: 6.7 MB 14.9 k allocs: 6.71 MB 0.998
linalg/dagger/N=256 (block 256)/syrk (A'*A) 2.78 k allocs: 2.13 MB 2.84 k allocs: 2.13 MB 0.999
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 1.89 M allocs: 0.0745 GB 1.81 M allocs: 0.0714 GB 1.04
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 2.4 M allocs: 0.129 GB 2.44 M allocs: 0.13 GB 0.992
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.131 M allocs: 5.27 MB 0.126 M allocs: 5.05 MB 1.04
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 1.87 M allocs: 0.0736 GB 1.81 M allocs: 0.0714 GB 1.03
sparse/dagger/N=256 (block 16)/spgemm (S*S) 2.39 M allocs: 0.0913 GB 2.47 M allocs: 0.0948 GB 0.963
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.13 M allocs: 5.2 MB 0.127 M allocs: 5.09 MB 1.02
stencil/dagger/N=1024 (block 128)/alloc (neighbors Wrap) 0.0989 M allocs: 12 MB 0.1 M allocs: 12.1 MB 0.996
stencil/dagger/N=1024 (block 128)/assign (const) 0.0335 M allocs: 1.38 MB 0.0342 M allocs: 1.42 MB 0.972
stencil/dagger/N=1024 (block 128)/multi-expr 0.0758 M allocs: 3.2 MB 0.0772 M allocs: 3.26 MB 0.983
stencil/dagger/N=1024 (block 128)/neighbors (Clamp) 0.0687 M allocs: 3.18 MB 0.0698 M allocs: 3.21 MB 0.99
stencil/dagger/N=1024 (block 128)/neighbors (Pad) 0.0684 M allocs: 3.17 MB 0.0692 M allocs: 3.2 MB 0.988
stencil/dagger/N=1024 (block 128)/neighbors (Reflect) 0.069 M allocs: 3.23 MB 0.0697 M allocs: 3.24 MB 0.996
stencil/dagger/N=1024 (block 128)/neighbors (Wrap) 0.068 M allocs: 2.96 MB 0.0692 M allocs: 3 MB 0.988
stencil/dagger/N=1024 (block 128)/update (+) 0.0424 M allocs: 1.83 MB 0.0429 M allocs: 1.84 MB 0.991
stencil/dagger/N=1024 (block 512)/alloc (neighbors Wrap) 6.87 k allocs: 8.29 MB 6.97 k allocs: 8.29 MB 1
stencil/dagger/N=1024 (block 512)/assign (const) 2.67 k allocs: 0.116 MB 2.72 k allocs: 0.118 MB 0.979
stencil/dagger/N=1024 (block 512)/multi-expr 5.84 k allocs: 0.256 MB 6.04 k allocs: 0.263 MB 0.973
stencil/dagger/N=1024 (block 512)/neighbors (Clamp) 5.03 k allocs: 0.28 MB 5.13 k allocs: 0.284 MB 0.987
stencil/dagger/N=1024 (block 512)/neighbors (Pad) 4.92 k allocs: 0.277 MB 5 k allocs: 0.28 MB 0.991
stencil/dagger/N=1024 (block 512)/neighbors (Reflect) 5.05 k allocs: 0.313 MB 5.1 k allocs: 0.314 MB 0.997
stencil/dagger/N=1024 (block 512)/neighbors (Wrap) 4.89 k allocs: 0.218 MB 4.96 k allocs: 0.22 MB 0.987
stencil/dagger/N=1024 (block 512)/update (+) 3.2 k allocs: 0.141 MB 3.3 k allocs: 0.144 MB 0.978
stencil/dagger/N=256 (block 128)/alloc (neighbors Wrap) 6.94 k allocs: 0.789 MB 7.03 k allocs: 0.792 MB 0.996
stencil/dagger/N=256 (block 128)/assign (const) 2.64 k allocs: 0.115 MB 2.68 k allocs: 0.117 MB 0.988
stencil/dagger/N=256 (block 128)/multi-expr 5.82 k allocs: 0.256 MB 6.03 k allocs: 0.263 MB 0.973
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 5.08 k allocs: 0.237 MB 5.1 k allocs: 0.237 MB 1
stencil/dagger/N=256 (block 128)/neighbors (Pad) 4.94 k allocs: 0.232 MB 5 k allocs: 0.234 MB 0.993
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 5.03 k allocs: 0.243 MB 5.11 k allocs: 0.245 MB 0.993
stencil/dagger/N=256 (block 128)/neighbors (Wrap) 4.94 k allocs: 0.22 MB 5.01 k allocs: 0.222 MB 0.988
stencil/dagger/N=256 (block 128)/update (+) 3.2 k allocs: 0.141 MB 3.29 k allocs: 0.144 MB 0.979
stencil/dagger/N=256 (block 256)/alloc (neighbors Wrap) 2.09 k allocs: 0.596 MB 2.12 k allocs: 0.597 MB 0.998
stencil/dagger/N=256 (block 256)/assign (const) 0.921 k allocs: 0.0461 MB 0.963 k allocs: 0.0478 MB 0.965
stencil/dagger/N=256 (block 256)/multi-expr 2.01 k allocs: 0.0998 MB 2.06 k allocs: 0.102 MB 0.98
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.58 k allocs: 0.0848 MB 1.61 k allocs: 0.0862 MB 0.984
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.53 k allocs: 0.083 MB 1.57 k allocs: 0.0847 MB 0.979
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.58 k allocs: 0.0926 MB 1.62 k allocs: 0.094 MB 0.985
stencil/dagger/N=256 (block 256)/neighbors (Wrap) 1.53 k allocs: 0.0766 MB 1.55 k allocs: 0.0775 MB 0.988
stencil/dagger/N=256 (block 256)/update (+) 1.06 k allocs: 0.0527 MB 1.11 k allocs: 0.0545 MB 0.967
time_to_load 0.199 k allocs: 11.5 kB 0.199 k allocs: 11.5 kB 1

Plots

No time regressions beyond 25.0% or allocation regressions beyond 25.0% (timing changes inside the reported ±spread don't count) 🎉

Improvements

  • linalg/dagger/N=256 (block 128)/matmul (A*A) (time): -51.3%
Within noise (8 metric(s) past threshold but inside the ±spread; not counted)
  • array/dagger/N=256 (block 128)/map (sin.(X)) (time): 95.5%
  • array/dagger/N=1024 (block 512)/norm (time): 64.8%
  • stencil/dagger/N=256 (block 128)/neighbors (Wrap) (time): 60.8%
  • array/dagger/N=256 (block 128)/reduce (sum) (time): 58.0%
  • stencil/dagger/N=256 (block 128)/assign (const) (time): 40.8%
  • array/dagger/N=256 (block 128)/transpose (permutedims) (time): 39.8%
  • stencil/dagger/N=1024 (block 512)/assign (const) (time): 29.2%
  • linalg/dagger/N=1024 (block 512)/matvec (A*x) (time): -34.1%
Distributed benchmarks (4 processes)

Dagger benchmarks: dirty vs master

Summary by job

Job Regressions Improvements Within noise
stencil/dagger ⚠️ 5 2 1
array/dagger ⚠️ 2 4 1
linalg/dagger ✅ 0 4 4
sparse/dagger 0 0 0

Median time

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 0.0828 ± 0.0086 s 0.0882 ± 0.0054 s 0.938 ± 0.11
array/dagger/N=1024 (block 128)/alloc (rand) 28.5 ± 1.3 ms 29.6 ± 1.6 ms 0.963 ± 0.068
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 0.0354 ± 0.0055 s 0.0391 ± 0.0012 s 0.905 ± 0.14
array/dagger/N=1024 (block 128)/map (sin.(X)) 0.0354 ± 0.0015 s 0.0414 ± 0.0012 s 0.855 ± 0.045
array/dagger/N=1024 (block 128)/norm 29.2 ± 1.4 ms 0.0325 ± 0.0041 s 0.899 ± 0.12
array/dagger/N=1024 (block 128)/reduce (sum) 0.0473 ± 0.0023 s 0.053 ± 0.002 s 0.894 ± 0.055
array/dagger/N=1024 (block 128)/transpose (permutedims) 0.0317 ± 0.0062 s 30.8 ± 5.4 ms 1.03 ± 0.27
array/dagger/N=1024 (block 512)/add (X + X) 15.8 ± 1.7 ms 16.1 ± 3.7 ms 0.981 ± 0.25
array/dagger/N=1024 (block 512)/alloc (rand) 3.44 ± 0.22 ms 2.33 ± 0.083 ms 1.48 ± 0.11
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 0.0521 ± 0.098 s 1.43 ± 0.036 ms 36.5 ± 68
array/dagger/N=1024 (block 512)/map (sin.(X)) 7.85 ± 0.25 ms 10.1 ± 1.1 ms 0.777 ± 0.085
array/dagger/N=1024 (block 512)/norm 2.55 ± 0.74 ms 1.42 ± 0.38 ms 1.8 ± 0.71
array/dagger/N=1024 (block 512)/reduce (sum) 3.25 ± 0.25 ms 1.65 ± 0.049 ms 1.97 ± 0.16
array/dagger/N=1024 (block 512)/transpose (permutedims) 6.55 ± 0.47 ms 6.96 ± 0.54 ms 0.941 ± 0.099
array/dagger/N=256 (block 128)/add (X + X) 1.82 ± 0.019 ms 1.82 ± 0.025 ms 1 ± 0.017
array/dagger/N=256 (block 128)/alloc (rand) 1.32 ± 0.022 ms 1.34 ± 0.044 ms 0.986 ± 0.036
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 0.753 ± 0.033 ms 0.758 ± 0.0037 ms 0.994 ± 0.044
array/dagger/N=256 (block 128)/map (sin.(X)) 1.21 ± 0.013 ms 1.22 ± 0.013 ms 0.989 ± 0.015
array/dagger/N=256 (block 128)/norm 0.746 ± 0.04 ms 0.794 ± 0.024 ms 0.94 ± 0.058
array/dagger/N=256 (block 128)/reduce (sum) 3.36 ± 60 ms 3.4 ± 40 ms 0.987 ± 21
array/dagger/N=256 (block 128)/transpose (permutedims) 1.03 ± 0.037 ms 1.04 ± 0.054 ms 0.993 ± 0.062
array/dagger/N=256 (block 256)/add (X + X) 1.43 ± 0.075 ms 1.48 ± 0.036 ms 0.965 ± 0.056
array/dagger/N=256 (block 256)/alloc (rand) 0.773 ± 0.055 ms 0.811 ± 0.051 ms 0.952 ± 0.091
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.32 ± 0.068 ms 0.354 ± 0.023 ms 0.903 ± 0.2
array/dagger/N=256 (block 256)/map (sin.(X)) 0.807 ± 0.019 ms 0.889 ± 0.031 ms 0.908 ± 0.038
array/dagger/N=256 (block 256)/norm 0.282 ± 0.0018 ms 0.289 ± 0.0083 ms 0.976 ± 0.029
array/dagger/N=256 (block 256)/reduce (sum) 0.461 ± 0.024 ms 0.52 ± 0.019 ms 0.887 ± 0.057
array/dagger/N=256 (block 256)/transpose (permutedims) 0.728 ± 0.032 ms 0.774 ± 0.073 ms 0.941 ± 0.098
linalg/dagger/N=1024 (block 128)/cholesky 0.62 ± 0.1 s 0.411 ± 0.01 s 1.51 ± 0.25
linalg/dagger/N=1024 (block 128)/lu 0.66 ± 0.23 s 1.29 ± 1.1 s 0.513 ± 0.46
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.283 ± 0.091 s 0.531 ± 0.25 s 0.532 ± 0.3
linalg/dagger/N=1024 (block 128)/matvec (A*x) 0.0866 ± 0.0065 s 0.0829 ± 0.0033 s 1.04 ± 0.089
linalg/dagger/N=1024 (block 128)/qr 4.37 s 0.437 ± 1.9 s 9.99
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.235 ± 0.23 s 0.205 ± 0.0036 s 1.15 ± 1.1
linalg/dagger/N=1024 (block 128)/svd 49.6 s 47.9 s 1.04
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.225 ± 0.21 s 0.398 ± 0.11 s 0.563 ± 0.56
linalg/dagger/N=1024 (block 512)/cholesky 0.299 ± 0.095 s 0.443 ± 0.24 s 0.675 ± 0.42
linalg/dagger/N=1024 (block 512)/lu 0.0357 ± 0.00026 s 0.0365 ± 0.00065 s 0.978 ± 0.019
linalg/dagger/N=1024 (block 512)/matmul (A*A) 29.4 ± 0.14 ms 0.0336 ± 0.0076 s 0.875 ± 0.2
linalg/dagger/N=1024 (block 512)/matvec (A*x) 7.51 ± 0.66 ms 7.59 ± 0.72 ms 0.989 ± 0.13
linalg/dagger/N=1024 (block 512)/qr 0.103 ± 0.0012 s 0.109 ± 0.0014 s 0.948 ± 0.016
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.0575 ± 0.014 s 0.0553 ± 0.0017 s 1.04 ± 0.26
linalg/dagger/N=1024 (block 512)/svd 0.0339 h 0.0355 h 0.955
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 31.2 ± 2.2 ms 0.0324 ± 0.0038 s 0.963 ± 0.13
linalg/dagger/N=256 (block 128)/cholesky 4.83 ± 0.4 ms 5.6 ± 0.27 ms 0.862 ± 0.082
linalg/dagger/N=256 (block 128)/lu 7.31 ± 0.024 ms 6.95 ± 0.13 ms 1.05 ± 0.019
linalg/dagger/N=256 (block 128)/matmul (A*A) 4.8 ± 0.64 ms 5.1 ± 0.093 ms 0.941 ± 0.13
linalg/dagger/N=256 (block 128)/matvec (A*x) 2.4 ± 0.046 ms 2.42 ± 0.11 ms 0.993 ± 0.051
linalg/dagger/N=256 (block 128)/qr 8.1 ± 0.45 ms 8.48 ± 0.36 ms 0.955 ± 0.067
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 14.3 ± 0.65 ms 16.4 ± 0.5 ms 0.875 ± 0.048
linalg/dagger/N=256 (block 128)/svd 0.626 ± 0.041 s 0.672 ± 0.048 s 0.93 ± 0.09
linalg/dagger/N=256 (block 128)/syrk (A'*A) 5.02 ± 0.04 ms 5.85 ± 0.23 ms 0.858 ± 0.035
linalg/dagger/N=256 (block 256)/cholesky 3.03 ± 0.12 ms 3.26 ± 0.1 ms 0.931 ± 0.046
linalg/dagger/N=256 (block 256)/lu 4.19 ± 0.07 ms 4.53 ± 0.055 ms 0.926 ± 0.019
linalg/dagger/N=256 (block 256)/matmul (A*A) 2.23 ± 0.21 ms 2.64 ± 0.071 ms 0.844 ± 0.083
linalg/dagger/N=256 (block 256)/matvec (A*x) 1.32 ± 0.062 ms 1.3 ± 0.047 ms 1.01 ± 0.06
linalg/dagger/N=256 (block 256)/qr 4.5 ± 0.16 ms 5.49 ± 0.28 ms 0.819 ± 0.051
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 7.7 ± 0.28 ms 8.27 ± 0.2 ms 0.931 ± 0.04
linalg/dagger/N=256 (block 256)/svd 0.495 ± 0.014 s 0.511 ± 0.021 s 0.969 ± 0.048
linalg/dagger/N=256 (block 256)/syrk (A'*A) 3.18 ± 0.21 ms 3.6 ± 0.2 ms 0.883 ± 0.076
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 10.8 s 8.52 s 1.27
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 5.78 s 6.21 s 0.93
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.458 ± 0.0096 s 0.402 ± 0.011 s 1.14 ± 0.039
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 9.85 s 8.68 s 1.14
sparse/dagger/N=256 (block 16)/spgemm (S*S) 5.97 s 6.13 s 0.973
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.494 ± 0.0067 s 0.404 ± 0.015 s 1.22 ± 0.047
stencil/dagger/N=1024 (block 128)/assign (const) 18.3 ± 3.1 ms 0.0364 ± 0.004 s 0.504 ± 0.1
stencil/dagger/N=1024 (block 128)/multi-expr 0.243 ± 0.015 s 0.175 ± 0.0025 s 1.39 ± 0.087
stencil/dagger/N=1024 (block 128)/update (+) 0.169 ± 0.019 s
stencil/dagger/N=1024 (block 512)/assign (const) 4.33 ± 0.39 ms 3.33 ± 1.7 ms 1.3 ± 0.66
stencil/dagger/N=1024 (block 512)/multi-expr 3.04 ± 24 ms 12 ± 2.1 ms 0.254 ± 2
stencil/dagger/N=1024 (block 512)/update (+) 14 ± 0.43 ms
stencil/dagger/N=256 (block 128)/assign (const) 1.12 ± 0.081 ms 1.14 ± 0.064 ms 0.987 ± 0.091
stencil/dagger/N=256 (block 128)/multi-expr 2.32 ± 0.017 ms 2.2 ± 0.096 ms 1.06 ± 0.047
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 2.43 ± 0.016 ms 2.37 ± 0.03 ms 1.02 ± 0.015
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 2.26 ± 0.035 ms 2.41 ± 0.016 ms 0.938 ± 0.016
stencil/dagger/N=256 (block 128)/update (+) 1.25 ± 0.056 ms
stencil/dagger/N=256 (block 256)/assign (const) 0.556 ± 0.012 ms 0.649 ± 0.019 ms 0.857 ± 0.031
stencil/dagger/N=256 (block 256)/multi-expr 1.3 ± 0.18 ms 1.55 ± 0.092 ms 0.839 ± 0.13
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 1.39 ± 0.053 ms 1.48 ± 0.043 ms 0.939 ± 0.045
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 1.33 ± 0.018 ms 1.46 ± 0.054 ms 0.911 ± 0.036
stencil/dagger/N=256 (block 256)/update (+) 0.684 ± 0.037 ms
stencil/dagger/N=256 (block 128)/neighbors (Pad) 2.32 ± 0.022 ms
stencil/dagger/N=256 (block 128)/neighbors (Wrap) 2.57 ± 0.021 ms
stencil/dagger/N=256 (block 256)/neighbors (Pad) 1.6 ± 0.027 ms
time_to_load 1.02 ± 0.016 s 0.997 ± 0.0019 s 1.02 ± 0.017

Allocations / memory

master dirty master / dirty
array/dagger/N=1024 (block 128)/add (X + X) 0.167 M allocs: 18 MB 0.163 M allocs: 16.4 MB 1.1
array/dagger/N=1024 (block 128)/alloc (rand) 0.0495 M allocs: 4.92 MB 0.048 M allocs: 4.93 MB 0.997
array/dagger/N=1024 (block 128)/broadcast (X .+ 1) 0.0769 M allocs: 6.96 MB 0.0919 M allocs: 6.94 MB 1
array/dagger/N=1024 (block 128)/map (sin.(X)) 0.0739 M allocs: 6.75 MB 0.0878 M allocs: 6.86 MB 0.984
array/dagger/N=1024 (block 128)/norm 0.0535 M allocs: 2.24 MB 0.0556 M allocs: 2.37 MB 0.945
array/dagger/N=1024 (block 128)/reduce (sum) 0.085 M allocs: 3.47 MB 0.0881 M allocs: 3.88 MB 0.894
array/dagger/N=1024 (block 128)/transpose (permutedims) 0.0618 M allocs: 7.68 MB 0.0603 M allocs: 6.38 MB 1.2
array/dagger/N=1024 (block 512)/add (X + X) 12.8 k allocs: 12.5 MB 12.4 k allocs: 12.6 MB 0.999
array/dagger/N=1024 (block 512)/alloc (rand) 3.15 k allocs: 2.16 MB 2 k allocs: 8.07 MB 0.267
array/dagger/N=1024 (block 512)/broadcast (X .+ 1) 2.14 k allocs: 4.35 MB 1.84 k allocs: 8.06 MB 0.54
array/dagger/N=1024 (block 512)/map (sin.(X)) 2.88 k allocs: 2.2 MB 2.72 k allocs: 2.28 MB 0.963
array/dagger/N=1024 (block 512)/norm 2.08 k allocs: 0.0721 MB 1.92 k allocs: 0.065 MB 1.11
array/dagger/N=1024 (block 512)/reduce (sum) 3.93 k allocs: 0.15 MB 3 k allocs: 0.103 MB 1.46
array/dagger/N=1024 (block 512)/transpose (permutedims) 5.88 k allocs: 2.34 MB 5.69 k allocs: 2.32 MB 1.01
array/dagger/N=256 (block 128)/add (X + X) 6.04 k allocs: 0.745 MB 6.2 k allocs: 0.751 MB 0.992
array/dagger/N=256 (block 128)/alloc (rand) 1.99 k allocs: 0.569 MB 1.99 k allocs: 0.569 MB 1
array/dagger/N=256 (block 128)/broadcast (X .+ 1) 1.84 k allocs: 0.564 MB 1.84 k allocs: 0.564 MB 1
array/dagger/N=256 (block 128)/map (sin.(X)) 1.73 k allocs: 0.558 MB 1.73 k allocs: 0.558 MB 1
array/dagger/N=256 (block 128)/norm 1.92 k allocs: 0.065 MB 1.92 k allocs: 0.065 MB 1
array/dagger/N=256 (block 128)/reduce (sum) 4.28 k allocs: 0.168 MB 4.29 k allocs: 0.168 MB 0.997
array/dagger/N=256 (block 128)/transpose (permutedims) 2.97 k allocs: 0.63 MB 3.08 k allocs: 0.634 MB 0.993
array/dagger/N=256 (block 256)/add (X + X) 2.29 k allocs: 0.605 MB 2.44 k allocs: 0.612 MB 0.988
array/dagger/N=256 (block 256)/alloc (rand) 0.588 k allocs: 0.521 MB 0.588 k allocs: 0.521 MB 1
array/dagger/N=256 (block 256)/broadcast (X .+ 1) 0.5 k allocs: 0.518 MB 0.5 k allocs: 0.518 MB 1
array/dagger/N=256 (block 256)/map (sin.(X)) 0.469 k allocs: 0.516 MB 0.469 k allocs: 0.516 MB 1
array/dagger/N=256 (block 256)/norm 0.516 k allocs: 18.1 kB 0.516 k allocs: 18.1 kB 1
array/dagger/N=256 (block 256)/reduce (sum) 0.675 k allocs: 26 kB 0.675 k allocs: 26 kB 1
array/dagger/N=256 (block 256)/transpose (permutedims) 1.24 k allocs: 0.563 MB 1.3 k allocs: 0.565 MB 0.996
linalg/dagger/N=1024 (block 128)/cholesky 0.68 M allocs: 0.0416 GB 0.597 M allocs: 0.0375 GB 1.11
linalg/dagger/N=1024 (block 128)/lu 0.473 M allocs: 0.0327 GB 0.463 M allocs: 0.0336 GB 0.974
linalg/dagger/N=1024 (block 128)/matmul (A*A) 0.514 M allocs: 30.7 MB 0.383 M allocs: 25.2 MB 1.22
linalg/dagger/N=1024 (block 128)/matvec (A*x) 0.158 M allocs: 9.22 MB 0.151 M allocs: 9.07 MB 1.02
linalg/dagger/N=1024 (block 128)/qr 1.1 M allocs: 0.0764 GB 0.397 M allocs: 0.0354 GB 2.16
linalg/dagger/N=1024 (block 128)/solve (A\b via lu) 0.602 M allocs: 0.039 GB 0.519 M allocs: 0.0355 GB 1.1
linalg/dagger/N=1024 (block 128)/svd 16.1 M allocs: 2.54 GB 15.2 M allocs: 2.5 GB 1.01
linalg/dagger/N=1024 (block 128)/syrk (A'*A) 0.388 M allocs: 29.9 MB 0.461 M allocs: 0.0335 GB 0.871
linalg/dagger/N=1024 (block 512)/cholesky 0.0449 M allocs: 10 MB 0.0424 M allocs: 9.9 MB 1.01
linalg/dagger/N=1024 (block 512)/lu 16.8 k allocs: 14.7 MB 16.5 k allocs: 14.7 MB 1
linalg/dagger/N=1024 (block 512)/matmul (A*A) 9.02 k allocs: 8.35 MB 8.21 k allocs: 8.32 MB 1
linalg/dagger/N=1024 (block 512)/matvec (A*x) 11.5 k allocs: 0.501 MB 11.2 k allocs: 0.486 MB 1.03
linalg/dagger/N=1024 (block 512)/qr 12.8 k allocs: 9.67 MB 13.1 k allocs: 9.68 MB 0.999
linalg/dagger/N=1024 (block 512)/solve (A\b via lu) 0.0361 M allocs: 15.6 MB 0.037 M allocs: 15.6 MB 0.997
linalg/dagger/N=1024 (block 512)/svd 0.179 M allocs: 0.333 GB 0.163 M allocs: 0.312 GB 1.07
linalg/dagger/N=1024 (block 512)/syrk (A'*A) 10.2 k allocs: 20.4 MB 9.69 k allocs: 20.4 MB 1
linalg/dagger/N=256 (block 128)/cholesky 9.49 k allocs: 1.03 MB 9.31 k allocs: 1.02 MB 1.01
linalg/dagger/N=256 (block 128)/lu 17.6 k allocs: 1.6 MB 17.3 k allocs: 1.59 MB 1.01
linalg/dagger/N=256 (block 128)/matmul (A*A) 9.4 k allocs: 0.868 MB 8.6 k allocs: 0.836 MB 1.04
linalg/dagger/N=256 (block 128)/matvec (A*x) 5.47 k allocs: 0.231 MB 5.62 k allocs: 0.236 MB 0.978
linalg/dagger/N=256 (block 128)/qr 13.4 k allocs: 1.33 MB 14.3 k allocs: 1.37 MB 0.972
linalg/dagger/N=256 (block 128)/solve (A\b via lu) 0.0339 M allocs: 2.34 MB 0.0351 M allocs: 2.4 MB 0.975
linalg/dagger/N=256 (block 128)/svd 0.175 M allocs: 27.4 MB 0.164 M allocs: 26.9 MB 1.02
linalg/dagger/N=256 (block 128)/syrk (A'*A) 10.6 k allocs: 1.69 MB 10.2 k allocs: 1.67 MB 1.01
linalg/dagger/N=256 (block 256)/cholesky 3.82 k allocs: 0.682 MB 4.06 k allocs: 0.693 MB 0.984
linalg/dagger/N=256 (block 256)/lu 7.38 k allocs: 1.35 MB 7.44 k allocs: 1.35 MB 0.999
linalg/dagger/N=256 (block 256)/matmul (A*A) 2.56 k allocs: 0.617 MB 2.65 k allocs: 0.62 MB 0.994
linalg/dagger/N=256 (block 256)/matvec (A*x) 2.58 k allocs: 0.121 MB 2.69 k allocs: 0.126 MB 0.963
linalg/dagger/N=256 (block 256)/qr 5.21 k allocs: 0.87 MB 5.31 k allocs: 0.874 MB 0.996
linalg/dagger/N=256 (block 256)/solve (A\b via lu) 14.9 k allocs: 1.72 MB 15.5 k allocs: 1.74 MB 0.986
linalg/dagger/N=256 (block 256)/svd 21.9 k allocs: 7.05 MB 22.5 k allocs: 7.08 MB 0.996
linalg/dagger/N=256 (block 256)/syrk (A'*A) 4.12 k allocs: 2.2 MB 4.24 k allocs: 2.2 MB 0.998
sparse/dagger/N=1024 (block 64)/cg solve (laplacian) 15.8 M allocs: 0.714 GB 14.4 M allocs: 0.65 GB 1.1
sparse/dagger/N=1024 (block 64)/spgemm (S*S) 9.57 M allocs: 0.448 GB 11 M allocs: 0.502 GB 0.892
sparse/dagger/N=1024 (block 64)/spmv (S*x) 0.782 M allocs: 0.0332 GB 0.71 M allocs: 30.9 MB 1.1
sparse/dagger/N=256 (block 16)/cg solve (laplacian) 15.1 M allocs: 0.68 GB 14.8 M allocs: 0.666 GB 1.02
sparse/dagger/N=256 (block 16)/spgemm (S*S) 9.76 M allocs: 0.437 GB 10.8 M allocs: 0.477 GB 0.917
sparse/dagger/N=256 (block 16)/spmv (S*x) 0.835 M allocs: 0.0351 GB 0.718 M allocs: 30.8 MB 1.17
stencil/dagger/N=1024 (block 128)/assign (const) 0.044 M allocs: 1.97 MB 0.0673 M allocs: 3.12 MB 0.633
stencil/dagger/N=1024 (block 128)/multi-expr 0.336 M allocs: 15.5 MB 0.3 M allocs: 13.8 MB 1.12
stencil/dagger/N=1024 (block 128)/update (+) 0.238 M allocs: 10.9 MB
stencil/dagger/N=1024 (block 512)/assign (const) 5.94 k allocs: 0.274 MB 3.25 k allocs: 0.143 MB 1.92
stencil/dagger/N=1024 (block 512)/multi-expr 6.89 k allocs: 0.305 MB 11.6 k allocs: 0.52 MB 0.586
stencil/dagger/N=1024 (block 512)/update (+) 17.6 k allocs: 0.819 MB
stencil/dagger/N=256 (block 128)/assign (const) 2.96 k allocs: 0.131 MB 2.99 k allocs: 0.131 MB 0.998
stencil/dagger/N=256 (block 128)/multi-expr 6.81 k allocs: 0.302 MB 7.07 k allocs: 0.309 MB 0.975
stencil/dagger/N=256 (block 128)/neighbors (Clamp) 6.3 k allocs: 0.288 MB 6.48 k allocs: 0.294 MB 0.981
stencil/dagger/N=256 (block 128)/neighbors (Reflect) 6.3 k allocs: 0.297 MB 6.47 k allocs: 0.301 MB 0.984
stencil/dagger/N=256 (block 128)/update (+) 3.86 k allocs: 0.171 MB
stencil/dagger/N=256 (block 256)/assign (const) 1.2 k allocs: 0.0613 MB 1.26 k allocs: 0.0638 MB 0.96
stencil/dagger/N=256 (block 256)/multi-expr 2.8 k allocs: 0.142 MB 3.11 k allocs: 0.155 MB 0.912
stencil/dagger/N=256 (block 256)/neighbors (Clamp) 2.06 k allocs: 0.108 MB 2.14 k allocs: 0.111 MB 0.975
stencil/dagger/N=256 (block 256)/neighbors (Reflect) 2.07 k allocs: 0.116 MB 2.14 k allocs: 0.119 MB 0.977
stencil/dagger/N=256 (block 256)/update (+) 1.42 k allocs: 0.0715 MB
stencil/dagger/N=256 (block 128)/neighbors (Pad) 6.35 k allocs: 0.29 MB
stencil/dagger/N=256 (block 128)/neighbors (Wrap) 6.36 k allocs: 0.277 MB
stencil/dagger/N=256 (block 256)/neighbors (Pad) 2.18 k allocs: 0.114 MB
time_to_load 0.199 k allocs: 11.5 kB 0.199 k allocs: 11.5 kB 1

⚠️ Regressions (time > 35.0% and outside the reported ±spread; allocs/memory > 25.0%)

  • array/dagger/N=1024 (block 512)/alloc (rand) (memory): +274.3%
  • stencil/dagger/N=1024 (block 128)/assign (const) (time): +98.3%
  • array/dagger/N=1024 (block 512)/broadcast (X .+ 1) (memory): +85.2%
  • stencil/dagger/N=1024 (block 512)/multi-expr (memory): +70.7%
  • stencil/dagger/N=1024 (block 512)/multi-expr (allocs): +68.1%
  • stencil/dagger/N=1024 (block 128)/assign (const) (memory): +58.1%
  • stencil/dagger/N=1024 (block 128)/assign (const) (allocs): +52.8%

Improvements

  • linalg/dagger/N=1024 (block 128)/qr (time): -90.0%
  • linalg/dagger/N=1024 (block 128)/qr (allocs): -63.8%
  • linalg/dagger/N=1024 (block 128)/qr (memory): -53.7%
  • array/dagger/N=1024 (block 512)/reduce (sum) (time): -49.3%
  • stencil/dagger/N=1024 (block 512)/assign (const) (memory): -47.9%
  • stencil/dagger/N=1024 (block 512)/assign (const) (allocs): -45.3%
  • array/dagger/N=1024 (block 512)/norm (time): -44.4%
  • array/dagger/N=1024 (block 512)/alloc (rand) (allocs): -36.7%
  • array/dagger/N=1024 (block 512)/reduce (sum) (memory): -31.5%
  • linalg/dagger/N=1024 (block 128)/matmul (A*A) (allocs): -25.4%
Within noise (6 metric(s) past threshold but inside the ±spread; not counted)
  • stencil/dagger/N=1024 (block 512)/multi-expr (time): 294.3%
  • linalg/dagger/N=1024 (block 128)/lu (time): 94.8%
  • linalg/dagger/N=1024 (block 128)/matmul (A*A) (time): 87.8%
  • linalg/dagger/N=1024 (block 128)/syrk (A'*A) (time): 77.5%
  • linalg/dagger/N=1024 (block 512)/cholesky (time): 48.1%
  • array/dagger/N=1024 (block 512)/broadcast (X .+ 1) (time): -97.3%
MPI benchmarks (4 ranks)

Results unavailable (job did not produce a report).

Full results and plots (download the benchmark-results-* artifacts).

@jpsamaroo
jpsamaroo force-pushed the jps/hierarchical-mp-mr branch from 4a2eeeb to 58a7165 Compare August 20, 2026 23:34
@jpsamaroo
jpsamaroo changed the base branch from jps/mpi-bad-scale-stencil to master August 20, 2026 23:38
@jpsamaroo
jpsamaroo marked this pull request as draft September 4, 2026 23:50
@jpsamaroo
jpsamaroo force-pushed the jps/hierarchical-mp-mr branch 2 times, most recently from 06ec1df to f513e22 Compare September 11, 2026 22:28
jpsamaroo and others added 5 commits September 12, 2026 17:44
@jpsamaroo @cursoragent
datadeps/hierarchical: Submit off the planning thread, and measure pl…
19dcb95 
…anning

Turning a prepared spec into scheduler thunks needs nothing from the
planner, but it was running inline and accounted for ~40% of per-task
planning cost in a multi-worker region. `AsyncEnqueueQueue` hands each
batch to a submitter task instead, preserving FIFO order (the syncdeps
recorded during planning rely on it) and keeping a synchronous drain for
the two points that need a task to really exist: a value dependency's
`fetch`, and the end of the region. It is used only when there is a
spare thread to submit on, and never under uniform execution, where a
rank's submission runs collectives that must stay ordered against
planning's own.

`DATADEPS_BATCH_LIMIT` goes from 4 to 16, which is where the scheduler
round-trip stops amortizing (256 independent `InOut` tasks over 4
workers: 69 us/task unbatched, 55 at 16, 53.5 unbounded) while still
bounding how far planning runs ahead of execution. Together these take
that region from 31.2 ms to 16.9 ms.

Attributing planning cost is hard from a profile, because the expensive
parts are blocking waits inside communication rather than hot loops, so
this also adds per-phase timing behind `JULIA_DAGGER_HIER_TIMING=1`
(off by default, one `Ref` read per phase) and records what it found in
the module header. The MPI hang warning now carries a backtrace, since
which call site is waiting is the whole diagnosis for a wait cycle.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jpsamaroo @cursoragent
Datadeps: Memoize, batch and defer aliasing round-trips
3f433bb 
A chunk's aliasing info cannot be computed locally: under Distributed it
is a `remotecall_fetch` to the owner, and under MPI a broadcast from the
owner that every rank must join. Planning asks the same questions
repeatedly -- once per unique argument to build the DAG, again for every
slot, again for the write-back epilogue -- so a region spent hundreds of
round-trips re-deriving a handful of distinct answers. Under MPI each is
a global synchronization point, which is what made replicated planning
scale so poorly with rank count.

Three changes, all invisible to the user:

* `ChunkAinfoMemo`, a per-region memo keyed on argument identity, the
  dependency modifier and the acceleration. Per-region because aliasing
  info describes where a value's memory currently is: stable while one
  region plans, but a later region's chunk may reuse a freed address.
  Keys are rank-uniform, so every rank hits and misses on exactly the
  same calls and the remaining broadcasts are still collective.
* `batch_aliasing` / `batch_ainfos`, which resolve a whole uniform list
  of arguments in one exchange per owning rank rather than one broadcast
  each, and seed the memo with the results. Phase 1 now goes through
  these, so the rest of planning finds its answers already computed.
* Copies made by Datadeps recorded their own destination-side ainfo
  eagerly, costing a second rendezvous per slot on top of the transfer.
  That ainfo only matters when the copy is itself the source of a later
  move, which most regions never do, so copies are now recorded
  unresolved and resolved lazily on the first `derived` miss, as one
  batch. Safe under SPMD because the trigger is uniform.

Slot generation halved for a 4-rank stencil sweep (1.97 -> 0.96
ms/sweep), and the 2-rank MPI test suite went from 12m20s to 10m07s.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jpsamaroo @claude
Benchmark CI: Add per-job regression/improvement summary, bump defaul…
ea63021 
…t samples

The regressions/improvements lists in the CI report require opening the
report to see, and the "within noise" breakdown was buried behind a
<details> section entirely -- there was no way to tell at a glance which
suite/method jobs actually moved. Add a summary table (job, regression
count, improvement count, within-noise count) at the top of both the
Markdown report and the stdout summary, grouped by the `suite/method`
prefix that the `BENCHMARK` spec already uses to identify one job.

Also bump the default BENCHMARK_SAMPLES from 5 to 7 for less noisy CI
comparisons.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NHPKqRaGTGLt7CPKyZGuPe
@jpsamaroo @claude
MPIExt: Judge cross-rank waits by progress, not by elapsed time
766e8b0 
`mpi_deadlock_detect` timed each wait on its own and threw once one
exceeded 120s. That reads a long wait as a deadlock, but how long a rank
waits for a peer is bounded by the peer's *backlog*, not by anything
about the wait itself -- and `DATADEPS_UNIFORM_DEFER` deliberately lets
that backlog grow, submitting a region's whole task set in one burst so
planning is not interleaved with execution. A non-owner then reaches a
task's `execute!` metadata wait long before the owner starts it, and if
the owner still has to JIT the task body first, one legitimate wait runs
past any fixed timeout.

That is the MPI CPU CI failure: the `@stencil` 4D case (81 blocks x 80
`Wrap()` neighbors, 82 inputs) takes minutes to compile on first use, so
rank 1 aborted the run on a wait that was making perfectly good progress.
The suite passed with the detector disabled, and every stall warning fell
in the first repetition -- the compiling one -- with later repetitions of
the same region clean at ~3s.

Count the cross-rank operations each rank completes (finished requests,
delivered broadcast payloads) and restart a wait's clock whenever that
counter moves. The thresholds then measure a stall rather than a wait. A
real cycle still trips them, once the work that does not depend on it has
drained; verified by a rank waiting on a message that never comes, which
still errors at exactly the timeout.

Wait-loop state moves into an isbits `DeadlockTimer` so the loops keep
allocating nothing, and the two periods pick up env-var overrides, which
is what made the false positive falsifiable in the first place.

2 ranks x 2 threads, test/mpi.jl: 420/446 pass, 10m04s, no warnings
(master: 18m03s; this branch before the fix: exit 1 at 14m42s).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NHPKqRaGTGLt7CPKyZGuPe
@jpsamaroo @claude
datadeps/hierarchical: Replace HIER_TIMING with TimespanLogging categ…
f2801a7 
…ories

TimespanLogging's typed-category storage (LogCategory/steal_typed, from
"Make TimespanLogging cheap enough to leave on") is a lock-free per-thread
alternative to the custom HierPlanStats struct HIER_TIMING built for itself
because the old TimespanLogging was too slow to measure planning with. It
is now cheap enough to reuse directly.

HierPlanStats (mutable struct with Atomics, a locked samples Dict, and a
ScopedValue to thread it through parallel partition planning) is replaced
by three LogCategory declarations (LogHierPhase/LogHierSlot/LogHierAinfo)
and hier_log!, a thin wrapper around TimespanLogging's internal `_emit`
gated by `HIER_TIMING[]` directly rather than the shared `enable!` bits --
so this diagnostic stays independent of whatever else `enable_logging!`/
`disable_logging!` is doing. report_hier_stats steals this region's events
right after planning finishes and builds the same breakdown the old code
printed. This assumes one region plans at a time (true today), so it's
documented rather than solved with a region id.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GaPaEAUCJMwpQjFFPbU7Ck
@jpsamaroo
jpsamaroo force-pushed the jps/hierarchical-mp-mr branch from f513e22 to f2801a7 Compare September 13, 2026 00:56
@jpsamaroo
benchmark/stencil: Await every asynchronous fixture
097d8cc 
Stencil setup launched both input and destination DArrays but waited only for the input. The timed body therefore absorbed a nondeterministic share of destination allocation and scheduling, producing multi-fold Distributed regressions. Materialize both arrays before timing so the benchmark measures only the stencil.
@jpsamaroo
benchmark/mpi: Synchronize SPMD sampling across ranks
0549877 
BenchmarkTools applies its seconds budget independently, so rank timing skew let one rank stop while peers entered another collective sample. It also let a fast rank time its wait for a peer's prior GC as part of the next operation. Drive samples in lockstep, barrier before timing, and stop from the maximum per-sample wall time across ranks. A synthetic skew test changed sample counts from [5,2,2,2] to [2,2,2,2]; a four-rank allocation sample measured 1.93 ms instead of CI's spurious 14-35 ms. Declare JSON3 explicitly because both benchmark workers import it.
@jpsamaroo
fixup! benchmark/mpi: Synchronize SPMD sampling across ranks
4dba369
@jpsamaroo
jpsamaroo marked this pull request as ready for review September 14, 2026 17:59
@jpsamaroo
benchmark/mpi: Support BenchmarkTools 1.8 sampling
851153c 
BenchmarkTools 1.8 changed generated sample functions to write into a Ref. The old internal call failed every MPI leaf, and the worker then reported an empty successful manifest. Dispatch to either sampling API, abort on SPMD leaf failures, and reject missing or empty manifests so CI cannot pass without measurements.\n\nVerified with 4 local ranks: the reduced array matrix now completes 14/14 leaves (previously 0/14 with MethodError).
@jpsamaroo
datadeps: Keep synchronous submission batches short
c4e94a7 
A 16-task synchronous batch withholds most small Krylov regions until planning finishes, serializing planning against execution on one-thread Distributed drivers. Retain 16 for the asynchronous submitter, where it amortizes handoff overhead, but restore the synchronous batch to 4.\n\nDistributed CG (1024, block 64, 4 processes; one post-warmup run):\n  batch 16: 8.22 s, 16.85M allocs, 832.9 MB\n  batch  8: 7.91 s, 16.59M allocs, 818.6 MB\n  batch  4: 6.92 s, 15.90M allocs, 783.1 MB\n\nThe Distributed datadeps suite passes (1462 pass, 2 broken), as does the allocation suite (19/19).
@jpsamaroo
benchmark: Await vector fixtures before sampling
4db50ca 
DArray construction is asynchronous. The sparse SpMV/CG and dense matvec/solve setups waited for their matrix but not their vector, allowing vector creation and placement to leak into the timed operation. Await every fixture before sampling so scheduler changes cannot shift setup work into these measurements.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

datadeps mpi performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpsamaroo