Repository navigation
fix(sync): check backend capability before granting it, and dispatch fail-closed - #314
Merged
Merged
Conversation
…fail-closed
optimization='metal' resolved to ('metal', 'mps') for every metric whenever
PyObjC Metal was importable, without checking whether the metric had a Metal
kernel. Only PLI, wPLI and ACCorr do. For the six einsum metrics the hand-written
if/elif chain in compute() then fell through to a bare
`return self._compute_numpy(...)`, so the caller got a numpy result while the
object reported _backend == 'metal', with no warning. The same happened via
priority=['metal', ...], a pattern the docs demonstrate.
Two independent defects combined here: the resolver never consulted metric
capability, and dispatch failed open so the mismatch stayed silent.
- Add BaseMetric.supports(), deriving capability from the presence of the
matching _compute_* method rather than a hand-kept list that can drift.
- Check capability before availability in _resolve_optimization: warn and fall
back to numpy, following the convention of the existing fallbacks.
- Skip unsupported backends in the _resolve_auto priority loop, so
priority=['metal', 'torch'] on an einsum metric lands on torch.
- Replace the nine duplicated if/elif chains with a table-driven compute() on
BaseMetric. An unknown backend now raises KeyError instead of silently
computing in numpy. Per-metric compute() docstrings are preserved.
- Make _compute_numpy abstract, enforcing that every metric ships the reference
implementation the accelerated backends are validated against.
No Metal shader for the six einsum metrics ever existed; torch/MPS is their
intended GPU path per the AUTO_PRIORITY benchmarks. optimization='auto' was
already correct and is unchanged.
Adds TestBackendCapability, which patches the availability flags instead of
gating on hardware so the contract is verified in CI.
Refs #299
…e real ones
The six einsum metrics have no Metal kernel — torch/MPS is their intended GPU
path — so test_{plv,ccorr,coh,imcoh,envcorr,powcorr}_metal_vs_numpy compared
numpy against numpy. They passed unconditionally on any macOS with PyObjC and
would have kept passing if a Metal kernel were added later and were wrong.
Their real subject, the documented fallback, is now covered for all nine metrics
by TestBackendCapability, so no coverage is lost by removing them.
test_ccorr_metal_vs_numpy also documented a shader that never existed in any
commit: "Uses Kahan summation with fastMath=OFF to preserve IEEE-754
compliance". Neither string appears anywhere in hypyp/sync/kernels/. That
docstring is what made the resolver bug look like deleted work.
Add `assert metric._backend == 'metal'` to the four tests that do exercise a
Metal kernel (PLI, wPLI, ACCorr, plus the 256-channel PLI case). Verified this
guard bites: deleting PLI._compute_metal makes the metric resolve to numpy and
the assertion fail, instead of the test silently comparing numpy to numpy.
Refs #300
…, keep the subclass contract Three adjustments on top of the capability check, from the October audit. - In the priority path, a backend skipped for lack of an implementation was reported as "No GPU backend available", which is false on a machine that has one. The fallback warning now names the backend the metric lacks. - An unknown or unimplemented backend at dispatch raised a bare KeyError (or an AttributeError). compute() now raises a ValueError naming the metric, the offending backend and the backends the metric implements. - _compute_numpy is no longer abstract. BaseMetric is public and its earlier contract was "override compute"; a third-party metric written that way could no longer be instantiated. The default now raises NotImplementedError when neither method is provided. No computed value changes. Changelog entries added for the whole branch. Refs #299, #300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.
Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.
…espect capability Follow-up to the independent review of this branch. - A subclass of the earlier contract, which overrides compute and branches on self._backend itself without any _compute_* method, was refused every accelerated backend with a false "no implementation" warning, and a super().compute() call from it raised. Such a class (own compute, no _compute_numpy) is now trusted with every known backend, as before the capability check, and super().compute() returns None as the former abstract method did. - The CPU fallback of the 'auto' path returned numba whenever numba was installed, without asking the metric. A metric implementing numpy alone then failed at compute. The fallback now goes through supports(). - supports() no longer counts a non-callable attribute, or the default _compute_numpy of the base class, as an implementation. - The priority fallback warning said "no other backend of the priority list is available", which is false when the CPU fallback is numba. It now speaks of GPU backends only. - Tests: compute() is checked to route every metric to every backend it implements, with a stub and no hardware; the four Metal tests spy on _compute_metal instead of trusting _backend; the fallback after a Metal request is checked to compute the numpy result. - Changelog and docstrings: the numerical implementations are unchanged, but a corrected priority list can now select another backend and therefore another precision (torch on MPS is float32). The dtype note in compute() was wrong for complex64 input. Refs #299, #300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…st a hand-written matrix Second follow-up to the independent review. - Whether a class dispatches through the table was inferred from the presence of its own _compute_numpy, which misread an earlier-contract subclass that happened to have a helper of that name, and a current-contract metric without numpy. It is now explicit: the class attribute _dispatch_via_table, set on the nine built-in metrics. Left unset, it is inferred from whether the class overrides compute, which was the earlier contract. - Tests: the capability matrix is written by hand and supports() is checked against it, so the routing cases no longer depend on the function under test; the Metal tests spy on the kernel functions themselves rather than on the method that calls them; earlier-contract subclasses are tested with and without a _compute_numpy helper. - Docstrings and changelog: the precision note no longer generalises beyond what was measured, the class docstring describes the current contract, and the changelog limits its claims to the built-in metrics and to the warning that was actually corrected. Refs #299, #300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Third follow-up to the independent review. - The dispatch flag was inherited, so a third-party subclass of a built-in metric that handles a backend in its own compute (for example a Metal branch added to PLV) was still refused that backend with a false warning. The flag now vouches only for the compute of the class that sets it: a descendant that overrides compute again is no longer capability-checked, while its super().compute() still gets the table dispatch of its parent. - Selection (supports) and dispatch (compute) no longer share one predicate: compute looks at the methods that exist, supports also at who owns compute. - Tests: the routing test takes its method names from a hand-written table instead of the table under test; a subclass of a built-in metric with its own backend is covered. - Docstrings and changelog state the conditions as they are, including the one behaviour that does change for a subclass of a built-in metric: delegating to super().compute() with a backend the parent lacks now raises instead of computing in NumPy. Refs #299, #300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s compute Fourth follow-up to the independent review. - A mixin carrying _dispatch_via_table made supports() raise AttributeError, because the owner of the flag was assumed to expose compute. Capability checking now compares positions in the MRO: it applies when compute is not overridden below the class that sets the flag. - A descendant that overrides compute only to delegate could be handed numba by the automatic CPU fallback although no numba method exists. Under the table dispatch the fallback now requires the method itself. - The compatibility test compared two results that could both be None; it now checks the type and compares with the numpy method. - Docstrings: a flag reset to None does not restore the inference, and the dispatch looks methods up on the instance. Refs #299, #300 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nt's compute A subclass that sets compute = PLV.compute was taken for a class doing its own dispatch, because the check compared positions in the MRO. It now compares the functions, so the same function rebound lower in the hierarchy stays capability-checked. The changelog states the scope of the unchanged 'auto' selection and the numba restriction of the CPU fallback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When the class that sets _dispatch_via_table comes after every class that defines compute, the lookup of the vouched function found nothing and raised StopIteration. It now falls back to the table dispatch of BaseMetric. The numba fallback test now builds the metric with 'auto' and computes, and the changelog states the exact condition of that restriction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e host On a CUDA node 'auto' rightly selected the CUDA kernel, so the test of the CPU fallback failed there. It now also declares MPS and CUDA unavailable. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ackend The 0.6 series changes no computed value. A priority list that reached an available backend the metric does not implement used to compute in numpy silently; skipping to the next backend of the list could change the result (torch in single precision on Apple GPUs, or numba). The selection now ends in numpy as before, with a warning and a truthful backend attribute. Moving on to the next backend is left to 0.7.0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e dispatch The 0.6 series changes no computed value. A subclass of a built-in metric that overrides compute and delegates to super().compute() with a backend that has no method now computes in numpy with a warning, as the former if/elif chains did silently, instead of raising. The CPU fallback of 'auto' trusts such a subclass with numba again. Stale docstrings are updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g in The 0.6 series changes no computed value. Before the table dispatch, a _compute_* method added by a subclass of a built-in metric for a backend its parent does not implement was never called. It stays unused, and the request computes in numpy with a warning, unless the subclass sets _dispatch_via_table itself. A priority list no longer ends in numpy for a metric that has no numpy implementation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The ownership lookup read the raw descriptor, which is not callable. Three docstrings now state that a metric without a numpy implementation does not end a priority list in numpy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
Asking for the Metal backend on a metric that has no Metal kernel used to run in NumPy silently, while the metric object reported
_backend == 'metal'. Only PLI, wPLI and ACCorr have a Metal kernel; for PLV, CCorr, Coh, ImCoh, EnvCorr and PowCorr,optimization='metal'andpriority=['metal', ...]gave a NumPy computation with no warning. Two defects combined: the resolver never asked whether the metric implements the backend, and the per-metricif/elifdispatch ended in a barereturn self._compute_numpy(...), so the mismatch stayed silent.Closes #299. Addresses the Metal half of #300.
The rule this pull request follows
The 0.6 series changes no computed value. Every request that worked on
mastertherefore still computes with the same method and gives the same result; what changes is that the silent cases now warn and that the metric object tells the truth about its backend. Making a request run a different backend than before is left to 0.7.0.What changes
The first two commits are the work of July 2026, replayed on top of the formatted
master. The following ones apply the adjustments asked for by the October audit and answer ten rounds of independent review.BaseMetric.supports(backend)tells whether a metric implements a backend, from the_compute_*methods that exist.optimization='metal'on a metric without a Metal kernel warns and reportsnumpy. Aprioritylist that reaches Metal on such a metric, on a machine where Metal is available, also ends in NumPy with a warning, as it silently did before; it does not move on to the next backend of the list. When the backend can neither run on the machine nor be computed by the metric, and no other GPU backend of the list can be used, the warning names the backend the metric lacks instead of saying "No GPU backend available".if/elifchains are replaced by one table-drivenBaseMetric.compute. A_backendset to a known backend the metric does not implement computes in NumPy with a warning, where it did so silently; a_backendthat is not the name of a backend raises aValueErrornaming the metric and the backends it implements._compute_numpyand the optional accelerated methods, and set_dispatch_via_table = Truein the class. The nine built-in metrics set it.computeis checked to route every metric to every backend it implements, with stubs and no GPU. The four tests that run a Metal kernel spy on the kernel function.Compatibility
For the nine built-in metrics, an independent review compared the method actually called, its arguments and its device against
masterover 1,130,688 simulated combinations of metric,optimizationvalue,prioritylist and available backends, and found no difference. The numerical methods and the kernels are unchanged.Third-party subclasses written the earlier way, with their own
computeand their own dispatch, are still granted the backend they request and are not capability-checked, whether they derive fromBaseMetricor from a built-in metric;_compute_numpyis not abstract, so they still instantiate. A subclass of a built-in metric that overridescomputeand delegates tosuper().compute()keeps its earlier results. A_compute_*method that a subclass adds for a backend its parent does not implement was never called before and is still not, unless the subclass sets_dispatch_via_table = Trueitself.Left for later pull requests
Moving on to the next backend of a
prioritylist when the metric does not implement one is planned with the GPU work of 0.7.0.compute_syncstill rewords everyValueErroras an unsupported metric (#306).'auto'cannot reach Metal or CuPy without torch (#304), and'numba'in aprioritylist is ignored (#305). The Numba, torch and CUDA tests do not yet assert which backend ran (#308).Measured
On macOS arm64 (M4 Max), Python 3.12, commit
9e4d4fe:tests/test_sync.py, the four Metal tests entering a real kernel;PLV(optimization='metal')gives NumPy with the warning,PLI(optimization='metal')gives Metal, PLV withpriority=['metal', 'torch']and withpriority=['metal']gives NumPy with the warning, ACCorr under'auto'gives Metal and PLV under'auto'gives torch on MPS.On a node of four H100 (CUDA, with torch, CuPy and Numba), commit
9e4d4fe: 343 passed, 9 skipped, 0 failed on the whole suite; under'auto'every metric resolves to its CUDA kernel, as onmaster.🤖 Generated with Claude Code