Skip to content

fix(sync): check backend capability before granting it, and dispatch fail-closed - #314

Merged
Ramdam17 merged 14 commits into
masterfrom
fix/sync-backend-capability-check
Oct 4, 2026
Merged

Ramdam17 merged 14 commits into
masterfrom
fix/sync-backend-capability-check

Conversation

@Ramdam17

@Ramdam17 Ramdam17 commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

What this fixes

Asking for the Metal backend on a metric that has no Metal kernel used to run in NumPy silently, while the metric object reported _backend == 'metal'. Only PLI, wPLI and ACCorr have a Metal kernel; for PLV, CCorr, Coh, ImCoh, EnvCorr and PowCorr, optimization='metal' and priority=['metal', ...] gave a NumPy computation with no warning. Two defects combined: the resolver never asked whether the metric implements the backend, and the per-metric if/elif dispatch ended in a bare return self._compute_numpy(...), so the mismatch stayed silent.

Closes #299. Addresses the Metal half of #300.

The rule this pull request follows

The 0.6 series changes no computed value. Every request that worked on master therefore still computes with the same method and gives the same result; what changes is that the silent cases now warn and that the metric object tells the truth about its backend. Making a request run a different backend than before is left to 0.7.0.

What changes

The first two commits are the work of July 2026, replayed on top of the formatted master. The following ones apply the adjustments asked for by the October audit and answer ten rounds of independent review.

  • BaseMetric.supports(backend) tells whether a metric implements a backend, from the _compute_* methods that exist.
  • optimization='metal' on a metric without a Metal kernel warns and reports numpy. A priority list that reaches Metal on such a metric, on a machine where Metal is available, also ends in NumPy with a warning, as it silently did before; it does not move on to the next backend of the list. When the backend can neither run on the machine nor be computed by the metric, and no other GPU backend of the list can be used, the warning names the backend the metric lacks instead of saying "No GPU backend available".
  • The nine duplicated if/elif chains are replaced by one table-driven BaseMetric.compute. A _backend set to a known backend the metric does not implement computes in NumPy with a warning, where it did so silently; a _backend that is not the name of a backend raises a ValueError naming the metric and the backends it implements.
  • The contract for writing a metric is explicit: implement _compute_numpy and the optional accelerated methods, and set _dispatch_via_table = True in the class. The nine built-in metrics set it.
  • Tests: the six Metal tests that compared NumPy with NumPy are replaced by tests of the fallback itself (warning, backend and result). The capability of every metric is checked against a hand-written matrix, and compute is checked to route every metric to every backend it implements, with stubs and no GPU. The four tests that run a Metal kernel spy on the kernel function.

Compatibility

For the nine built-in metrics, an independent review compared the method actually called, its arguments and its device against master over 1,130,688 simulated combinations of metric, optimization value, priority list and available backends, and found no difference. The numerical methods and the kernels are unchanged.

Third-party subclasses written the earlier way, with their own compute and their own dispatch, are still granted the backend they request and are not capability-checked, whether they derive from BaseMetric or from a built-in metric; _compute_numpy is not abstract, so they still instantiate. A subclass of a built-in metric that overrides compute and delegates to super().compute() keeps its earlier results. A _compute_* method that a subclass adds for a backend its parent does not implement was never called before and is still not, unless the subclass sets _dispatch_via_table = True itself.

Left for later pull requests

Moving on to the next backend of a priority list when the metric does not implement one is planned with the GPU work of 0.7.0. compute_sync still rewords every ValueError as an unsupported metric (#306). 'auto' cannot reach Metal or CuPy without torch (#304), and 'numba' in a priority list is ignored (#305). The Numba, torch and CUDA tests do not yet assert which backend ran (#308).

Measured

On macOS arm64 (M4 Max), Python 3.12, commit 9e4d4fe:

  • standard dev environment: 310 passed, 42 skipped, 0 failed on the whole suite;
  • with torch, numba and PyObjC Metal installed: 160 passed, 9 skipped (the CuPy tests) on tests/test_sync.py, the four Metal tests entering a real kernel;
  • real resolution on that machine: PLV(optimization='metal') gives NumPy with the warning, PLI(optimization='metal') gives Metal, PLV with priority=['metal', 'torch'] and with priority=['metal'] gives NumPy with the warning, ACCorr under 'auto' gives Metal and PLV under 'auto' gives torch on MPS.

On a node of four H100 (CUDA, with torch, CuPy and Numba), commit 9e4d4fe: 343 passed, 9 skipped, 0 failed on the whole suite; under 'auto' every metric resolves to its CUDA kernel, as on master.

🤖 Generated with Claude Code

Ramdam17 and others added 3 commits October 3, 2026 22:26
…fail-closed

optimization='metal' resolved to ('metal', 'mps') for every metric whenever
PyObjC Metal was importable, without checking whether the metric had a Metal
kernel. Only PLI, wPLI and ACCorr do. For the six einsum metrics the hand-written
if/elif chain in compute() then fell through to a bare
`return self._compute_numpy(...)`, so the caller got a numpy result while the
object reported _backend == 'metal', with no warning. The same happened via
priority=['metal', ...], a pattern the docs demonstrate.

Two independent defects combined here: the resolver never consulted metric
capability, and dispatch failed open so the mismatch stayed silent.

- Add BaseMetric.supports(), deriving capability from the presence of the
  matching _compute_* method rather than a hand-kept list that can drift.
- Check capability before availability in _resolve_optimization: warn and fall
  back to numpy, following the convention of the existing fallbacks.
- Skip unsupported backends in the _resolve_auto priority loop, so
  priority=['metal', 'torch'] on an einsum metric lands on torch.
- Replace the nine duplicated if/elif chains with a table-driven compute() on
  BaseMetric. An unknown backend now raises KeyError instead of silently
  computing in numpy. Per-metric compute() docstrings are preserved.
- Make _compute_numpy abstract, enforcing that every metric ships the reference
  implementation the accelerated backends are validated against.

No Metal shader for the six einsum metrics ever existed; torch/MPS is their
intended GPU path per the AUTO_PRIORITY benchmarks. optimization='auto' was
already correct and is unchanged.

Adds TestBackendCapability, which patches the availability flags instead of
gating on hardware so the contract is verified in CI.

Refs #299
…e real ones

The six einsum metrics have no Metal kernel — torch/MPS is their intended GPU
path — so test_{plv,ccorr,coh,imcoh,envcorr,powcorr}_metal_vs_numpy compared
numpy against numpy. They passed unconditionally on any macOS with PyObjC and
would have kept passing if a Metal kernel were added later and were wrong.
Their real subject, the documented fallback, is now covered for all nine metrics
by TestBackendCapability, so no coverage is lost by removing them.

test_ccorr_metal_vs_numpy also documented a shader that never existed in any
commit: "Uses Kahan summation with fastMath=OFF to preserve IEEE-754
compliance". Neither string appears anywhere in hypyp/sync/kernels/. That
docstring is what made the resolver bug look like deleted work.

Add `assert metric._backend == 'metal'` to the four tests that do exercise a
Metal kernel (PLI, wPLI, ACCorr, plus the 256-channel PLI case). Verified this
guard bites: deleting PLI._compute_metal makes the metric resolve to numpy and
the assertion fail, instead of the test silently comparing numpy to numpy.

Refs #300
…, keep the subclass contract

Three adjustments on top of the capability check, from the October audit.

- In the priority path, a backend skipped for lack of an implementation was
  reported as "No GPU backend available", which is false on a machine that
  has one. The fallback warning now names the backend the metric lacks.
- An unknown or unimplemented backend at dispatch raised a bare KeyError (or
  an AttributeError). compute() now raises a ValueError naming the metric,
  the offending backend and the backends the metric implements.
- _compute_numpy is no longer abstract. BaseMetric is public and its earlier
  contract was "override compute"; a third-party metric written that way
  could no longer be instantiated. The default now raises
  NotImplementedError when neither method is provided.

No computed value changes. Changelog entries added for the whole branch.

Refs #299, #300

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Code review skipped — your organization has no extra usage available to pay for this review.

If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.

Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.

Ramdam17 and others added 11 commits October 3, 2026 22:49
…espect capability

Follow-up to the independent review of this branch.

- A subclass of the earlier contract, which overrides compute and branches on
  self._backend itself without any _compute_* method, was refused every
  accelerated backend with a false "no implementation" warning, and a
  super().compute() call from it raised. Such a class (own compute, no
  _compute_numpy) is now trusted with every known backend, as before the
  capability check, and super().compute() returns None as the former abstract
  method did.
- The CPU fallback of the 'auto' path returned numba whenever numba was
  installed, without asking the metric. A metric implementing numpy alone
  then failed at compute. The fallback now goes through supports().
- supports() no longer counts a non-callable attribute, or the default
  _compute_numpy of the base class, as an implementation.
- The priority fallback warning said "no other backend of the priority list
  is available", which is false when the CPU fallback is numba. It now speaks
  of GPU backends only.
- Tests: compute() is checked to route every metric to every backend it
  implements, with a stub and no hardware; the four Metal tests spy on
  _compute_metal instead of trusting _backend; the fallback after a Metal
  request is checked to compute the numpy result.
- Changelog and docstrings: the numerical implementations are unchanged, but
  a corrected priority list can now select another backend and therefore
  another precision (torch on MPS is float32). The dtype note in compute()
  was wrong for complex64 input.

Refs #299, #300

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…st a hand-written matrix

Second follow-up to the independent review.

- Whether a class dispatches through the table was inferred from the presence
  of its own _compute_numpy, which misread an earlier-contract subclass that
  happened to have a helper of that name, and a current-contract metric
  without numpy. It is now explicit: the class attribute _dispatch_via_table,
  set on the nine built-in metrics. Left unset, it is inferred from whether
  the class overrides compute, which was the earlier contract.
- Tests: the capability matrix is written by hand and supports() is checked
  against it, so the routing cases no longer depend on the function under
  test; the Metal tests spy on the kernel functions themselves rather than on
  the method that calls them; earlier-contract subclasses are tested with and
  without a _compute_numpy helper.
- Docstrings and changelog: the precision note no longer generalises beyond
  what was measured, the class docstring describes the current contract, and
  the changelog limits its claims to the built-in metrics and to the warning
  that was actually corrected.

Refs #299, #300

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Third follow-up to the independent review.

- The dispatch flag was inherited, so a third-party subclass of a built-in
  metric that handles a backend in its own compute (for example a Metal
  branch added to PLV) was still refused that backend with a false warning.
  The flag now vouches only for the compute of the class that sets it: a
  descendant that overrides compute again is no longer capability-checked,
  while its super().compute() still gets the table dispatch of its parent.
- Selection (supports) and dispatch (compute) no longer share one predicate:
  compute looks at the methods that exist, supports also at who owns compute.
- Tests: the routing test takes its method names from a hand-written table
  instead of the table under test; a subclass of a built-in metric with its
  own backend is covered.
- Docstrings and changelog state the conditions as they are, including the
  one behaviour that does change for a subclass of a built-in metric:
  delegating to super().compute() with a backend the parent lacks now raises
  instead of computing in NumPy.

Refs #299, #300

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s compute

Fourth follow-up to the independent review.

- A mixin carrying _dispatch_via_table made supports() raise AttributeError,
  because the owner of the flag was assumed to expose compute. Capability
  checking now compares positions in the MRO: it applies when compute is not
  overridden below the class that sets the flag.
- A descendant that overrides compute only to delegate could be handed numba
  by the automatic CPU fallback although no numba method exists. Under the
  table dispatch the fallback now requires the method itself.
- The compatibility test compared two results that could both be None; it
  now checks the type and compares with the numpy method.
- Docstrings: a flag reset to None does not restore the inference, and the
  dispatch looks methods up on the instance.

Refs #299, #300

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nt's compute

A subclass that sets compute = PLV.compute was taken for a class doing its
own dispatch, because the check compared positions in the MRO. It now
compares the functions, so the same function rebound lower in the hierarchy
stays capability-checked. The changelog states the scope of the unchanged
'auto' selection and the numba restriction of the CPU fallback.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When the class that sets _dispatch_via_table comes after every class that
defines compute, the lookup of the vouched function found nothing and raised
StopIteration. It now falls back to the table dispatch of BaseMetric. The
numba fallback test now builds the metric with 'auto' and computes, and the
changelog states the exact condition of that restriction.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e host

On a CUDA node 'auto' rightly selected the CUDA kernel, so the test of the
CPU fallback failed there. It now also declares MPS and CUDA unavailable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ackend

The 0.6 series changes no computed value. A priority list that reached an
available backend the metric does not implement used to compute in numpy
silently; skipping to the next backend of the list could change the result
(torch in single precision on Apple GPUs, or numba). The selection now ends
in numpy as before, with a warning and a truthful backend attribute. Moving
on to the next backend is left to 0.7.0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e dispatch

The 0.6 series changes no computed value. A subclass of a built-in metric
that overrides compute and delegates to super().compute() with a backend
that has no method now computes in numpy with a warning, as the former
if/elif chains did silently, instead of raising. The CPU fallback of 'auto'
trusts such a subclass with numba again. Stale docstrings are updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g in

The 0.6 series changes no computed value. Before the table dispatch, a
_compute_* method added by a subclass of a built-in metric for a backend its
parent does not implement was never called. It stays unused, and the request
computes in numpy with a warning, unless the subclass sets
_dispatch_via_table itself. A priority list no longer ends in numpy for a
metric that has no numpy implementation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The ownership lookup read the raw descriptor, which is not callable. Three
docstrings now state that a metric without a numpy implementation does not
end a priority list in numpy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Ramdam17
Ramdam17 merged commit dae92d9 into master Oct 4, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

optimization='metal' silently runs NumPy for the 6 metrics that have no Metal kernel, with no warning

1 participant