Add ParametricInverter, which fits a spectral line profile to every pixel - #23
Open
roytsmart wants to merge 7 commits into
Open
Add ParametricInverter, which fits a spectral line profile to every pixel#23roytsmart wants to merge 7 commits into
ParametricInverter, which fits a spectral line profile to every pixel#23roytsmart wants to merge 7 commits into
Conversation
… pixel `MartInverter` solves for the radiance in every voxel of the scene, which is underdetermined: the tutorial configuration has 81,920 unknowns and only 32,768 measurements. `ParametricInverter` instead solves for a handful of parameters in every *spatial* pixel, which makes the problem overdetermined (12,288 unknowns for a three-parameter model) and lets the Doppler shift be measured to a small fraction of a velocity bin. The parameters of neighboring pixels are not independent, since each sensor pixel collects light from many spatial pixels, so the fit is a single optimization over every parameter of every pixel simultaneously. This is done with the Adam optimizer on a GPU. Adds three pieces: * `ctis.Regridder` assembles the sparse `weights` of a linear instrument into a single `torch` CSR matrix, so the forward model is differentiable and the adjoint comes from automatic differentiation. CSR is used instead of scatter-add because it is both faster and, unlike scatter-add, bitwise deterministic on CUDA devices. * `AbstractLinearInstrument.response` exposes the two diagonal factors of the noiseless forward model, so that an external differentiable implementation can reproduce `image()` exactly without reimplementing it. Both `IdealInstrument` and `OptikaInstrument` implement it, and a test asserts the `torch` forward model matches `image()` for each. * `GaussianModel` evaluates a Gaussian line profile, integrated analytically across each velocity bin. Bin integration is essential rather than cosmetic, since the reconstruction grid is usually comparable to or coarser than the width of the line. The free width parameter is the *nonthermal* width, with the thermal and instrumental widths held fixed, so the physically interesting quantity is the one which receives an uncertainty. `torch` is an optional dependency, installed with `pip install ctis[torch]`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 23 30 +7
Lines 861 1658 +797
==========================================
+ Hits 861 1658 +797
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The Read the Docs build failed because `plt.colorbar` cannot hold an `astropy` quantity in its norm: `quantity_support` covers the axes but not the colorbar. Strip the unit from the color values and move it into the label. The `Regridder` example built its values on the CPU while the operator defaults to a CUDA device when one is available, so it raised a device mismatch on any machine with a GPU. This did not show up on Read the Docs, which has no GPU. Create the values on `regridder.device` instead. Also fixes `ParametricInverter` rejecting images which carry an uncertainty, which is the documented way to use the instrument's own noise model: `MartInverter` cannot consume an `UncertainScalarArray`, so the uncertainty is now dropped before computing the initial guess. Adds tests for the three previously uncovered branches: weights which carry a unit, an explicitly supplied uncertainty, and an uncertainty attached to the images. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL
Mirrors the structure of the MART tutorial and uses the same synthetic scene, so the two inversion approaches can be compared side by side. Beyond reconstructing the scene, the notebook plots the fitted intensity, Doppler velocity, and nonthermal width maps, which a voxel-based inversion can only produce by post-processing the reconstructed cube, and compares the fitted velocity against the intensity-weighted centroid of the true scene. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL
The notebook used a narrower velocity grid than the MART tutorial, which clipped the wings of the test pattern: `ctis.scenes.gaussians` places components at +/-200 km/s with a width of 30 km/s. It now uses the same velocity grid, rest wavelength, scene and sensor grids, plate scale, dispersion, and dispersion angles, so the two inversions can be compared directly. The MART tutorial also adds a background equal to 1 percent of the maximum of the scene. That background is flat in wavelength, so a single Gaussian cannot represent it, and the velocity fitted in the faint pixels it dominates is not meaningful. The notebook now says so, and masks the velocity comparison to the brightest pixels using the same total-radiance threshold that `plot_moments` applies through `percentile_radiance`. The fitted velocity agrees with the median velocity of the true scene with a Pearson's r of 0.82 in those pixels. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL
The natural units of a backprojection differ between instruments: `IdealInstrument` returns an energy radiance, while `OptikaInstrument` returns a photon radiance. `ParametricInverter` starts its fit from the moments of a short MART reconstruction and converted that cube into the unit the fit works in, which raised a `UnitConversionError` for the latter, since a photon radiance cannot be converted into an energy radiance without the energy per photon. `backproject` already accepts a `unit` argument for exactly this reason, but `MartInverter` did not expose it. Add a `unit` field to `MartInverter` which is forwarded to `backproject`, and have `ParametricInverter` request the unit its model works in. The iteration is unaffected, since the multiplicative correction is a ratio of two backprojections and is therefore dimensionless. The existing tests missed this because the only tests which ran the whole inversion used `IdealInstrument`; the `OptikaInstrument` test exercised only the forward model. Adds a test which runs the whole inversion against an `OptikaInstrument`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL
The `guess` argument of `ParametricInverter` accepted only a reconstructed
scene, which was then reduced to its moments. It now also accepts a dictionary
of physical parameters, named according to the `parameters` of the model, which
is used directly.
Since `ParametricInversionResult.parameters` is a dictionary of exactly this
form, the result of one fit can warm-start another:
result = inverter(images)
result = inverter(images, guess=result.parameters)
which is useful for continuing a fit that ran out of iterations, for a raster
where each exposure starts from the previous one, or for supplying a velocity
field which is already known.
Each value may be a full map over the spatial axes of the scene, or a scalar
which is broadcast over them.
To support this, the unit of each physical parameter is now declared by the
spectral model through the new `AbstractSpectralModel.unit` method, rather than
being hardcoded by the inverter, and the inverter exposes it as
`ParametricInverter.unit_parameters`. This also removes the assumption that
every model has exactly the three parameters of `GaussianModel`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL
The merit function assumed the measurement was shot-noise limited, with an arbitrary floor of one electron squared. A real sensor has a floor set by its read noise, which for ESIS is four electrons, so the variance was underestimated by a factor of sixteen. Because only a small fraction of the detector is lit by the scene, the merit function was then almost entirely read noise: on an ESIS observation it started at 9.58 and could not be moved below 9.57 no matter how many iterations were taken. Faint pixels, which are the ones dominated by read noise, were also weighted far too heavily against bright ones. The instrument's own noise model is now evaluated once at the starting guess and held fixed. Evaluating it once means the weights do not follow the model, which would bias the fitted radiance. On the same ESIS observation the merit function now starts at 1.00, as it should for a fit which reaches the noise floor, and the Doppler velocity of the brightest pixels is recovered with a correlation of 0.74 rather than 0.52. A measurement whose variance is zero now receives zero weight rather than an infinite one, so `variance_min` is no longer needed to avoid a division by zero and defaults to zero. With the noise model corrected it becomes clear that the fit overfits: in the lit region of that observation it reproduces the measurement better than the true scene does, because a CTIS measures only a few projections and three parameters per spatial pixel leaves the fit only marginally overdetermined. Adds an optional `regularization` weight, off by default, which penalizes the mean squared first difference of the physical parameters. The penalty is applied to the physical parameters rather than the unconstrained ones, since a large change in an unconstrained parameter is a small change in a velocity near its bound. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MartInvertersolves for the radiance in every voxel of the scene, which is underdetermined: the tutorial configuration has 81,920 unknowns and only 32,768 measurements.ParametricInverterinstead solves for a handful of parameters in every spatial pixel, which makes the problem overdetermined (12,288 unknowns for a three-parameter model) and lets the Doppler shift be measured to a small fraction of a velocity bin.The parameters of neighboring pixels are not independent, since each sensor pixel collects light from many spatial pixels, so the fit is a single optimization over every parameter of every pixel simultaneously. This is done with the Adam optimizer on a GPU.
What's here
ctis.Regridderassembles the sparseweightsof a linear instrument into a singletorchCSR matrix, so the forward model is differentiable and the adjoint comes from automatic differentiation. CSR is used instead of scatter-add (index_add_) because it is both ~2.5x faster and, unlike scatter-add, bitwise deterministic on CUDA devices. Determinism matters for the conjugate-gradient solvers planned next.Note this adjoint is the exact transpose of the forward model, which
backprojectis not —backprojectapplies an extra normalization to conserve flux.AbstractLinearInstrument.responseexposes the two diagonal factors of the noiseless forward model, so an external differentiable implementation can reproduceimage()exactly without reimplementing it. BothIdealInstrumentandOptikaInstrumentimplement it.GaussianModelevaluates a Gaussian line profile, integrated analytically across each velocity bin. Bin integration is essential rather than cosmetic: the reconstruction grid is usually comparable to or coarser than the width of the line, so sampling at bin centers would make a sub-bin Doppler shift nearly unobservable.The free width parameter is the nonthermal width, with the thermal and instrumental widths held fixed. This makes the lower bound on the observed width exact rather than a penalty, and means the physically interesting quantity is the one which receives an uncertainty, instead of being recovered afterwards from a difference of comparable squares.
Verification
test_forward_matches_instrumentasserts thetorchforward model reproducesinstrument.image()for bothIdealInstrumentandOptikaInstrument(agreement to ~2e-7). This is the test that guards the whole approach.test__call__recoverybuilds a scene the model can represent exactly, forward-models it, inverts, and asserts every parameter is recovered with a correlation above 0.95. In practice the fit reaches r = 0.98 (intensity), 0.99 (velocity), 0.99 (width); with Poisson noise, 0.90 / 0.97 / 0.88.Regridderis checked againstregridding.regrid_from_weights, for the adjoint identity, for bitwise determinism, and withtorch.autograd.gradcheck.Notes for review
torchis an optional dependency (pip install ctis[torch]), imported lazily. It is added to thetestanddocextras so the new tests actually run in CI.responseis a new abstract member ofAbstractLinearInstrument, so any subclass outside this repo would need to implement it.jupyter-executeexamples in the docstrings.Next step is to swap Adam for Levenberg-Marquardt, which converges in fewer operator applications and yields per-pixel error bars from
JᵀWJ.🤖 Generated with Claude Code
https://claude.ai/code/session_01B6wnJue6RxMmKBn5sD8gDL