Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 40 additions & 31 deletions docs/source/analyses/pipeline_evaluation_metrics/openuniv_eval.rst
Original file line number Diff line number Diff line change
@@ -1,30 +1,29 @@
Evaluation of OpenUniverse Simulated Data Pipeline Tests
##################################################################

Overview
************************************

The test evalutions described below are organized by processing date. See :ref:`testing` for specifications
of each test run.
Evaluations are organized by processing date. See :ref:`testing` for each
test run's specifications.

9/27/2025
************************************

This test run consists of 6,875 science images across 7 filters. We evalutate the performance of the pipeline
for two difference image-subtraction methods: ZOGY and SFFT (with PSF cross-convolution).

This run evaluates pipeline performance on 6,875 science images across
7 filters using two difference image-subtraction methods: ZOGY and SFFT
(with PSF cross-convolution).

Source Injections
====================================

Fake-source injections were performed with the scheme described in :ref:`fake_source_injection` with
100 sources per image and ``size_factor`` :math:`= 1.5`.
Fake sources were injected as described in :ref:`fake_source_injection`,
with 100 sources per image and ``size_factor`` :math:`= 1.5`.

Source Detection and Filtering
====================================
We evalute the performance of source detection with SExtractor, with detection performed on the matched-filtered difference images
where available (Scorr for ZOGY and cross-convolved difference images for SFFT).
SExtractor photometry is performed on the difference images. The primary SExtractor parameters used for source detection were:

SExtractor detects sources on matched-filtered difference images where
available (Scorr for ZOGY and cross-convolved difference images for SFFT)
and measures photometry on the difference images. The primary detection
parameters were:

* ``DETECT_MINAREA`` :math:`=5`
* ``DETECT_THRESH`` :math:`=2.5`
Expand All @@ -34,7 +33,8 @@ SExtractor photometry is performed on the difference images. The primary SExtrac
* ``DEBLEND_NTHRESH`` :math:`=32`
* ``PHOT_APERTURES`` :math:`=2.0,3.0,4.0,6.0,10.0,14.0` (aperture diameter in pixels)

Raw catalogs were filtered as described in :ref:`filtering` with the following thresholds:
Raw catalogs were filtered as described in :ref:`filtering`, using these
thresholds:

1. Catalog level
a. ``diffimedgetol`` :math:`=5`
Expand All @@ -44,12 +44,13 @@ Raw catalogs were filtered as described in :ref:`filtering` with the following t
2. Pixel-value level
a. ``nnegthres`` :math:`=18`
b. ``nbadthres`` :math:`=12`
c. ``sumratthres`` :math:`=0.25`
3. PSF photometry
c. ``sumratthres`` :math:`=0.25`
3. PSF photometry
a. ``rchipsfthres`` :math:`=10`
b. ``magdiffthres`` :math:`=0.3`

Detections are cross-matched to the injected source catalogs using a matching radius of ``injmatchrad`` :math:`=0.5` WFI pixels.
Detections are cross-matched to injected-source catalogs within
``injmatchrad`` :math:`=0.5` WFI pixels.

Filtering results:

Expand All @@ -68,14 +69,28 @@ SFFT H158 1142 2935660 74690 148362 (5.1%) 66434 (8

Evaluation of Figures of Merit
====================================
We evaluate the Figure of Merit (FOM; as defined in :ref:`figure_of_merit`) for both difference imaging methods for H158. We define two
groups sources for this analysis based on their separations from the core of the nearest truth-catalog galaxy brighter than
``galmatchthres`` :math:`= 25` mag, off-nuclear sources at :math:`\gt 1.5 \times` ``injmatchrad`` and nuclear sources
at :math:`\leq 1.5 \times` ``injmatchrad``. This grouping allows us to account for possible contamination of the recovered True Positive (TP)
sample by spurious detections arising from imperfect galaxy subtraction in the nuclear regions. We apply overall relative weights of 1.0 and 0.5
for the off-nuclear and nuclear source groups, respectively, in the FOM calculations. The relative weights for each term in the FOM
are :math:`w_{\mathrm{th}} = 1.0`, :math:`w_{80} = 0.8`, :math:`w_{20} = 0.6`, :math:`w_{5\sigma} = 0.3`, and :math:`w_{\mathrm{ph}10} = 0.2`.
We adopt ``fpratetol`` :math:`=10` as the maximum acceptable false-positive rate per image.

The Figure of Merit (FOM; defined in :ref:`figure_of_merit`) is evaluated
for both methods in H158. Sources are grouped by their separation from
the core of the nearest truth-catalog galaxy brighter than
``galmatchthres`` :math:`= 25` mag:

* Off-nuclear: :math:`\gt 1.5 \times` ``injmatchrad``.
* Nuclear: :math:`\leq 1.5 \times` ``injmatchrad``.

This grouping accounts for possible contamination of the recovered True
Positive (TP) sample by spurious detections from imperfect galaxy
subtraction in nuclear regions. The FOM calculations assign overall
relative weights of 1.0 to off-nuclear sources and 0.5 to nuclear sources.
The relative weights of the FOM terms are :math:`w_{\mathrm{th}} = 1.0`,
:math:`w_{80} = 0.8`, :math:`w_{20} = 0.6`, :math:`w_{5\sigma} = 0.3`, and
:math:`w_{\mathrm{ph}10} = 0.2`. The maximum acceptable false-positive
rate per image is ``fpratetol`` :math:`=10`.

.. note::
This preliminary analysis illustrates how to evaluate RAPID pipeline
performance using the FOM. Thresholds, weights, and grouping criteria
for uses such as algorithm down-selection remain under development.

====== ====== ======== ======================= ============== ============== =================== ========================= ==========
Method Filter Group :math:`m_{\mathrm{th}}` :math:`m_{80}` :math:`m_{20}` :math:`m_{5\sigma}` :math:`m_{\mathrm{ph}10}` FOM
Expand All @@ -88,14 +103,8 @@ ZOGY H158 Overall 24.60 23.95 25.
SFFT H158 Overall 25.59 24.66 25.59 26.78 24.49 **25.38**
====== ====== ======== ======================= ============== ============== =================== ========================= ==========

.. note::
This analysis is preliminary and meant to illustrate the procedure for evaluation the performance of the RAPID pipeline
based on the FOM. The chosen thresholds, weights, and grouping criteria that will be used for, e.g., algorithm down-selection,
are under development.

.. toctree::
:maxdepth: 1

FOM_20250927_ZOGY_H158.rst
FOM_20250927_SFFT_H158.rst

Loading
Loading