diff --git a/docs/source/dev/socsim_tests.rst b/docs/source/dev/socsim_tests.rst index fdf25c81..101590c0 100644 --- a/docs/source/dev/socsim_tests.rst +++ b/docs/source/dev/socsim_tests.rst @@ -4,70 +4,67 @@ Testing with SOC Simulated Data Overview ************************************ -The SOC has generated simulated L2 files for the GBTDS survey (also referred to here as "socsims"). -These are located here:: +The Roman Space Telescope Science Operations Center (SOC) generated simulated +Level 2 (L2) Calibrated Science Data Files for the GBTDS survey. These +"socsims" cover the Wide Field Instrument (WFI) and are stored in the +Advanced Scientific Data Format (ASDF):: s3://stpubdata/roman/nexus/soc_simulations/r00340/l2/ -The term refers to Level 2 (L2) Calibrated Science Data Files generated by the -Roman Space Telescope Science Operations Center (SOC) simulation tools for the -Wide Field Instrument (WFI), stored in the Advanced Scientific Data Format (ASDF). -These files are typically produced by simulation tools like Roman-I-Sim or -STIPS to mimic real-world data that the telescope will produce, allowing researchers -to prepare for data analysis before launch. +Simulation tools such as Roman-I-Sim and STIPS mimic telescope data so +researchers can prepare for analysis before launch. The +wfi_soc-simulation_l2_cal.asdf files support workflow testing, WFI +field-of-view visualization, and astrometric and photometric precision checks. -Key aspects of wfi_soc-simulation_l2_cal.asdf files: +* Content and calibration: A simulation of the romancal pipeline processes + Level 1 (L1) raw data (uncalibrated ramps), removing instrumental effects + to produce science-ready calibrated rate images in Digital Numbers per + second (DN/s). +* Format: ASDF, the standard Roman data format, stores image arrays and metadata. +* WFI coverage: 0.281 degrees-squared across 18 detectors (SCAs). +* Astrometry: The files are designed to align with the Gaia reference frame. +* Generators: Roman I-Sim is a GalSim-based simulator for high-fidelity, + pipeline-compatible data; STIPS (Space Telescope Imaging Product Simulator) + creates synthetic astronomical scenes. -* Content: They contain calibrated rate images in units of Digital Numbers per second (DN/s), - where raw detector signals have been processed to remove instrumental effects. +RAPID input preparation +==================================== -* Format: ASDF is the standard file format for Roman data, storing both image arrays and metadata. - -* WFI Specifics: WFI data covers a 0.281 degrees-squared field of view using 18 detectors (SCAs). - -* Calibration Level (L2): L2 files are derived from Level 1 (L1) raw data (uncalibrated ramps) - and have been processed through a simulation of the romancal pipeline to become science-ready rate images. - -* Astrometry: These simulation files are designed to align with the Gaia astrometric reference frame. - -* Simulation Tools: These files are generated by: - - - Roman I-Sim: A GalSim-based simulator for high-fidelity, pipeline-compatible data. - - - STIPS (Space Telescope Imaging Product Simulator): Used for creating synthetic astronomical scenes. - -These files are used to test data analysis workflows, visualize the WFI field of view, and -verify astrometric and photometric precision. - -The SOC sims have filenames like ``r0034001001001001001_0001_wfi01_f062_cal.asdf``. -Each file is for a given exposure and SCA. -There are 88,038 of these files available, covering 4,891 exposures and all bandpass filters. -Assuming the exposure time is 66.4 seconds, which is the predominant exposure time in the -GBTDS observation-planning files, this dataset represents approximately 3.75 days of -cumulative exposure time, which is approximately 34% of the entire GBTDS survey. - - -For the RAPID pipeline, fake variable sources with fixed sky positions have been added to -the ASDF files and stored here:: +Fake variable sources with fixed sky positions were added to the ASDF files +for the RAPID pipeline and stored here:: s3://socsims-fakesrc-asdf-20260709/ -It was discovered that the gWCS in the SOC sims is incorrect (there were no GAIA stars, -so the astrometry step failed). We corrected this using the following Python code:: +The SOC sims' gWCS was incorrect because there were no GAIA stars and the +astrometry step failed. It was corrected with:: import roman_datamodels as rdm from romancal.assign_wcs import AssignWcsStep original_dm = rdm.open(asdf_path) dm = AssignWcsStep.call(original_dm) -The ASDF files have been converted into FITS files and stored here:: +The ASDF files were converted to FITS and stored here:: s3://socsims-fakesrc-fits-20260709-lite/ -.. note:: - Metadata about the SOC sims are stored in a dedicated RAPID-operations PostgreSQL database. +The FITS WCS uses TAN-SIP projection with fifth-order SIP distortion. Tests +show very good agreement with the original ASDF gWCS: two examples (SCAs 2 +and 9) were examined for absolute WCS error between FITS and ASDF. Across +all 18 SCAs, the worst deviations never exceed ~1e-6 of a pixel. + +Dataset coverage +==================================== + +Each file covers one exposure and SCA, with filenames such as +``r0034001001001001001_0001_wfi01_f062_cal.asdf``. The 88,038 available files +cover 4,891 exposures and all bandpass filters. Assuming 66.4 seconds per +exposure, the predominant exposure time in the GBTDS observation-planning +files, the dataset represents approximately 3.75 days of cumulative +exposure time, approximately 34% of the entire GBTDS survey. + +SOC-sim metadata are stored in a dedicated RAPID-operations PostgreSQL +database. The precise cumulative exposure time in days is: -Indeed, the precise cumulative exposure time in days is: .. code-block:: @@ -77,7 +74,7 @@ Indeed, the precise cumulative exposure time in days is: 3.7593229166666666 (1 row) -The minimum and maximum observation times of the SOC sims cover about 8 days: +The observation times span about 8 days: .. code-block:: @@ -96,8 +93,8 @@ The images overlap a total of 109 sky tiles (a.k.a. fields): ------- 109 -Here is a breakdown of the number of SOC-sim images, all SCAs, by bandpass filter (where the filter-name convention -of the Open Univers sims is retained): +Image counts across all SCAs by bandpass filter, retaining the Open Univers +sims' filter-name convention: .. code-block:: @@ -114,26 +111,23 @@ of the Open Univers sims is retained): 8 | W146 | 76250 (8 rows) -It is seen that most of the exposures by a large margin are taken with the W146 bandpass filter (``fid=8``). - -The WCS in the FITS files is represented by the TAN-SIP projection with fifth-order SIP distortion. -Tests show this represents the WCS very well. Two examples were examined to compare the -absolute error in the WCS between fits and the original ASDF gWCS (SCAs 2 and 9). -Across all 18 SCAs, the worst deviations are never larger than ~1e-6 of a pixel. +W146 (``fid=8``) accounts for most exposures by a large margin. 7/6/2026 ************************************ -The first socsims test is limited to a subset of the first day of observations and the -W146 bandpass filter(``fid=8``). This test covers 6,917 science images. -Fake variable sources with fixed sky positions were -injected into the ASDF files prior to conversion to FITS, about 200 fake sources per science image. -Thus, lightcurves can be generated from extractions -of these fake sources over time. The science images that are used to build the reference images also -have fake variable sources. +The first socsims test covers 6,917 science images from part of the first +observation day, limited to W146 (``fid=8``). About 200 fake variable sources +with fixed sky positions were injected per science image before ASDF-to-FITS +conversion, allowing lightcurves to be generated from extractions over time. +The science images used to build reference images also contain fake variable +sources. + +Setup +==================================== -Here are details about how the test was executed via the Virtual Pipeline Operator (VPO): +Virtual Pipeline Operator (VPO) invocation: .. code-block:: @@ -146,9 +140,9 @@ Here are details about how the test was executed via the Virtual Pipeline Operat python3.11 /code/pipeline/virtualPipelineOperator.py 20260706 >& virtualPipelineOperator_20260706.out & -The ``STARTDATETIME`` and ``ENDDATETIME`` date/times exclude the first 20 or so images per field, -which are reserved for reference-image generation. The input configuration file has specified the -following parameters for reference images:: +``STARTDATETIME`` and ``ENDDATETIME`` exclude the first 20 or so images per +field, reserving them for reference-image generation. Reference-image +parameters in the input configuration file:: [REF_IMAGE] # Pipeline number of reference-image pipeline. @@ -166,7 +160,10 @@ following parameters for reference images:: max_n_images_to_coadd = 25 -Here is a summary of the pipeline exit codes after the test: +Results +==================================== + +Pipeline exit codes: .. code-block:: @@ -178,8 +175,8 @@ Here is a summary of the pipeline exit codes after the test: 17 | 0 | 6917 (2 rows) -Reference images were generated for 109 unique fields, again, only for the W146 bandpass filter(``fid=8``). -Because the socsims have sub-pixel dithers, the ``cov5percent`` coverage metric is only ~30-50 percent. +Reference images were generated for 109 unique fields in W146 (``fid=8``). +Sub-pixel dithers limit the ``cov5percent`` coverage metric to ~30-50 percent. .. code-block:: @@ -299,12 +296,14 @@ Because the socsims have sub-pixel dithers, the ``cov5percent`` coverage metric (109 rows) -As shown in the table below for one of the pipeline instances that generated a -reference image (jid = 114725), the computation of PSF-fit PhotUtils catalogs for the -reference image and the difference images are the dominant factors affecting pipeline -performance. Setting up and executing awaicgen for reference-image generation took -about 5 minutes (depends on the number of input images; NFRAMES=21 for this case). -Executing SFFT was relatively quick. +Performance +==================================== + +For pipeline instance jid = 114725, which generated a reference image, +PSF-fit PhotUtils catalog generation for the reference and difference images +dominated execution time. Setting up and executing awaicgen took about +5 minutes, depending on the input-image count (NFRAMES=21 here). SFFT was +relatively quick. ================================================================== ===================== @@ -349,71 +348,68 @@ Uploading products at pipeline end to S3 product bucket 0.08 Total time to run this instance of the science pipeline 2996.133 ================================================================== ===================== -The above pipeline instance took about 50 minutes to execute. Pipeline instances -that made use of already-generated reference images took about 20 minutes each to run. - -The entire set of 6,917 science images took 3.7 hours to run the science pipelines that -generated the aforementioned 109 reference images and basic products (``ppid=15``), -run the post-processing pipelines (``ppid=17``), and load product metadata into the -RAPID-operations PostgresSQL database. This translates into an overall throughput rate of -1.926 seconds per input science image. This does not include loading SFFT-difference-image -PhotUtils catalogs into the RAPID-operations PostgresSQL database and subsequent -source cross-matching. - -The PSF-fit catalogs made by the Python PhotUtils package from the SFFT difference images, -both positive and negative, were loaded into Sources child PostgreSQL database tables. -There were 259,157,881 Sources records loaded into the PostgreSQL database. -This number of sources scales up to about 10 billion sources for the entire GBTDS survey. -The elapsed time to load all sources into the database was ~3.5 hours with 8 parallel processes -(and both the VPO machine and the database-server machine have 8 vCPUs). - -Cross-matching the sources with astronomical objects (called AstroObjects), -resulting in records loaded into the Merges_ and -AstroObjects_ database tables, for all 358 fields of the sources -(i.e., fields overlapped by this test), was done. -The elapsed time to cross-match all sources was 19.2 hours with 8 parallel processes. -This includes cross-matching across field boundaries for sources near field edges. -The cross-matching was done with ``match_radius = 0.00001528`` degrees (half a Roman WFI pixel). -There were 88,747,880 AstroObjects records and 215,703,276 Merges records loaded -into the PostgreSQL database. Of those merges (a.k.a. lightcurve data points), 33,261 merges -resulted from cross-matching across field boundaries (i.e., the match radius can extend -across a field boundary), which is an increase of 0.0154% in terms of number of merges. - -The lightcurve statistics stored in the AstroObjects_ database tables are updated -after the cross-matching. This is done as a separate process from the cross-matching. -Any AstroObjects_ record with no associated sources in the Merges_ database table are deleted. -A new Q3C index on the (meanra, meandec) columns is computed for all AstroObjects_ database tables, -and then these tables are set to logged, clustered, and analyzed. -The AstroObjects_ database tables are explicitly vacuumed at the end of this process. -For this test, all of these items within the process took 15.9 hours with 8 parallel processes. +This instance took about 50 minutes; instances reusing existing reference +images took about 20 minutes each. + +Processing all 6,917 science images took 3.7 hours: science pipelines +generated the 109 reference images and basic products (``ppid=15``), +post-processing pipelines ran (``ppid=17``), and product metadata were loaded +into the RAPID-operations PostgreSQL database. Overall throughput was +1.926 seconds per input science image, excluding SFFT-difference-image +PhotUtils catalog loading and subsequent source cross-matching. + +Source loading and lightcurves +==================================== + +Python PhotUtils PSF-fit catalogs from positive and negative SFFT difference +images supplied 259,157,881 records to Sources child PostgreSQL tables, +scaling to about 10 billion sources for the entire GBTDS survey. Loading took +~3.5 hours with 8 parallel processes; both the VPO and database-server +machines have 8 vCPUs. + +Cross-matching sources with astronomical objects (AstroObjects) across all +358 source fields overlapped by this test took 19.2 hours with 8 parallel +processes. The ``match_radius = 0.00001528`` degrees (half a Roman WFI pixel) +includes matches across field boundaries for sources near field edges. +The Merges_ and AstroObjects_ PostgreSQL tables received +88,747,880 AstroObjects records and 215,703,276 Merges records (lightcurve +data points). Cross-boundary matching, where the match radius extends across +a field boundary, added 33,261 merges, an increase of 0.0154%. + +A separate process after cross-matching updates lightcurve statistics in +AstroObjects_ and deletes records with no associated sources in +Merges_. It builds a new Q3C index on (meanra, meandec) for all +AstroObjects_ tables, then sets them to logged, clusters and analyzes +them, and explicitly vacuums them at the end. This took 15.9 hours with +8 parallel processes. 7/22/2026 ************************************ -This test is similar to the 7/6/2026, with the following exceptions and improvements: +Similar to the 7/6/2026 test, with these changes: -* The SOC-sim ASDF images now have variable sources injected correctly, as well as - corrected WCS. These are located in: +* SOC-sim ASDF images have correctly injected variable sources and corrected WCS: .. code-block:: s3://socsims-fakesrc-asdf-20260709/ -* The SOC-sim ASDF images were converted to FITS files for input to the RAPID pipeline, with - 5th-order TAN-SIP distortion. +* FITS conversions for RAPID input use 5th-order TAN-SIP distortion: .. code-block:: s3://socsims-fakesrc-fits-20260709-lite/ -* A standalone RAPID reference-image pipeline has been implemented and integrated into the - VPO, and was executed successfully in this test to generate all required reference images - (109 in all for ``fid = 8``), - before the RAPID science pipelines were run. These reference image are registered in - the RAPID operations database under ``ppid = 12`` (i.e., socsimsdb). +* A newly implemented standalone RAPID reference-image pipeline, integrated + into the VPO, successfully generated all 109 required reference images for + ``fid = 8`` before science processing. They are registered under + ``ppid = 12`` in the RAPID operations database socsimsdb. + +Setup +==================================== -Here are details about how the reference images were configured: +Reference-image configuration: .. code-block:: @@ -432,7 +428,7 @@ Here are details about how the reference images were configured: min_n_images_to_coadd = 2 max_n_images_to_coadd = 25 -Here are details about how the test was executed via the Virtual Pipeline Operator (VPO): +Virtual Pipeline Operator (VPO) invocation: .. code-block:: @@ -445,13 +441,16 @@ Here are details about how the test was executed via the Virtual Pipeline Operat python3.11 /code/pipeline/virtualPipelineOperator.py 20260706 >& virtualPipelineOperator_20260706.out & -The pipeline product files are located in the usual place: +Results +==================================== + +Pipeline product files: .. code-block:: s3://rapid-product-files/20260722/ -Here is a summary of the pipeline exit codes after the test: +Pipeline exit codes: .. code-block:: @@ -464,8 +463,8 @@ Here is a summary of the pipeline exit codes after the test: 17 | | 205 (4 rows) -One of the AWS-Batch RAPID post-processing pipelines failed, presumably due to a network -glitch, with the following error: +One AWS-Batch RAPID post-processing pipeline failed, presumably because of +a network glitch: .. code-block:: @@ -475,44 +474,42 @@ glitch, with the following error: Head "https://public.ecr.aws/v2//rapid_science_pipeline/manifests/latest": dial tcp 75.2.101.78:443: i/o timeout -This caused the database-registration code to quit early (hence the 205 pipeline instances -with null exitcodes). -More work on the VPO and pipeline infrastructure are needed to identify and rerun failed pipelines. +The failure caused database registration to quit early, leaving 205 pipeline +instances with null exitcodes. The VPO and pipeline infrastructure need +further work to identify and rerun failed pipelines. 8/21/2026 ************************************ -This test is different from the 7/22/2026 test in several ways, as documented below. -Notably, the observation-date range is shifted to 6 days later, resulting in a -slightly higher number of science images processed. +Relative to the 7/22/2026 test, the observation-date range shifts 6 days +later, yielding slightly more science images. Other changes: -* The SOC-sim ASDF images now have variable sources injected with variability on a - shorter time scale more suitable for the available one week of simulated observations - (with Jacob's recent script modification and a new set of injection catalogs), - as well as corrected WCS. These are located in: +* Jacob's recent script modification and new injection catalogs give the + injected variable sources a shorter variability time scale, better suited + to the available one week of simulated observations. These SOC-sim ASDF + images also have corrected WCS: .. code-block:: s3://socsims-fakesrc-asdf-20260807/ -* The SOC-sim ASDF images were converted to FITS files for input to the RAPID pipeline, - with 5th-order TAN-SIP distortion (4th-order was compared in a spot test, which showed - a maximum deviation of 0.0533 pixels between ASDF vs. FITS computed sky positions). +* FITS conversions for RAPID input use 5th-order TAN-SIP distortion. A + 4th-order comparison in a spot test showed a maximum deviation of + 0.0533 pixels between ASDF and FITS computed sky positions: .. code-block:: s3://socsims-fakesrc-fits-20260807-lite/ * Jacob compiled F146 PSF models (both sci and ref) for the SOC sims based on the CRDS - reference-file ePSFs (``fid=8`` only). Because the GBTDS strategy keeps the orientation - and SCA fixed, Jacob modeled the ref PSFs to mimic, and there is only one per SCA. - Records for the science-image PSFs had to be inserted into the PSFs database table - (with ``vbest = 1``). - He made three pipeline changes, all off by default: SFFT masking settings become explicitly - configured, ZOGY's SN/SR can come from the uncertainty maps rather than the - source-dominated image scatter, and extreme artifact pixels get repaired before differencing. - To turn it on for this run: + reference-file ePSFs (``fid=8`` only). The GBTDS strategy keeps orientation + and SCA fixed; Jacob modeled the ref PSFs to mimic this, with one per SCA. + Science-image PSF records had to be inserted into the PSFs database table + with ``vbest = 1``. His three pipeline changes, all off by default, make + SFFT masking settings explicit, allow ZOGY's SN/SR to use uncertainty maps + instead of source-dominated image scatter, and repair extreme artifact + pixels before differencing. Settings for this run: .. code-block:: @@ -520,11 +517,14 @@ slightly higher number of science images processed. refimage_psf_filename = refimage_psf_f146_scaSCAID.fits repair_extreme_artifact_pixels = True -* A new set of reference images was generated (109 in all for ``fid = 8``), - before the RAPID science pipelines were run. These reference image are registered in - the RAPID operations database under ``ppid = 12`` (i.e., socsimsdb). +* All 109 new reference images for ``fid = 8`` were generated before science + processing and registered under ``ppid = 12`` in the RAPID operations + database socsimsdb. + +Setup +==================================== -Here are details about how the reference images were configured: +Reference-image configuration: .. code-block:: @@ -543,7 +543,7 @@ Here are details about how the reference images were configured: min_n_images_to_coadd = 2 max_n_images_to_coadd = 25 -Here are details about how the test was executed via the Virtual Pipeline Operator (VPO): +Virtual Pipeline Operator (VPO) invocation: .. code-block:: @@ -556,13 +556,16 @@ Here are details about how the test was executed via the Virtual Pipeline Operat python3.11 /code/pipeline/virtualPipelineOperator.py 20260821 >& virtualPipelineOperator_20260821.out & -The pipeline product files are located in the usual place: +Results +==================================== + +Pipeline product files: .. code-block:: s3://rapid-product-files/20260821/ -Here is a summary of the pipeline exit codes after the test (``exitcode = 0`` is normal): +Pipeline exit codes (``exitcode = 0`` is normal): .. code-block:: @@ -574,11 +577,10 @@ Here is a summary of the pipeline exit codes after the test (``exitcode = 0`` is 17 | 0 | 7380 (3 rows) - - -Below are numbers for this test related to extraction of lightcurves from PSF-fit SFFT-difference-image catalogs. -The number of records in the AstroObjects_, AstroObjectsMeta_, and Merges_ database tables -include contributions from the 7/22/26 socsims test, which processed SOC-sim science images observed 6 days earlier. +Lightcurve extraction from PSF-fit SFFT-difference-image catalogs: +AstroObjects_, AstroObjectsMeta_, and Merges_ record +counts include contributions from the 7/22/26 socsims test, whose science +images were observed 6 days earlier. ========================================================================================= ===================== Item Number diff --git a/docs/source/fp/fp_backend.rst b/docs/source/fp/fp_backend.rst index 3da75e04..bffec849 100644 --- a/docs/source/fp/fp_backend.rst +++ b/docs/source/fp/fp_backend.rst @@ -4,12 +4,14 @@ RAPID Forced-Photometry Backend Overview ************************************ -The python script ``pipeline/forcedPhotometryForField.py`` -is the RAPID forced-photometry backend. -For a given set of sky positions in the same sky tile (a.k.a. field), -separate forced-photometry lightcurve files will be generated, one per sky position. -Access to a RAPID operations PostgreSQL database is required, and the read-only -``DBUSER=apollo`` can be used. +The Python script ``pipeline/forcedPhotometryForField.py`` is the RAPID +forced-photometry backend. Run it inside a RAPID-pipeline container with +one or more sky positions in the same sky tile (a.k.a. field). It generates +one forced-photometry lightcurve file per sky position. + +The backend requires access to a RAPID operations PostgreSQL database; +the read-only ``DBUSER=apollo`` can be used. The ``Fields`` table defines +field centers and corners for the entire sky. For Open Univ sims, use:: @@ -27,14 +29,8 @@ For Soc-sim images, use:: Instructions ************************************ -The forced-photometry backend should be executed inside a RAPID-pipeline container. - -A set of one or more sky positions must be in same field (a.k.a sky tile) for a -given forced-photometry backend execution. -The PostgreSQL database table called ``Fields`` defines field centers and corners for the entire sky. -For now, the ``reqid`` is just an arbitrary unique index. - -First, set up a text file with input sky positions of interest:: +Create a text file of input sky positions. For now, ``reqid`` is an +arbitrary unique index:: vi input_sky_positions.txt @@ -44,7 +40,8 @@ First, set up a text file with input sky positions of interest:: 3,8.5593654,-42.272997 -Here is how to execute the forced-photometry backend inside a a RAPID-pipeline container. +Inside the RAPID-pipeline container, configure the environment and run +the backend: .. code-block:: @@ -79,7 +76,10 @@ Here is how to execute the forced-photometry backend inside a a RAPID-pipeline c tail -f forcedPhotometryForField.out -The forced-photometry lightcurve files produced in the above backend-execution example are: +Output +************************************ + +The example produces these forced-photometry lightcurve files: .. code-block:: @@ -87,6 +87,5 @@ The forced-photometry lightcurve files produced in the above backend-execution e rapid_req2_lc.txt rapid_req3_lc.txt -These output files contain a table with useful columns of pertinent metadata. -Each table row is a lightcurve data point. The table contains -lightcurve data points for all available Roman WFI bandpass filters. +Each file contains a table with metadata columns and one lightcurve data +point per row, covering all available Roman WFI bandpass filters. diff --git a/docs/source/index.rst b/docs/source/index.rst index 2673e4ab..d0dc3e9a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -4,71 +4,63 @@ contain the root `toctree` directive. RAPID Image-Difference Pipeline Documentation -#################################################### - -Welcome! This is the documentation for the RAPID Image-Difference -Pipeline, under development at IPAC/Caltech. - +############################################# .. note:: - Development of source code and documentation is currently ongoing. + The RAPID Image-Difference Pipeline source code and documentation are + under development at IPAC/Caltech. -.. note:: This Sphinx site documents the pipeline as it exists on the ``dev`` - branch. On the ``rebuild`` branch the pipeline lives under - ``rapidpipe/``, and its design and operations pages are the rapid_docs - site instead of this one. Paths named on the pages below, such as + branch. On ``rebuild``, the pipeline lives under ``rapidpipe/``; + its design and operations pages are on the rapid_docs site instead. + Paths named on the pages below, such as ``pipeline/``, ``alerts/``, ``database/schema/`` and ``database/scripts/``, exist on ``dev`` only. +Getting the Source Code +*********************** + +The source code is in the `RAPID GitHub Repository `_. + + Running the Latest RAPID Pipeline -************************************* +********************************* -A Docker image has been pre-built from a recent git-clone of the RAPID Github -repository (8/21/26). -This Docker image offers the convenience of having the RAPID -pipeline already installed and ready to run. It is publicly available from +A Docker image pre-built from a recent git-clone of the RAPID GitHub +repository (8/21/26) has the pipeline installed and ready to run. +It is publicly available at: .. code-block:: public.ecr.aws//rapid_science_pipeline:latest -It is currently approximately 8.6 GB in size, and requires sufficient disk space on the target machine. -It can be used to ``docker-run`` a container and from within execute -code for image-differencing, etc., using a ``docker-run`` command like the -following (note that an entry point to bash is required for interactive use -and to inhibit running the automated pipeline): +The image is approximately 8.6 GB; the target machine needs sufficient +disk space. Use it to ``docker-run`` a container and execute code for +image-differencing and other tasks inside it. Interactive use requires a +bash entry point, which also inhibits the automated pipeline. For example, +use a ``docker-run`` command like: .. code-block:: docker run -it --entrypoint bash --name my_test -v /home/ubuntu/work/test_20241206:/work public.ecr.aws//rapid_science_pipeline:latest -The Docker file used to generate this Docker image is +The image was built using this Docker file in the RAPID git repo: .. code-block:: rapid/docker/Dockerfile_ubuntu_runSingleSciencePipeline -in the RAPID git repo. The Docker image self-contains a -RAPID git-clone in the /code directory (no volume binding to an -external filesystem containing the RAPID git repo is necessary). The -Docker image also contains a -C-code build of the RAPID software stack with the following run-time environment: +It contains a RAPID git-clone in /code, so no volume binding to an external +filesystem containing the RAPID git repo is needed. It also contains a +C-code build of the RAPID software stack with this run-time environment: .. code-block:: export PATH=/code/c/bin:/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin export LD_LIBRARY_PATH=/code/c/lib - - -Getting the Source Code -***************************** - -Please refer to the `RAPID GitHub Repository `_ for the source code. - .. Separate file for installation of the pipeline and building the C code.