Overview and Pipeline

ALMA calibrates and images PI data following the "Quality Assurance level 2" or QA2 process.  Most (>95%) of ALMA PI data is now processed by the pipeline through calibration and imaging, with the remainder processed manually by ALMA staff (the fraction processed manually was larger in earlier cycles).  Pipeline and manual processing both use CASA.  All data are reviewed by a data analyst before delivery, to ensure quality and that the PI-requested angular resolution and sensitivity are achieved.

The processed data are packaged for distribution through the archive, and as described in the ALMA QA2 Data Products Document, products include images, but not calibrated visibilities.

For pipeline-processed data, the most useful resource is the pipeline weblog, included in the qa/ directory of a product package, which contains a very detailed log, with heuristics, quality assurance, and plots of the processing steps and outcomes.  The Pipeline User's Guide (links below) describes in detail how the pipeline works, and the User's Guide version corresponding to the pipeline version listed in the weblog is the most accurate for that version. A list of known issues at the time of publication is included in the User's Guide, and a living version is maintained here.

Users can recover calibrated visibilities by several methods:

• by running scriptForPI.py contained in the archived QA2 package.  In general, one must run that script in nearly the same version of CASA used to originally calibrate the data as noted below (CASA task parameters and inputs may have changed).  For pipeline processed data, one can use the latest accepted version of CASA, with some exceptions for older data as noted in the Table below.  See the ALMA QA2 Data Products Document for more details on running scriptForPI.py and its options.
• for Cycle 5 and later data, by running hifa_restoredata.py in the latest CASA - see instructions on the ReImaging Help Page.
• by request to the helpdesk, staff at your ARC will restore the visibilities and provide them to you for download. (knowledgebase instructions)

For all data analysis other than restoring older calibrated visibilities, ALMA strongly recommends using the most recent version of CASA that has been accepted by ALMA for data processing (see Table below). These accepted versions of CASA also contain the data processing pipeline, which the user can rerun themselves, as described in the Pipeline User's Guide.   Additionally, a CASAguide describing interferometric pipeline re-imaging examples is available at: https://casaguides.nrao.edu/index.php/ALMA_Imaging_Pipeline_Reprocessing. 

In general, pipeline tasks from any pipeline release should run successfully on data from that cycle or earlier.  Using the most recent pipeline version will include improved calibration, flagging, and imaging heuristics that should improve the products, resulting in differences from previous versions (from modest to significant, depending on how early the original version was). To get the identical results as an archived pipeline product, one should use the version of CASA+PL that was used to produce the archived product.

The pipeline is not officially validated on MacOS.

PIs are strongly requested to note the versions of both CASA and of the Pipeline in their publications, or that the data were calibrated manually by ALMA staff, to provide a clear record of what heuristics were used to process the data. 

CASA versions accepted for ALMA data processing: