A Comparison of Panel Review and Distributed Peer Review at ALMA
|
Proposal Review Table of Contents |
Overview
Beginning in 2021, the Atacama Large Millimeter/submillimeter Array (ALMA) transitioned from traditional panel-based review to a distributed peer review (DPR) system, becoming one of the largest astronomical facilities to adopt this approach. This page summarizes results from a multi-cycle empirical study comparing proposal outcomes under these two review systems.
The analysis spans 13 ALMA cycles (Cycles 0–12) and includes more than 20,000 proposals and over 160,000 individual reviews. It examines whether the statistical patterns of proposal evaluation changed following the transition from panel review to DPR.
A link to the peer-reviewed paper is provided at the end of this page for readers interested in methods, figures, and statistical tests.
Results at a glance
- DPR produces proposal outcome patterns that are statistically consistent with those from traditional panel review.
- Scientific diversity among top-ranked proposals is similar under both systems.
- Individual reviewer assessments of a proposal show substantial dispersion under both DPR and panel review. Under DPR, this dispersion is more visible because individual reviewer assessments are shared with proposers.
- Most reviews are rated by reviewers as high quality (54%) or adequate (36%), with no strong dependence on the career stage of the reviewer who wrote the comment. However, about 10% of reviews are rated as low quality, illustrating the challenge of ensuring all reviews meet a minimum standard across ~1,000 independent reviewers each cycle.
These results do not imply that individual proposal rankings would be identical under the two systems. Rather, they show that the statistical structure of proposal evaluation across the proposal population is similar.
What is Distributed Peer Review?
Under ALMA's DPR system, each Principal Investigator (PI) designates a member of their proposing team to serve as a reviewer. Approximately 1,000 reviewers participate in each cycle. Each reviewer independently evaluates one or more sets of 10 proposals matched to their expertise, without panel discussion. Independent assessments for each proposal are then combined to produce the final ranked list.
In contrast, traditional panel review assigns 6–8 reviewers to evaluate ~100 proposals collectively, with face-to-face discussion to establish consensus rankings.
Why was this study conducted?
ALMA has experienced sustained growth in proposal submissions since the start of operations, increasing the logistical and community burden of traditional panel review. DPR was introduced to address these scalability challenges by relying on independent assessments from a larger reviewer pool.
Building on the initial assessments of Cycle 8 (Donovan Meyer et al. 2022; Carpenter et al. 2022), this study extends the analysis to five complete DPR cycles (Cycles 8–12). The goal is to evaluate whether the transition to DPR was associated with systematic differences in proposal rankings, scientific breadth, the consistency of reviewer assessments, or the quality of written feedback.
What was compared?
- Panel-based review in Cycles 0–7 (2011–2019)
- Distributed peer review in Cycles 8–12 (2021–2025)
Large Programs, Director's Discretionary Time proposals, and special calls are excluded. Results are aggregated across cycles to reflect persistent patterns rather than year-to-year fluctuations.
Important Context
The transition from panel review to DPR coincided with the introduction of dual-anonymous review. The analysis therefore compares the combined effect of these changes relative to earlier panel-based review, rather than isolating the impact of DPR alone. As a result, this is an observational study rather than a controlled experiment.
Main results
1. Do proposal rankings change under DPR?
Proposal ranking distributions were examined across multiple dimensions, including PI demographics (region, experience, gender), technical characteristics (observing mode, receiver band, requested time), and special proposal categories.
Across the full proposal population, the distribution of proposal ranks produced under DPR closely resembles those from panel review. Long-standing statistical patterns persist in both systems, including regional trends. Highly experienced PIs ranked above average under panel review, but this advantage is no longer present under DPR, likely because dual-anonymous review was introduced at the same time. No statistically significant differences are observed with respect to PI gender. Ranking behavior as a function of observing mode, receiver band, requested observing time, and special proposal categories also remains mostly consistent across review systems (see Figure below).
These population-level patterns are already present in pre-discussion (Stage 1) panel rankings. While panel discussion alters individual proposal ranks, it does not introduce new systematic trends or eliminate existing ones.
2. Is scientific diversity affected under DPR?
Scientific diversity was evaluated by comparing the research areas represented among the top-ranked proposals under each system, using two complementary approaches: PI-selected ALMA science keywords and a machine-learning-based topic modeling method (described in Carpenter et al. 2025). For both diversity metrics examined, the fraction of research areas represented among the highest-ranked proposals is similar under both review systems.
3. How much do reviewers disagree, and is this unique to DPR?
In ALMA's DPR system, each proposal is evaluated by ten independent reviewers. The resulting ranks are shared with PIs alongside the corresponding comments. With access to the full set of individual rankings, PIs often notice substantial variation among reviewer assessments. For example, in Cycles 8–12, more than a quarter of proposals received both the highest rank (1) and the lowest rank (10) from different reviewers. This has raised questions about the consistency of proposal evaluations in the absence of panel discussion.
To place these results in context, DPR rankings were compared with simulated rankings derived from panel reviewer assessments both before and after panel discussion. The analysis shows that the wide dispersion observed in DPR is not unique to that system: comparable variation is also present in independent panel assessments prior to discussion, with the two differing by only 1.6% in median RMS dispersion. A noticeable level of dispersion remains even after discussion, which reduces the median RMS by only 9.1% relative to DPR. The key difference lies in visibility: under DPR, PIs see all individual rankings, whereas under the panel system they historically received only a single consensus assessment.
4. Is review quality maintained under DPR?
During the initial DPR cycles, review quality was assessed through PI surveys rating the usefulness of written comments. These surveys have a known limitation: ratings tend to correlate with review outcomes, making it difficult to assess comment quality independently of the rank received (Donovan Meyer et al. 2022). To address this, a reviewer-based assessment was used in Cycle 12, in which reviewers evaluated comments written by peers, without knowledge of the writer's identity, career stage, or assigned rank.
Reviewers rated most reviews as high quality (54%) or adequate (36%). The quality ratings show no significant dependence on the career stage or self-reported expertise of the reviewer who wrote the comment. Reviews written by students and postdocs are rated comparably to those written by senior researchers.
Nevertheless, the 10% of reviews rated as low quality highlights the challenge of ensuring all reviews meet a minimum standard across ~1,000 independent reviewers. ALMA continues to refine reviewer guidance and training, and to identify and follow up with reviewers whose reviews do not meet expected standards.
Frequently Asked Questions
Substantial variation in individual assessments is an inherent feature of peer review. Different experts may evaluate the same proposal from different scientific perspectives, leading to a range of judgments about its relative merit. Under the previous panel-based system this variation was less visible because proposers received only the panel's consensus assessment. Under DPR, the individual reviewer assessments are shared with proposers, making the full range of views more transparent.
No. While individual reviewer assessments vary, the overall proposal ranking reflects the combined evaluations of multiple independent reviewers. Aggregating multiple assessments reduces the influence of any single reviewer and produces rankings that capture systematic patterns in reviewer judgment rather than random variation.
With approximately 1,000 independent reviewers participating each cycle, some variation in review quality is unavoidable. ALMA evaluates review quality using several indicators to identify reviews that may not meet expected standards and follows up when appropriate. Efforts to improve the identification of low-quality reviews and to strengthen reviewer guidance are ongoing.
Learn more
Current Study
For full methodology, figures, and statistical analyses, readers are encouraged to consult the complete paper.
- Full paper: Carpenter, J. M. & Corvillón, A. 2026, "Distributed Peer Review at ALMA: An Empirical Comparison with Panel-Based Evaluation", PASP (arxiv: 2606.22160)
Previous ALMA Review Process Studies
This study builds on systematic evaluation of ALMA's proposal review process since 2016:
- 2016: Lonsdale et al., "Gender-Related Systematics in the NRAO and ALMA Proposal Review Processes, arXiv 1611.04795" — Analysis of gender-based systematics at NRAO facilities, including ALMA Cycles 2–4
- 2020: Carpenter, "Systematics in the ALMA Proposal Review Rankings", PASP, 132, 024503 — Analysis of panel-based review systematics in ALMA Cycles 0–6
- 2020: Carpenter et al., "Analysis of the ALMA Cycle 7 Supplemental Call", ALMA Memo 616 — Analysis of the Cycle 7 Supplemental Call, a pilot implementation of DPR
- 2022: Donovan Meyer et al., "Analysis of the ALMA Cycle 8 Distributed Peer Review Process", BAAS, 54, 43 — Analysis of DPR outcomes in ALMA Cycle 8, the first main call to use DPR
- 2022: Carpenter et al., "Update on the Systematics in the ALMA Proposal Review Process After Cycle 8", PASP, 134, 45001 — Analysis of proposal review systematics in ALMA Cycles 0–8, including the first DPR and dual-anonymous cycle
- 2025: Carpenter, Corvillón, & Shah, "Enhancing Peer Review in Astronomy: A Machine Learning and Optimization Approach to Reviewer Assignments for ALMA", PASP, 137, 4501 — Description of a machine learning approach to proposal-reviewer matching
Related Resources
- ALMA proposal review process: ALMA Science Portal
Return to the main ALMA Proposal Review page