You are here: Home / Proposing / ALMA Proposal Review / Distributed Peer Review

Distributed Peer Review

All proposals submitted to the Main Call requesting less than 25 h on the 12-m Array or less than 150 h on the 7-m Array, and all proposals submitted to the Supplemental Call, will be peer reviewed using a distributed system in which each proposal team selects a member of the team to participate as a reviewer in the review process. 

Basic rules

  1. All participants in the review process are expected to behave in an ethical manner. If it is found that a reviewer has not behaved in an ethical manner, the proposal(s) on which the reviewer is acting as the designated reviewer may be rejected. 
  2. Each proposal must designate one reviewer to participate in the review process. The designated reviewer may be the PI of the proposal or one of the co­-Is. 
  3. The reviewer must be specified in the Observing Tool (OT) at the time of proposal submission. The reviewer can be changed after the proposal deadline only in exceptional circumstances (e.g., illness, urgent care for family member). A PI can request to change the reviewer by contacting the JAO Proposal Handling Team (PHT) at pht@alma.cl. If the PHT approves the request, the new reviewer will be given access to the assigned proposals and will assume the responsibility of completing the review assignments. The Stage 1 deadline for the new reviewer will remain the same since the Stage 2 process starts shortly after Stage 1 is completed.

  4. PIs who do not have a PhD may be selected as the designated reviewer. In such cases, a mentor must be specified who will assist the PI in the review process. The mentor does not need to be part of the proposal team, but must have a PhD in astronomy or a related field, and must be specified in the OT at the time of proposal submission. Co-Is who do not have a PhD are not eligible to be selected as reviewers.
  5. Proposals will be assigned to reviewers based on the expertise of the reviewer as specified on the reviewer's user profile. If a reviewer has not registered their expertise, then the assignment algorithm will use the keywords of their submitted proposals.
  6. Reviewers must declare any major conflicts of interest of their assigned proposals. Any proposals with a major conflict of interest will be replaced by another proposal.
  7. During Stage 1, each designated reviewer will be assigned to review ten proposals. The reviewer must rank the proposals relative to each other in order of scientific priority from 1 to 10 (1 being the strongest, and 10 being the weakest), and write a review for each one of them. If a person is the designated reviewer on multiple proposals, they will receive ten unique review assignments per submitted proposal. 
  8. Stage 1 is mandatory. If a reviewer does not submit their reviews and ranks by the Stage 1 deadline (16 November 2021 15:00 UT), the proposal for which they were identified as the designate reviewer will be rejected.
  9. Stage 2 is optional. During Stage 2, the Stage 1 reviews will be shown anonymously to the other (9) reviewers of each proposal. The reviewers will then be able to re-rank their assigned proposals and edit their reviews.

  10. Reviews and ranks will be sent anonymously to PIs without any editing by the JAO. If reviewers participate on Stage 2, then their edited reviews and modified ranks will be sent to the PIs, otherwise the Stage 1 reviews and ranks will be sent.
  11. All participants in the review process agree to keep the materials confidential and will not use the materials for any other means other than the proposal review. Participants will delete any proposals and any other review materials after they have completed their assessments.
  12. All communications between the PHT and the reviewers will be done by email. The PHT will use the email address associated with the reviewers’ ALMA user account. Please make sure to keep your user profile updated so you do not lose important information.

Stage 1

The JAO PHT will assign 10 proposals per each submitted proposal; this set of 10 proposals is called a "Proposal Set". During Stage 1, reviewers will rank the proposals in their Proposal Set relative to each other in order of scientific priority (from 1 to 10, strongest to weakest), and will write a review for each one of them. Reviewers access their Proposal Set through the Reviewer Tool. Usage of this tool is explained in How to use the Reviewer Tool. All reviewers should refer to that documentation for a detailed description of the tool functionalities; here a broad outline of the main actions is provided.

At Stage1, all reviewers should carry out the following sequence of steps:

  1. Access the Reviewer Tool.
  2. Download the proposal package.
  3. Browse through the proposals in their Proposal Set and declare in the Reviewer Tool any possible conflict of interest that the assignment algorithm did not identify.

  4. Submit the conflict decisions before starting to review the proposals. Reviewers should submit their decisions by 26 October 2021 15:00 UT, at which time the JAO replace any conflicted proposals with other proposal assignments. If a person is the designated reviewer on more than one proposal (i.e. they have more than one Proposal Set), they will need to identify conflicts of interest on all their Proposal Sets before being able to submit them. Even if no conflicts are detected, the reviewer should submit the decisions by the deadline. Conflict decisions can still be submitted after the mentioned date, but the earlier they are submitted, the more likely a reviewer will be assigned replacement proposals that are aligned with their expertise.

  5. Review the proposals to provide a rank from 1 to 10 (strongest to weakest) and write a review that summarizes the strengths and weaknesses of the proposal.
  6. Submit the ranks and reviews. Stage 1 is mandatory. If a reviewer does not submit the Stage 1 ranks and reviews by the Stage 1 deadline (16 November 2021 15:00 UT), then the proposal for which they were identified as the designated reviewer will be rejected. If a person is acting as designated reviewer on more than one proposal, then they will be able to submit the reviews for each Proposal Set independently. 
  7. Submit the optional reviewer survey.

Stage 2 

In Stage 2, reviewers who submitted their Stage 1 ranks and reviews have the option to read the anonymized reviews from the other (9) reviewers who were assigned each proposal. This will allow reviewers to determine if they overlooked critical strengths and weaknesses in their original review that were identified by other reviewers. If desired, reviewers can then modify their original ranks and reviews. If a reviewer participates in Stage 2 but does not submit their ranks and reviews in Stage 2, the Stage 1 ranks and reviews will be considered final. Note that the purpose of Stage 2 is not to start a conversation among reviewers to drive to a consensus among them, but to help them identify important issues in the proposal that may have been overlooked initially. 

If reviewers participate in Stage 2, they should carry out the following sequence of steps:

  1. Access the Reviewer Tool.
  2. Access the anonymized reviews from the other (9) reviewers following the instructions in Section 7 of How to use the Reviewer Tool.
  3. Edit the Stage 1 reviews as needed, and/or re-rank the proposals in their Proposal Set.

  4. Submit the updated reviews and ranks. The deadline to complete the Stage 2 review is 24 November 2021 15:00 UT. Note that Stage 2 is optional, no proposals will be subject to disqualification if their designated reviewer does not submit the Stage 2 ranks and reviews by the deadline.

Once Stage 2 is completed, the ranks from all reviewers of each proposal are combined to produce a global ranked list. For the Main Call for Proposals, this global ranked list will then be merged with the results of the panel review and used to produce the observing queue. In the case of the Supplemental Call proposals, the global ranked list from the distributed peer review will be used to produce the observing queue.

 


Return to the main ALMA Proposal Review page