You are here: Home / Proposing / ALMA Proposal Review / Frequently Asked Questions

Frequently Asked Questions

General

 

Q: What important changes have been made to the proposal review process for Cycle 8 2021?

There are three important changes to the proposal review process in Cycle 8.

i) ALMA will adopt a distributed peer review process to scientifically review a majority of proposals submitted to the Cycle 8 2021 Call for Proposals, in which one member of the proposal team on each submitted proposal will be responsible for reviewing ten proposals. 

ii) Review panels will evaluate a subset of proposals, specifically Large Programs and proposals requesting more than 25 hours on the 12-m array.

iii) All Cycle 8 2021 proposals will be reviewed using a dual anonymous procedure, in which the reviewers will not see the names of the investigators of any proposal that they review. 

 

Q: I have a question about the process that is not covered here. What should I do? 

Contact the Helpdesk and specify the Project Planning department. Maybe your question will end up on the FAQ!

 

Dual-Anonymous proposal review

 

Q: What is dual-anonymous peer review, and why is it being implemented?  

ALMA is strongly committed to ensuring that the proposal review process is as fair and impartial as possible. Analysis of the proposal rankings in previous cycles has identified systematics that may signify the presence of biases in the review process (Carpenter 2020). To ensure that the proposal review process is as fair and unbiased as possible, ALMA is beginning dual-anonymous review in Cycle 8 2021. In a dual-anonymous review, the proposal team does not know the identity of the reviewers and the reviewers do not know the identity of the proposal team. While proposers will still enter their names and affiliations in the ALMA Observing Tool (OT), this information will not appear on the proposal cover sheet, nor in the tools used by the reviewers. It is the responsibility of the proposers to ensure anonymity is preserved when writing their proposals.

Large Programs have special procedures that will allow the ALMA Proposal Review Committee (APRC) to see the identity of the proposal team, but only after the scientific review, rankings, and recommendations have been completed. Details on the review procedures for Large Programs are described in the Call for Proposals. 

 

Q: Am I still required to enter the names of myself and co-Investigators in the ALMA Observing Tool (OT)? Why?

Yes, you will still need to enter your name and the names of your co-Investigators in the ALMA OT, even when this information will not be accessible to the reviewers. 

ALMA requires this information in the first stage of the proposal assignment process to prevent the assignment of a proposal to a reviewer with a clear conflict of interest. Later on, this information is required to associate the proposal, and eventual data if the proposal is accepted, to you and your co-Investigators.

 

Q: How will reviewers identify conflicts of interest if they have no access to the list of proposers?

Institutional conflicts will be automatically identified based on user profiles of the investigators and the reviewers (to check and/or update your user profile please click here). Potential collaborative conflicts will also be identified based on the proposal submission history of the PI and the reviewers over previous and the current cycles. No proposal will be assigned to a reviewer who is flagged with any of these types of conflicts. In addition, reviewers will be able to manually declare conflicts on their assigned proposals that are not identified by our system, such as cases where the assigned proposal is in conflict with the reviewers own proposal. In such cases, the JAO will replace the conflicted proposal with a new one.

 

Q: I would like to resubmit a proposal from a previous cycle.  May I resubmit the same “Scientific Justification” PDF as before?   

All proposals, including resubmissions, must comply with the anonymization rules. Since the dual-anonymous process is new to Cycle 8 2021, most likely you will need to modify the Scientific Justification following the dual-anonymous guidelines to ensure that it complies with the anonymization rules.

 

Q: I would like to reference our previous ALMA proposal(s), because our Cycle 8 2021 proposal is an extension of that previous work.  How can I reference proprietary ALMA data, or a previous incomplete ALMA project? 

To properly reference proprietary ALMA data in an anonymized way, please use the corresponding project code or the phrase “obtained via private communication", but a name should not be specified since it could strongly imply who may be an investigator on the proposal.   

 

Q: What will happen to a proposal that is not fully anonymized?

Any proposals that appear to be in flagrant violation of the guidelines will be discussed internally between the Joint ALMA Observatory (JAO) Proposal Handling Team and the ALMA Director. If the violation is confirmed, the proposal will be rejected and will not be further reviewed.  Less serious violations of the guidelines will be flagged by the JAO, but the proposal will continue to be reviewed.  The JAO will provide proper feedback to the Principal Investigator if any violation is detected.

 

Q: How can I know if I am following the guidelines on anonymity? 

Guidelines are given in the dual-anonymous guidelines. If you have questions about making your proposal fully anonymized according to the guidelines, you can always contact your ALMA Regional Center (ARC) through the Helpdesk. Your ARC support staff will be able to help answer your question, or they will forward your question to the JAO Proposal Handling Team for additional assistance with the guidelines.

 

Q: I’m an expert in a small, niche field, and many of the references relevant to my proposal are by members of our team.  Even if I follow all of the guidelines to anonymize my proposal, I’m worried that my proposal may be identifiable.  Could my proposal still be disqualified?

Your proposal will not be disqualified as long as you have followed the dual-anonymous guidelines to anonymize your proposal. The purpose of dual-anonymous is not to disregard your expertise or foundational work, but to guide the reviewer to focus and assess the science of a proposal rather than the group behind it. As long as the proposal is written in such a way that you do not explicitly identify yourself or your team, your proposal will be in compliance.

 

Q: In my proposal, I would like to refer to data and/or software that are unpublished. How can we incorporate this information without disclosing our identity?

As described in the dual-anonymous guidelines, unpublished data or software can be referenced as "obtained via private communication".  Unpublished data can also be referenced by a project code as long as it is not accompanied by the name of the PI.

 

Q: Is the usage of the words "we" or "I" (or other identifying pronouns) prohibited in dual-anonymous peer review?

No, it is not entirely prohibited. First person pronouns (we, I, our, my) can be used to express the intent of the proposal itself, but not to identify the proposal authors.  For example, you may use the phrases, “We plan to observe the CO molecular line…” or “Our preferred target is Target X due to…” or even, "Following the observational strategy employed in Perez et al. (2019), we plan to..." On the other hand, you should not use phrases like “We showed in Perez et al. (2019)...” that connect the selected pronouns to the proposal authors.  

 

Q: How do I show that I’m an expert equipped to work with the data, without revealing myself? 

Your description of the scientific background, motivation, planned observations, and technical justification should express a level of expertise sufficient to convince the reviewers or panels, without needing to identify yourself personally.

 

Q: Do proposals for Large Programs also need to follow the dual-anonymous guidelines?  

Yes, the Scientific and Technical Justification of Large Programs must be fully anonymized in the same way as all proposals. In addition, Large Programs are required to include a one-page PDF file (the "management plan") that describes the roles of the proposal team and the available computing resources. The management plan should include the names of at least the key members of the proposal team. 

Large Programs will be ranked based only on the scientific and technical justification. After the Large Programs have been scientifically ranked by the ALMA Proposal Review Committee (APRC), the APRC will be able to review the management plans. The APRC will be permitted to note any concerns on the qualifications of the management team to carry out the proposed science. The concerns will be forwarded to the ALMA Director for consideration. However, the scientific ranking of the Large Programs will remain the same. Further information of the review process for Large Programs can be found in the Proposer's Guide.

                                                            

Q: Where can I find more information about dual-anonymous peer review, as it has been applied and assessed in other proposal review processes for other facilities? 

JWST, HST, other NASA science missions, and ESO, have already implemented dual-anonymous in their call for proposals. You can find more information, including additional links and references, in the following:

JWST → https://jwst-docs.stsci.edu/jwst-opportunities-and-policies/jwst-call-for-proposals-for-cycle-1/jwst-cycle-1-anonymous-proposal-review

HST → https://outerspace.stsci.edu/display/APRWG/Recommendations+of+the+Working+Group+on+Anonymizing+Proposal+Reviews

NASA → https://science.nasa.gov/researchers/dual-anonymous-peer-review

ESO → https://www.eso.org/sci/observing/phase1/dual-anonymous-guidelines.html

 

Distributed peer review

 

Q: What is distributed peer review?

Distributed peer review is a process in which one member of the proposal team, either the Principal Investigator (PI) or a co-Investigator (co-I), commits to review ten other submitted proposals. Therefore, the best proposals are selected by peer review, but instead of having a small committee review a large number of proposals, a larger number of people each review a smaller number of proposals. In this manner, the review load is distributed among many peers, and any individual reviewer will have a lower workload, with more time to spend reviewing each assigned proposal.  

 

Q: Are other observatories using distributed peer review?

Yes: Gemini’s Fast Turnaround proposal system uses distributed peer review and ESO initiated a pilot program in 2018, which received a very positive response from the community.

 

Q: Why is ALMA using distributed peer review instead of the traditional panels to review a subset of the proposals on the Main Call?

Distributed peer review proved to be a viable review system for ALMA and its community after the success of the pilot program for the Cycle 7 Supplemental Call in 2019. The advantages that this review system provides include an increased involvement of the community in the review process and a significant reduction of the workload for each reviewer.  Based on the success of the pilot program, feedback from the community who participated in the pilot program, and advice from the ALMA Science Advisory Committee, the ALMA Board approved using distributed peer review for the smaller proposals in Cycle 8 2021 while using traditional panel reviews for the larger proposals.

 

Q: How will the distributed peer review process work for the Cycle 8 2021 Call for Proposals?

Proposals requesting less than 25 hours on the 12-m array and less than 150 hours on the 7-m array will be evaluated through distributed peer review.  At proposal submission, the PI of such proposals must designate someone from the proposal team to be the reviewer. In most cases the PI will be the designated reviewer, but the PI may also designate a co-I. After the proposal deadline, the Proposal Handling Team (PHT) at the Joint ALMA Observatory (JAO) will assign ten proposals to the designated reviewer of each proposal. If reviewers have a conflict of interest with any of their assigned proposals, they can request a replacement proposal through the Reviewer Tool. Then during Stage 1, the reviewer will rank the ten proposals (1-10) in order of scientific priority, and write a brief review for each proposal. If ranks and reviews are not submitted by the time of the Stage 1 review deadline, the proposal on which the reviewer is acting as the designated reviewer will be rejected. After the Stage 1 review deadline, each reviewer will have the option to participate in a second stage (Stage 2) of the review process where the anonymized comments from the other reviewers of the same proposals will be made available. Reviewers can modify their own ranks and comments if desired in Stage 2.  Once this second stage is completed, the ranks from all reviewers of each submitted proposal will be combined to produce a global ranked list of proposals, which will be used to produce the observing queue after merging with the results of the panel process.

 

Q: Who can I designate to be the reviewer for my submitted proposal? As a PI, how do I designate a reviewer?

Any PI and most co-Is on the proposal can be designated as the reviewer. If the PI does not have a PhD at the time of proposal submission (e.g., a student), the PI can still be the reviewer, but a mentor (who must have a PhD) must be identified at the time of the proposal submission. A PI may designate a co-I as the reviewer as long as the co-I has a PhD in astronomy or a closely related field. The reviewer and if needed, the mentor, must be designated in the Observing Tool (OT) at the time of proposal submission and cannot be changed after the proposal deadline.

 

Q: Can a student (without a PhD) be a designated reviewer?

A student can be the designated reviewer if they are the PI of the proposal and also designates a mentor, who must have a PhD in astronomy or a closely related field. The mentor must be identified when the proposal is submitted. The mentor does not have to be a co-I on the student’s proposal. A student co-I is not eligible to be a designated reviewer.

 

Q: What is the role of a mentor?

A mentor should provide guidance as needed to a student reviewer during the process, but there are no formal requirements. The mentor must abide by the same confidentiality requirements as the student reviewer. 

 

Q: As a PI, is there anything else I should keep in mind when submitting my proposal?

PIs should ensure that the designated reviewer has an up-to-date email address in the Observing Tool. If the email is not up-to-date, please ask the reviewer to update their ALMA user profile through the ALMA Science Portal. The JAO will use email to communicate with your reviewer, so if the email address is out of date, there is a chance that your reviewer will not submit their reviews on time and your proposal will be rejected as a result. It is the responsibility of the PI and the designated reviewer to ensure the reviews are submitted on time.

Additionally, please encourage the reviewer to enter their fields of expertise in their ALMA Science Portal profile. This information is extremely important for the optimal assignment of the proposals that each person will review. 

 

Q: How will proposals be assigned to reviewers?

To the extent possible, the JAO will assign proposals based on the expertise of the reviewer. A reviewer can indicate their expertise in their user profile on the ALMA Science Portal, where they can specify the scientific category/keywords that best match their expertise. If a reviewer does not complete their user profile, the category/keywords of the submitted proposal will be used for the assignment. The JAO will also avoid potential conflicts of interest by checking for frequent collaborations on submitted proposals. However, because of conflicts and the limited number of proposals submitted in some category/keywords, the JAO may also assign proposals in similar category/keywords, and on rare occasions, a proposal from a different category may be assigned. Keyword/category matches can be made most of the time.

Due to the nature of the assignment algorithm, it is very important that all reviewers add their expertise information to their profile on the ALMA Science Portal and keep it up-to-date.  This will allow the JAO to optimize the proposal assignments to each reviewer. 

 

Q: What if I am participating in the distributed review process and I do not submit my ranks and reviews by the Stage 1 review deadline? If I submit the ranks for one proposal set but not the other, will both of my proposals be rejected, or only the one for which I failed to submit the ranks?

If a designated reviewer does not submit their reviews before the Stage 1 review deadline (see question 5 for information about Stage 1), the proposal for which they were identified as the designated reviewer will be rejected. If a user is the designated reviewer for more than one proposal, they will be assigned a distinct proposal set to review for each submitted proposal. If the reviewer submits ranks and reviews for one proposal set but not for another, only the submitted proposal(s) associated with unsubmitted ranks and reviews will be rejected.

 

Q: Will my proposal really be rejected if I don't submit all my reviews before the Stage 1 deadline of the distributed review process?

Yes. 

 

Q: What if I am the PI of several proposals that qualify for the distributed review process? Will I have to review proposals for each of my submitted proposals?

A PI may submit several proposals that qualify for the distributed peer review process. For each of those proposals, a PI will need to designate a reviewer. If you have submitted several proposals as PI, you do not have to be the designated reviewer on all (or any) of them. The designated reviewer for each proposal can be the PI or a co-I, and that person commits to review ten other proposals. If the same person is the designated reviewer associated with more than one proposal, then that person will review ten proposals for each respective proposal. For example, if a reviewer is the designated reviewer on behalf of 3 proposals, then that person should expect to review 30 other proposals.

In addition, the same PI may submit proposals requesting more than 25 hours of time on the 12-m array, and/or Large proposals, which will instead be reviewed by panels.

 

Q: What is the timeline and deadlines for the panel and distributed peer review processes? 

The schedule for the distributed peer review process will be outlined in detail in the Cycle 8 2021 Call for Proposals, and the schedule for the panel review is being communicated with the selected panelists.  Broadly, the various stages of the review processes will take place in May and June, with results communicated with PIs by early August. This is all in preparation for the beginning of the Cycle 8 2021 observations in October.

 

Panel review

 

Q: For proposals not included in the distributed peer review process, how will the panel review process work for the Cycle 8 2021 Main Call?

All Large Programs, and any proposals requesting more than 25 hours on the 12-m array, will be peer reviewed by volunteers from the community serving on traditional review panels with topical expertise. The JAO will assign each submitted proposal to a panel based on the scientific category selected by the PI. 

In the first stage of the panel review process, panelists will individually score each non-conflicted proposal assigned to them and will write a brief review. In the second stage, panelists will meet (virtually, in Cycle 8) to discuss each proposal assigned to their panels. For Large Programs, the panels will vote to decide which Large Programs should advance to the ALMA Proposal Review Committee (APRC) for further review. For the remaining proposals (i.e., proposals requesting between 25 and 50 hours on the 12-m array), the panelists will re-score the proposals based on the panel discussion and the average score will be used to determine the proposal rank. Panelists will finalize the consensus reports based on the individual comments and the discussion during the meeting. The ranked lists from each panel will be merged, and later used to create the observing queue.

Following the ALMA Review Panel meetings, the ALMA Proposal Review Committee (APRC) will meet to review the Large Programs selected by the panels. The APRC will then re-score the Large Programs and make a recommendation on which Large Programs to schedule.

 

Q: I submitted a proposal that requests more than 25 hours of time on the 12-m array, but is not a Large Program.  Do I still have to designate a reviewer?

Yes. The Observing Tool will automatically require a reviewer to be designated for any proposal that is not a Large proposal. For all proposals that request between 25 to 50 hours on the 12-m array, the PI is required to designate a reviewer, and the reviewer will not need to review proposals as long as the final time request confirmed by the JAO is more than 25 h on the 12-m array.

 


Return to the main ALMA Proposal Review page