Frequently Asked Questions
|
Proposal Review Table of Contents |
General
Q1: I have a question about the process that is not covered here. What should I do?
Submit an ALMA helpdesk ticket to the department called "Proposal Review Support".
Q2: What important changes have been made to the proposal review process for Cycle 11?
You can find the major changes introduced in Cycle 11 here.
Q3: What is the list of conflicts of interest and for what it is used?
Reviewers and mentors have the option to provide a list of investigators for whom they have a major conflict of interest. This list can be provided in their user preferences through the ALMA Science Portal. Reviewers will not be assigned a proposal in which the PI, a co-PI, or one of the co-Is is in the list of conflicts of interest provided by the reviewer or their mentor.
Q4: How do I know if I have a conflict of interest with someone or not?
To identify your conflicts of interest please follow the guidelines described here.
Q5: If I am a reviewer, or mentor, is it mandatory for me to provide a list of conflicts of interest?
No, providing a list of conflicts of interest is optional for all ALMA users.
Q6: If I am a reviewer or mentor, what will happen if I do not provide a list of conflicts of interest?
If a reviewer does not provide the list of their conflicts of interest, a list of potential conflicts of interest will be identified based on the proposal submission history of the reviewer, as described in the conflict criteria.
Dual-Anonymous proposal review
Q1: What is dual-anonymous peer review, and why has it been implemented?
ALMA is strongly committed to ensuring that the proposal review process is as fair and impartial as possible. Analysis of the proposal rankings in previous cycles has identified systematics that may signify the presence of biases in the review process (Carpenter et al. 2022). To ensure that the proposal review process is as fair and unbiased as possible, the ALMA proposal review process follows a dual-anonymous system. In a dual-anonymous review, the proposal team does not know the identity of the reviewers and the reviewers do not know the identity of the proposal team. While proposers will still enter their names and affiliations in the ALMA Observing Tool (OT), this information will not appear on the proposal cover sheet, nor in the tools used by the reviewers. It is the responsibility of the proposers to ensure anonymity is preserved when writing their proposals.
Large Programs have special procedures that will allow the ALMA Proposal Review Committee (APRC) to see the identity of the proposal team, but only after the scientific review, rankings, and recommendations have been completed. Details on the review procedures for Large Programs are described in the Call for Proposals.
Q2: Am I still required to enter the names of myself and co-I's in the ALMA Observing Tool (OT)? Why?
Yes, you will still need to enter your name and the names of your co-I in the OT, even though this information will not be accessible to the reviewers.
ALMA requires this information in the first stage of the proposal assignment process to prevent the assignment of a proposal to a reviewer with a clear conflict of interest. Later, this information is required to associate the proposal, and eventual data if the proposal is accepted, to you and your co-I.
Q3: How will reviewers identify conflicts of interest if they have no access to the list of proposers?
The Proposal Handling Team (PHT) at the Joint ALMA Observatory (JAO) will identify conflicts of interest based on the list of conflicts of interest provided by the reviewers in their user preferences. Reviewers will not be assigned a proposal in which a PI, co-PI, or co-I is in this list. If a reviewer does not provide this information, potential collaborative conflicts will be identified based on the proposal submission history of the reviewer. In addition, reviewers will be able to manually declare conflicts on their assigned proposals that are not identified by our system, such as cases where the assigned proposal is in conflict with the reviewer's own proposal. In such cases, the PHT will replace the conflicted proposal with a new one.
Q4: I would like to resubmit a proposal from a previous cycle. May I resubmit the same “Scientific Justification” PDF as before?
All proposals, including resubmissions, must comply with the anonymization rules. PIs should verify that the resubmitted proposal complies with the current dual-anonymous guidelines.
Q5: I would like to reference our previous ALMA proposal(s), because our new proposal is an extension of that previous work. How can I reference proprietary ALMA data, or a previous incomplete ALMA project?
To properly reference proprietary ALMA data in an anonymized way, please use the phrase “obtained via private communication". Note that a name should not be specified since it could strongly imply who may be an investigator on the proposal. Further information and examples can be found in the dual-anonymous guidelines.
Q6: What will happen to a proposal that is not fully anonymized?
Any proposals that appear to be in significant violation of the guidelines may be rejected. Any such violations will be reviewed by the Proposal Handling Team and the final decision will be made by the ALMA Director.
Q7: How can I know if I am following the guidelines on anonymity?
Guidelines are given in the dual-anonymous guidelines. If you have questions about making your proposal fully anonymized according to the guidelines, you can contact your ALMA Regional Center (ARC) through the ALMA helpdesk . Your ARC support staff will be able to help answer your question, or they will forward your question to the Proposal Handling Team for additional assistance with the guidelines.
Q8: I’m an expert in a small, niche field, and many of the references relevant to my proposal are by members of our team. Even if I follow all of the guidelines to anonymize my proposal, I’m worried that my proposal may be identifiable. Could my proposal still be disqualified?
Your proposal will not be disqualified as long as you have followed the dual-anonymous guidelines to anonymize your proposal. The purpose of dual-anonymous is not to disregard your expertise or foundational work, but to guide the reviewer to focus and assess the science of a proposal rather than the group behind it. As long as the proposal is written in such a way that you do not explicitly identify yourself or your team, your proposal will be in compliance.
Q9: In my proposal, I would like to refer to data and/or software that are unpublished. How can we incorporate this information without disclosing our identity?
As described in the dual-anonymous guidelines, unpublished data or software can be referenced as "obtained via private communication".
Q10: Is the usage of the words "we" or "I" (or other identifying pronouns) prohibited in dual-anonymous peer review?
No, it is not entirely prohibited. First person pronouns (we, I, our, my) can be used to express the intent of the proposal itself, but not to identify the proposal authors. For example, you may use the phrases, “We plan to observe the CO molecular line…” or “Our preferred target is Target X due to…” or even, "Following the observational strategy employed in Perez et al. (2019), we plan to..." On the other hand, you should not use phrases like “We showed in Perez et al. (2019)...” that connect the selected pronouns to the proposal authors.
Q11: How do I show that I’m an expert equipped to work with the data, without revealing myself?
Your description of the scientific background, motivation, planned observations, and Technical Justification should express a level of expertise sufficient to convince the reviewers or panel, without needing to identify yourself personally.
Q12: Do proposals for Large Programs also need to follow the dual-anonymous guidelines?
Yes, the Scientific Justification (which includes a Management Plan) and Technical Justification of Large Programs must be fully anonymized in the same way as all proposals. In addition, Large Programs are required to include a one-page PDF file (the Team Expertise) that describes the team's expertise and background. The document should include the names of at least the key members of the proposal team.
Large Programs will be ranked based only on the Scientific Justification and Technical Justification. After the Large Programs have been scientifically ranked by the ALMA Proposal Review Committee (APRC), the APRC will review the Team Expertise. The APRC will be permitted to note any concerns on the qualifications of the team to carry out the proposed science. The concerns will be forwarded to the ALMA Director for consideration. However, the scientific ranking of the Large Programs will remain the same. Further information of the review process for Large Programs can be found in the Proposer's Guide.
Q13: Where can I find more information about dual-anonymous peer review, as it has been applied and assessed in other proposal review processes for other facilities?
JWST, HST, plus other NASA science missions, and ESO, have already implemented dual-anonymous in their call for proposals. You can find more information, including additional links and references, in the following:
NASA → https://science.nasa.gov/researchers/dual-anonymous-peer-review
ESO → https://www.eso.org/sci/observing/phase1/dual-anonymous-guidelines.html
Distributed peer review
Q1: What is distributed peer review?
Distributed peer review is a process in which one member of the proposal team, either the PI or a co-I, commits to review ten other submitted proposals. Therefore, the best proposals are selected by peer review, but instead of having a small committee review a large number of proposals, a larger number of people each review a smaller number of proposals. In this manner, the review load is distributed among many peers, and any individual reviewer will have a lower workload, with more time available to review each assigned proposal.
Q2: Are other observatories using distributed peer review?
Yes: Gemini’s Fast Turnaround proposal system uses distributed peer review and ESO initiated a pilot program in 2018 for La Silla Paranal Observatory, which received a very positive response from the community, this led ESO to implement distributed peer review for subset of proposals in their regular call for proposals in 2022 (https://www.eso.org/sci/observing/phase1/distributed-peer-review.html).
Q3: What if I am the PI of several proposals that qualify for the distributed review process? Will I have to review proposals for each of my submitted proposals?
A PI must designate a reviewer for each proposal that will be reviewed through the distributed peer review process. The designated reviewer for each proposal can be the PI or a co-I, and that person commits to review ten other proposals. If the same person is the designated reviewer associated with more than one proposal, then that person will review ten proposals for each respective proposal. For example, if a reviewer is the designated reviewer on behalf of two proposals, then that person should expect to review twenty proposals.
Q4: Is there a maximum number of proposals on which I can be the designated reviewer?
Yes. The maximum number of proposals on which the same user can be the designated reviewer is FIVE. For more information, please check the basic rules of Distributed Peer Review.
Q5: Who can I designate to be the reviewer for my submitted proposal?
Any PI and most co-Is on the proposal can be designated as the reviewer. If the PI does not have a PhD at the time of proposal submission (e.g., a student), the PI can still be the reviewer, but a mentor (who must have a PhD) must be identified at the time of the proposal submission. A PI may designate a co-I as the reviewer as long as the co-I has a PhD in astronomy or a closely related field. The reviewer and if needed, the mentor, must be designated in the Observing Tool at the time of proposal submission.
Q6: Can a reviewer split the workload with other co-investigators?
No. Reviewers are committed to review all ten proposals assigned to the Proposal Set associated with the proposal on which they are the designated reviewer. The review of these assignments should be carried out by the same person, as this ensures uniform and consistent scientific ranking and review. If a PI submits many proposals, then for each proposal they can select one of their co-Is (with a PhD) to be the designated reviewer, and that person will be responsible for reviewing the respective Proposal Set. Thus, in this way the workload of reviewing many Proposal Sets does not have to rest with a single reviewer unless they choose to do so.
Q7: Can a student (without a PhD) be a designated reviewer?
A student can be the designated reviewer if they are the PI of the proposal. In this case, the student has to also designate a mentor, who must have a PhD in astronomy or a closely related field. The mentor must be identified in the Observing Tool (OT) when the proposal is submitted. The mentor does not have to be a co-I on the student’s proposal. A student co-I is not eligible to be a designated reviewer.
Q8: What is the role of a mentor?
A mentor provides guidance as needed to a student reviewer during the process. This includes providing advice to the student on the scientific merit of the proposals, and on writing the comments to the PI. The mentor must abide by the same confidentiality requirements as the student reviewer.
For more information about the role of a mentor please check the Guidelines for Mentors.
Q9: Who will see my proposal?
Each proposal will be sent to ten reviewers. In the case of a student reviewer, the corresponding mentor will also see the proposal.
Q10: What is the expertise level of the reviewers?
Each submitted proposal will receive ten reviews. Most reviewers will have a PhD, and student reviewers (without a PhD) will be required to designate a mentor with a PhD to assist with the reviews. Reviewers will preferentially be assigned proposals in the same category/ies as their expertise.
Q11: As a PI, is there anything else I should keep in mind when submitting my proposal?
PIs should ensure that the designated reviewer has an up-to-date email address in the Observing Tool (OT). If the email is not up-to-date, please ask the reviewer to update their ALMA user preferences through the ALMA Science Portal. The Proposal Handling Team will use email to communicate with your reviewer, so if the email address is out of date, there is a chance that your reviewer will not submit their reviews on time and your proposal will be rejected as a result. It is the responsibility of the PI and the designated reviewer to ensure the reviews are submitted on time.
Additionally, please encourage the reviewer to enter their fields of expertise and conflicts of interest in their ALMA Science Portal preferences. This information is important for the optimal assignment of the proposals that each person will review.
Q12: How will proposals be assigned to reviewers?
To the extent possible, the Proposal Handling Team (PHT) will assign proposals based on the expertise of the reviewer. To match proposal assignments to a reviewer's expertise, reviewers can specify their areas of expertise (which is highly recommended!) in the user preferences accessible on the ALMA Science Portal. If a reviewer does not specify their expertise, they will be assumed to be experts in the category/keyword(s) of their submitted proposal.
The PHT will first attempt to assign proposals in the same scientific category as the reviewer's expertise. In rare instances a reviewer may be assigned a proposal in a category outside their expertise.
Q13: Is lack of expertise a reason to declare a conflict in the review of a proposal? That is, if I am a total outsider to the field, can I declare a conflict when reviewing a proposal?
No, a lack of expertise with a given proposal scientific category and keyword is not a reason to declare a conflict. Reviewers are asked to rank their assigned proposals based on scientific merit. PIs should always try to make the science case clear even to non-experts.
To further assist reviewers who may not consider themselves expert on a given assignment, distributed peer review contains a "Stage 2" step where reviewers can view the comments (but not the ranks) of other reviewers and modify their own rankings and reviews if they choose. In this way, reviewers who are possibly less experienced in a given science area, or who may have missed important information, can take the comments of other reviewers into consideration in forming their final ranks and reviews.
Q14: As a reviewer, if I declare a conflict with one of my assignments, how long should it take to receive a new assignment in its place?
During the first week of the process, the Proposal Handling Team (PHT) will collect conflict declarations from all reviewers and perform one batch reassignment at the end of that period. The quality of the reassignments improves as more assignments become available to reassign, which is why the aim is to do this first reassignment all at once. Reviewers will not receive any replacement assignments until this first batch reassignment has been performed. After the first week, as reviewers assess their conflicts on the new assignments, smaller batch reassignments will be performed more often. It may take less than one working day and as long as a few working days to receive a replacement assignment in these cases. While the reassignments are done you can start to review those proposals for which you are not conflicted.
Q15: What should I do if I identify a new conflict of interest after I already declared them?
If a designated reviewer realizes they have a conflict on a proposal after submitting their conflict decisions, they should contact the Proposal Handling Team by opening an ALMA helpdesk ticket to the department called "Proposal Review Support".
Q16: How much time does it take to review a Proposal Set?
Reviewing a Proposal Set takes time. First the reviewer has to go through all assignments to identify possible conflicts of interest. Then, the reviewer has to read each proposal carefully, prepare the comment to the PI, and rank the proposals in the Proposal Set. Once all proposals have been reviewed and ranked, it is recommended to go through the comments and ranks in case something needs to be adjusted. Reviewers can anticipate spending one to two days reviewing a single Proposal Set. It is highly recommended for reviewers to start the review process soon after they receive their Proposal Sets.
Q17: What if I am participating in the distributed review process and I do not submit my ranks and reviews by the Stage 1 review deadline? If I submit the ranks for one Proposal Set but not the other, will both of my proposals be rejected, or only the one for which I failed to submit the ranks?
If a designated reviewer does not submit their reviews before the Stage 1 review deadline, the proposal for which they were identified as the designated reviewer will be rejected. If a user is the designated reviewer for more than one proposal, they will be assigned a distinct Proposal Set to review for each submitted proposal. If the reviewer submits ranks and reviews for one Proposal Set but not for another, only the submitted proposal(s) associated with unsubmitted ranks and reviews will be rejected.
Q18: Will my proposal really be rejected if I don't submit all my reviews before the Stage 1 deadline of the distributed review process?
Yes.
Q19: During Stage 2 will I have the option to interact with other reviewers?
No. During Stage 2 there will be no interaction among the reviewers, other than the ability to see other comments that were written in Stage 1. The purpose of this stage is to help reviewers identify any important issue that they may have overlooked initially. This can be done by reading the reviews written by other reviewers who have different experiences and points of view.
Q20: How should reviewers determine the rank for a given assignment? Is this a score or a rank?
The ranks submitted for a Proposal Set are relative ranks, not absolute scores. As a result, a reviewer should rank their assignments from 1 to 10. For example, the strongest proposal should be assigned rank=1, the second-best proposal rank=2, etc…, with the weakest proposal assigned rank=10. The ranks within a Proposal Set may not be duplicated.
Q21: What feedback will PIs receive on their submitted proposals?
After the proposals have been reviewed and the priority grades are assigned, PIs will be sent their priority grade as well as the proposal ranks and reviews, verbatim, from each designated reviewer. The reviewer names will remain anonymous.
Q22: What is the timeline and deadlines for the distributed peer review processes?
The schedule timeline for the distributed peer review process can be found here.
Q23: As a reviewer, how will I receive information about the review process?
Once the review process starts, all official communication between the Proposal Handling Team and the reviewer will be done through the reviewer's email address that is registered in their ALMA account. For smooth communication during the review process, it is crucial that the information in your ALMA account (email address, institution, etc.) is up to date. PIs and reviewers are urged to check that the emails from the Proposal Handling Team (PHT) with address ph1m_noreply@ALMA.cl are not in their SPAM folder. General questions about the process can be submitted to the ALMA helpdesk department called "Proposal Review Support".
Q24: What browsers are supported for the Reviewer Tool used in distributed peer review?
The Reviewer Tool works well with Chrome and Firefox, but we have identified some minor issues with Safari. If you use Safari to run the Reviewer Tool and encounter a technical issue, please try reloading the browser and/or try a different browser. If these workarounds do not solve your problem, please contact the Proposal Handling Team by opening an ALMA helpdesk ticket to the department called "Proposal Review Support".
Large Programs
Q1: How are Large Programs reviewed?
Large Programs will be reviewed by a single panel of experts, the ALMA Proposal Review Committee (APRC). In the first stage of the panel review process, all members of the APRC will individually score their assigned Large Program and will write a review. In the second stage, panelists will meet to discuss and re-score the Large Programs, and make a recommendation on which Large Programs to schedule. Panelists will also finalize the consensus reports based on the individual comments and the discussion during the meeting.
Additionally, to gain further expertise, Large Programs will also be reviewed by external reviewers. These reviewers will provide written reports on the Large Programs, which will be considered during the APRC meeting, but will not participate in the meeting.
Return to the main ALMA Proposal Review page