How to use the Reviewer Tool

Proposal Review Table of Contents
ALMA Proposal Review
Dual-anonymous Guidelines
Distributed Peer Review
Guidelines for Reviewers
How to use the Reviewer Tool
Reviewer Tool
Frequently Asked Questions

These guidelines describe the usage of the ALMA Reviewer Tool from a reviewer’s and mentor’s perspective. The Reviewer Tool is required for the execution of proposal review tasks during the Distributed Peer Review process. Best practices to follow with regard to various reviewer responsibilities can be found at the following links on the ALMA Science Portal.

Reviewers should please read these pages before starting to compose reviews.

The ALMA Reviewer Tool will allow reviewers to:

Acknowledge your agreement to keep materials confidential, to behave in an ethical manner and rank proposals based purely on their scientific merit, and to submit your conflicts and ranks/reviews by the deadlines.
Identify and submit conflicts of interest for all review assignments that have been assigned to you.
- Each proposal for which you will serve as a reviewer is associated with a set of review assignments, which is called a Proposal Set. The Reviewer Tool requires the conflict decisions to be made on all Proposal Sets before any can be submitted.
Enter a rank and review for each of your review assignments during both Stage 1 and Stage 2 of the process.
- The Reviewer Tool will allow you to submit your ranks and reviews for each Proposal Set individually.
- Stage 1 is the initial review stage. During Stage 2, you can read anonymized reviews written by other reviewers of the same proposals and adjust your own ranks and reviews if desired.
- Stage 1 is mandatory, but Stage 2 is not mandatory.

Please note that running the Reviewer Tool requires you to be connected to the internet, and it employs an auto-saving function at all steps of the process to save drafts of your entries to the ALMA database. However, it is still necessary to click the “submit” button at various stages to record your final entries.

If you need help at any point in the review process, please contact at the Proposal Handling Team (open an ALMA helpdesk ticket to the department “Proposal Review Support”) at the Joint ALMA Observatory (JAO).

Step 1: Acknowledge the process.

The PHT will announce the beginning of the distributed peer review process to all reviewers via email, including a link to the Reviewer Tool, using the email address associated with each reviewer's ALMA account in the User Database. (If this address is not up to date in the ALMA User Database, please update your ALMA user profile information.) When you first log into the Reviewer Tool, you will be required to acknowledge four statements regarding the confidentiality of the process, your commitment to reviewing and ranking proposals based purely on their scientific merit, and the deadlines for conflict and rank/review submission. You will be able to do so using the following screen:

Once you click “Accept”, you will not be shown this screen again.

Step 2: Check your Proposal Sets.

As a reviewer, you will be notified by email (using the email address associated with your ALMA account in the User Database) with a list of the proposals on whose behalf you are expected to serve as a reviewer after the proposal deadline passes.

Each proposal for which you will serve as a reviewer is associated with a set of review assignments, which is called a Proposal Set. Your “home”, or landing, page in the Reviewer Tool contains a link to each of your Proposal Sets, which are labelled according to your submitted proposals. If you are serving as a reviewer on behalf of one submitted proposal, you will have one Proposal Set in your landing screen. If you are a reviewer for multiple submitted proposals, or you are also acting as a mentor, multiple Proposal Sets will be shown.

If you don’t see a Proposal Set for each submitted proposal for which you are serving as a reviewer or as a mentor, please contact the PHT by submitting an ALMA helpdesk ticket to the department “Proposal Review Support”.

The following are also displayed on the landing screen:

a countdown reminder of the time until the final review submission deadline,
a tool tip containing instructions for your next step, and
the status of each Proposal Set.

Further, you will see a “Submit conflict decisions” button at the top right corner of the landing screen, which will not be activated until you indicate all of your conflict decisions in Step 3.

Step 3: Indicate your conflicts.

In this step, you should determine the conflicts of interest that you may have with your review assignments. Please consult the conflict criteria to help you determine whether you are conflicted with any of your assignments. Click on the link to your Proposal Set, or one of your Proposal Sets (if you have more than one), to get started. This step should be completed by 15 May 2024, 15:00 UT in order to ensure timely reassignments of the conflicted proposals for all reviewers. Even if you do not detect any conflicts, please submit all of your conflict decisions on time.

The Reviewer Tool will display the review assignments associated with that Proposal Set, each of which will have a “Pending” status. The proposal code and title will be displayed for each review assignment.

Step 3.1: Finding more information about each assignment.

You can expand the information shown for any given assignment by clicking the “Details” button. Now you can see the abstract, scientific category and keywords, and requested observing time on the 12-m, 7-m, and Total Power (TP) Arrays.

Reviewers only have access to the abstract of their assignments at this stage of the process. Reviewers will gain access to the full PDF files after they submit their conflict decisions, and only for those assignments that were declared as free of conflict.

Step 3.2: Marking each assignment with your conflict decision.

For each assignment, if you are not conflicted, click the green “Accept” button in the same row.

If you feel that you are conflicted, click the red “Conflict” button in the same row; you will be prompted to select the nature of your conflict from a drop-down menu. A text window is available to provide a mandatory explanation of your conflict. Conflicts of interest relating to the (anonymized) author list have already been handled by the assignment algorithm.

When you make your conflict selection, the “pending” label of that assignment will change to “free of conflict” or “conflicted”. For conflicted proposals, you can access your reasoning by expanding the assignment information with the “Details” button and checking the “Assessment” tab.

Step3.2-fig.png

If you change your mind about your conflict decision, you can select the other option at any time as long as you have not yet submitted your conflicts.

Step 3.3: Submitting your conflict decisions.

Your conflict decisions are auto-saved while you’re working (as long as you are connected to the internet), but the decisions are not considered final until you click “Submit conflict decisions”. The auto-saving happens every few seconds, but if you make a selection and navigate away before the auto-saving takes effect, your selection will be lost. If you only have one Proposal Set, as soon as every assignment has a selected decision (“Accept” or “Conflict”) and all conflicts have a reason, this button will be enabled (in dark blue) above the list of your review assignments:

If you have more than one Proposal Set, the Reviewer Tool will prompt you to return to the landing screen to select your other Proposal Set and enter its corresponding conflict decisions. All conflict decisions on ALL Proposal Sets must be made before any conflict decisions can be submitted.

Now the landing screen indicates that the conflict decisions have been made for one Proposal Set, but the other is still pending:

As a result, the “Submit conflict decisions” button is still not enabled.

Repeat the conflict declaration process for the second (and any other) Proposal Sets. Once all conflict decisions on all Proposal Sets are made, submit your conflict decisions using the button on the landing screen or above any individual Proposal Set assignment list.

Step 4: Review your assignments.

Once your conflict decisions have been submitted, you have entered the Stage 1 “Reviewing” stage of the process. The status “Reviewing” of your Proposal Set(s) on the landing page will reflect this stage. If you submitted any conflicts, those assignments are identified with the “Conflicted Pending Decision” status on the landing page and within your Proposal Set page.

Within each Proposal Set assignment list, the unconflicted assignments will now be shown as “Ready to review”. Conflicted assignments will retain the status “Conflicted pending decision” until the PHT approves or rejects the submitted conflicts and, if necessary, assigns a replacement assignment to your Proposal Set. All approved conflicts will result in a replacement assignment; you will always have to accept 10 review assignments per Proposal Set.

Be aware that you should expect a delay between the time that you submit your conflicts and when you receive your new assignments. Running the reassignment algorithm requires other reviewers to reject assignments as well, and the quality of the reassignments improves as more assignments become available to reassign. Further, you may not be notified at the same time about conflicts that the PHT accepts and those that the PHT rejects; since accepted conflicts require a replacement proposal before you receive a notification, the delay may be longer before you learn of these decisions.
During the first week of Stage 1, the PHT will collect conflict declarations from all reviewers and perform one batch reassignment at the end of that week. The quality of the reassignments improves as more assignments become available to reassign, which is why we aim to do this first reassignment all at once. You will not receive any replacement assignments until this first batch reassignment has been performed. After the first week, as reviewers assess their conflicts on the new assignments, smaller batch reassignments will be performed more often. It may take less than a day and as long as a few days to receive a replacement assignment in these cases.

An individual Proposal Set assignment list will appear as below.

A button called “Download proposals” will appear in the top right corner of your Reviewer Tool. Clicking on this button will allow you to download a package containing all PDFs for the non-conflicted assignments in your Proposal Set. Please be patient, as the download can take a few minutes.

There is also a button in this expanded information section to allow you to download the proposal PDF file. Your browser settings can be set to a default which will allow the PDF to be opened in a new tab.

Step 4.1: Assignments with no conflicts.

For assignments with the status “Ready to review”, please proceed to review the proposal. Please see Step 4.3 for a description of this process.

Step 4.2: Assignments with conflicts.

If you have declared conflicts in your Proposal Set, they will be shown immediately upon submission of your conflicts as “Conflicted pending decision”, as described above. The PHT will handle your declared conflicts in one of two ways:

If the declared conflict is deemed to be sufficiently minor that it should not keep the reviewer from performing their review, the declared conflict will be rejected by the PHT. In this case, you will receive an email indicating that the assignment is ready to review. In the Reviewer Tool, the status of the assignment will change from “Conflicted pending decision” to “Ready to review”.
If the declared conflict is deemed to be sufficiently major, the declared conflict will be accepted by the PHT. The status of the assignment will remain in the state “Conflicted pending decision” until a replacement assignment is available to the reviewer. The PHT will collect conflicted assignments from all reviewers and periodically shuffle them among reviewers needing new assignments. Once your new assignment is available, it will show up in your Proposal Set list with the status “Pending”, as in the beginning of Step 3. You will receive an email indicating that you should declare conflicts for your new assignments just as you did in Step 3. It is important to review your assignments and submit your conflicts as quickly and early in the process as possible in order for the PHT to ensure that all reviewers have complete assignment lists.

Once your conflicts have been resolved, either by a conflict rejection by the PHT or with the acceptance of a new replacement assignment, please proceed to review the proposal as described in Step 4.3.

If you begin to review your assignments before all of your “Conflicted pending decision” assignments have their conflicts resolved or reassigned, keep in mind that your final rankings cannot be submitted until you have accepted (and reviewed) a complete assignment list.

Step 4.3: Entering your ranks and reviews.

To add reviews for those assignments with “Ready to review” status, click “Add Review” for the desired assignment. A pop-up window will open where you can record your review (“Comments to the PI”), as well as an optional field for any comments to the PHT. Next to the “Assessment” tab is the “Proposal information” tab, which displays more specific information about the proposal.

Two text fields are available to record your Comments to the PI and Comments to the JAO.

Please consult the review criteria described on the ALMA Science Portal before reviewing your proposals.
Guidelines to help you review your assignments and compose your Comments to the PI are available on the ALMA Science Portal.
Keep in mind that the PHT will not edit your comments before they are sent to the PI, so it is critical to keep your comments professional and courteous.
The red ! character will appear next to the text field where you enter your review if the comment is too short (less than 200 characters).

The field Comments to the JAO is optional and is only needed if private communication with the PHT is required. This field can be used to flag proposals with possible violations to the dual-anonymous guidelines, or with possible violations to the PDF format and/or minimum font size, or to present any concern about the technical observational setup, or other items about the proposal that you consider that should be addressed by the PHT. If you need advice during the distributed review process about how to handle a technical question, please contact the PHT by submitting an ALMA helpdesk ticket to the department “Proposal Review Support”.

If you select one of the four check boxes under “Comments to the JAO”, then you must provide text in the comment box. The red ! character will appear if this is not done.

Your ranks and review texts are auto-saved while you’re working (as long as you are connected to the internet), but they are not considered final until you click “Submit reviews”. The auto-saving happens every few seconds, but if you enter text and navigate away before the auto-saving takes effect, your text will be lost.

Once a review longer than the 200 characters minimum has been added to an assignment, the yellow “Add Review” button becomes a blue “Modify Review” button.

In addition to the review, you must also provide a rank for each proposal by dragging and dropping each assignment from the initial “Assignment list” at the bottom of the Proposal Set page into the “Ranked list” at the top of the Proposal Set page. Ranks are expressed as “N of X”, where N is the Reviewer’s rank of this assignment out of X assignments in this Proposal Set.

For instance, if you judge this assignment to be the strongest proposal in this Proposal Set, and your Proposal Set has 10 assignments, the rank would be 1 of 10. Even if you have multiple Proposal Sets, it is necessary to rank the assignments relative to one another in a given Proposal Set.
Note that this is a relative rank, not an absolute score. Before you can submit the ranks and reviews for your Proposal Set, you must have an assignment with rank “1”, an assignment with rank “2”, an assignment with rank “3”, and so on up to X, the number of assignments in your Proposal Set.
Assignments in “Conflicted pending decision” cannot be dragged and dropped or have review comments added, and their PDF files are not available for download.

As long as all assignments have a rank and review, it is possible to review your comments to all assigned proposals in a single view by clicking “Proceed to final submission page”, as described in Step 5. Some reviewers may find this view helpful during the review process even before they are ready to actually submit a Proposal Set since it is possible to still make changes before the final submission.

Step 5: Submit your ranks and reviews.

Once you have completed your reviews and ranks for a given Proposal Set, you should proceed to the final submission page. The Reviewer Tool will allow you to submit your ranks and reviews one Proposal Set at a time. In any given Proposal Set page, the button to “Proceed to final submission page” will be enabled once all assignments in that Proposal Set have a review of the appropriate length (i.e., more than 200 and less than 4000 characters) and a valid rank in the “Ranked list”.

Upon clicking “Proceed to final submission page”, you will be taken to a final confirmation screen where you can review the text of all of your reviews one last time. The list is ordered per your ranking for each assignment. Each row can be expanded by clicking the “+” button or minimized by clicking the “-” button, which is useful for reviewing your comments all in one place.

If you would like to make further edits, please click “Back” to make changes. Otherwise, click “Confirm final submission of ranks and reviews” to submit your final reviews for this Proposal Set. You may not make any changes to your ranks and reviews for this Proposal Set after this point.

If you have another Proposal Set, navigate back to your landing page to complete the reviews for your other Proposal Set(s). You can tell on the landing page which Proposal Sets have had all of their reviews completed by checking the status indicators on that page.

Section 6: Fill out the Reviewer Survey.

Upon submission of each Proposal Set, you will be directed to a Reviewer Survey meant to capture your comfort level with each of your assignments. Your thoughts are crucial to improving the tool and gauging the success of the distributed review process. The survey will be made available beginning with the submission of the Stage 1 ranks and reviews and should be submitted separately for each Proposal Set.

You can submit your answers to the survey right after you submit your ranks and reviews for a Proposal Set, or later in the process. In your home page, at the top left corner of each one of your Proposal Sets, you will be able to see the submission status of your survey. Before the answers to the survey are submitted you will see the status “Survey – Not done”, and once you submit your answers you will see the status “Survey – Done”.

Section 7: Participate in Stage 2.

After the Stage 1 deadline, the PHT will close the submission server such that ranks and reviews can no longer be edited, but they are still available to be viewed.

Once Stage 2 is available, reviewers who submitted their Stage 1 ranks and reviews on time can read the anonymous reviews (only the comments, not the ranks) of the other reviewers who were assigned the same proposals. Reviewers can change their own ranks and comments if desired and resubmit their Proposal Set by the Stage 2 deadline.

Participation in Stage 2 is optional, and strongly encouraged. If a reviewer decides not to participate in Stage 2, their Stage 1 ranks will be used in the creation of the final ranked list, and their Stage 1 comments to PI will be sent to the PI. The Stage 1 ranks and reviews will also be used if a reviewer starts the Stage 2 process but does not submit any new ranks and reviews before the Stage 2 deadline.

Reviewers will be notified by email when Stage 2 is available. To participate in Stage 2, access the Reviewer Tool and click “Accept” on the pop-up window:

From this point, the tool functions in a similar way to Stage 1.

The Proposal Set landing page indicates that the Proposal Set status is “Stage 2 Review”.

The Proposal Set is presented in submitted Stage 1 rank order. Anonymized reviews written by other reviewers of the same proposals (who also submitted their ranks and reviews before the Stage 1 deadline) can be viewed by clicking “Modify Review” and selecting the “Anonymized Reviews” tab.

The reviewer’s submitted Stage 1 review text is available, and editable, in the “Assessment” tab. At this point, you can make edits to the text of your reviews and drag and drop any assignments for which you wish to change your rank and/or review.

If during Stage 2 you modify one of your Stage 1 reviews, an orange pencil icon and a button to display the original Stage 1 review will appear in the corresponding Assessment tab. The orange pencil icon will also appear in your assignment list, next to the “Modify Review” button of the corresponding assignment. If you modify the Stage 1 rank of any of your assignments, a green or red arrow with a tooltip indicating the Stage 1 rank will appear next to the corresponding rank to indicate if that assignment moved up or down in the ranked list.

Once you have completed your ranking and reviews, you can proceed to the final submission page for a final look at your list. If during Stage 2 you applied any modifications to your reviews and/or ranks, you will be able to see the corresponding visual aid in this page too.

Please make sure that you have responded to the Reviewer Survey for each of your Proposal Sets.

Section 8: Mentor’s view.

Mentors can use the Reviewer Tool to access their mentees’ Proposal Sets in read-only mode. The mentees’ Proposal Set will be listed under the header “Mentored Review Sets”.

Mentors can use the Reviewer Tool to access the abstract of the review assignments of their mentees so the mentor can identify possible conflicts of interest. If any conflict is identified, it needs to be declared by the corresponding mentee through the Reviewer Tool. After conflict declaration, mentors can use the Reviewer Tool to access the full PDF files of the review assignments that were declared free of conflict by their mentees, and from then on, mentors can use the tool to access the review information (comments to PI, comment to JAO, rankings) provided by their mentees.

Return to the main ALMA Proposal Review page

Science Highlight