How to Use the Reviewer Tool
|
Table of Contents |
These guidelines describe the usage of the ALMA Reviewer Tool from a Reviewer’s perspective. The Reviewer Tool is required for the execution of proposal review tasks during the Distributed Peer Review process being followed in the Cycle 7 Supplemental Call. Descriptions of best practices to follow with regard to various parts of the review process can be found at the following links on the ALMA Science Portal. Reviewers should please read these pages before starting to compose reviews.
The ALMA Reviewer Tool will allow you to:
-
Acknowledge your agreement to keep materials confidential, to behave in an ethical manner and rank proposals based purely on their scientific merit, and to submit your conflicts and ranks/reviews by the deadlines (15:00 UT on October 22 for conflicts, and 15:00 UT on November 12 for the ranks/reviews).
-
Identify and submit conflicts of interest for all review assignments that have been assigned to you.
-
Each proposal for which you will serve as a Reviewer is associated with a set of review assignments, which is called a Proposal Set. The Reviewer Tool requires the conflict decisions to be made on all Proposal Sets before any can be submitted.
-
Enter a rank and review for each of your review assignments.
-
Each proposal for which you will serve as a Reviewer is associated with a set of review assignments, which is called a Proposal Set. The Reviewer Tool will allow you to submit your ranks and reviews for each Proposal Set individually.
Please note that running the Reviewer Tool requires you to be connected to the internet, and it employs an auto-saving function at all steps of the process to save drafts of your entries to the ALMA database. However, it is still necessary to click the “submit” button at various stages to record your final entries.
Step 1: Acknowledge the process.
The JAO will announce the beginning of the distributed peer review process to all Reviewers via email, including a link to the Reviewer Tool, using the email address associated with each Reviewer's ALMA account in the User Database. (If this address is not up to date in the ALMA User Database, please update your ALMA user profile information.) When you first log into the Reviewer Tool, you will be required to acknowledge four statements regarding the confidentiality of the process, your commitment to reviewing and ranking proposals based purely on their scientific merit, and the deadlines for conflict and rank/review submission. You will be able to do so using the following screen:
Once you click “Accept”, you will not be shown this screen again.
Step 2: Check your Proposal Sets.
As a Reviewer, you will be notified by email (using the email address associated with your ALMA account in the User Database) with a list of the proposals on whose behalf you are expected serve as a Reviewer after the proposal deadline passes.
Each proposal for which you will serve as a Reviewer is associated with a set of review assignments, which is called a Proposal Set. Your “home”, or landing, page in the Reviewer Tool should contain a link to each of your Proposal Sets, which are labelled according to your submitted proposals. If you are serving as a Reviewer on behalf of one submitted proposal, you will have one Proposal Set in your landing screen. If you are a Reviewer for multiple submitted proposals, multiple Proposal Sets will be shown.
If you don’t see a Proposal Set for each submitted proposal for which you are serving as a Reviewer, please contact pht@alma.cl.
The following are also displayed on the landing screen:
-
a countdown reminder of the time until the final review submission deadline,
-
a tool tip containing instructions for your next step, and
-
the status of each Proposal Set.
Further, you will see a “Submit conflict decisions” button at the top right corner of the landing screen, which will not be activated until you indicate all of your conflict decisions in Step 3.
Step 3: Indicate your conflicts.
In this step, you should determine the conflicts of interest that you may have with your review assignments. Please consult the criteria described here to help you determine whether you are conflicted with any of your assignments. Click on the link to your Proposal Set, or one of your Proposal Sets (if you have more than one), to get started. In order to ensure timely assignments to all Reviewers, this step must be completed by 15:00 UT on October 22, 2019. Even if you do not detect any conflicts, you must submit all of your conflict decisions by the deadline.
The Reviewer Tool will display the review assignments associated with that Proposal Set, each of which should have a “Pending” status. The proposal code, title, receiver band(s), and array(s) will be displayed for each review assignment.
Step 3.1: Finding more information about each assignment.
You can expand the information shown for any given assignment by clicking the “+”. Now you can see the abstract, scientific category and keywords, and requested observing time on the 7m and TP arrays.
There is also a button in this expanded information section to allow you to download the proposal PDF file, where the author list can be found.
-
In general, conflicts of interest relating to the author list have already been handled by the assignment algorithm, and declared conflicts of this nature will only be considered if the Reviewer has a close collaborator on the proposal team.
Step 3.2: Marking each assignment with your conflict decision.
For each assignment, if you are not conflicted, click the green “Accept” button in the same row.
If you feel that you are conflicted, click the purple “Conflict” button in the same row; you will be prompted to select the nature of your conflict from a drop-down menu. A text window is available if more explanation is required, and this information is optional in most cases. If you select “I am part of a competing proposal / project / team” or “Other”, an explanation in the text field is mandatory.
When you make your conflict selection, the “pending” label of that assignment will change to “free of conflict” or “conflicted”. For conflicted proposals, you can access your reasoning by expanding the assignment information with the “+” button and checking the “assessment” tab.
If you change your mind about your conflict decision, you can select the other option at any time as long as you have not yet submitted your conflicts.
Step 3.3: Submitting your conflict decisions.
Your conflict decisions are auto-saved while you’re working (as long as you are connected to the internet), but the decisions are not considered final until you click “Submit conflict decisions”. The auto-saving happens every few seconds, but if you make a selection and navigate away before the auto-saving takes effect, your selection will be lost. If you only have one Proposal Set, as soon as every assignment has a selected decision (“Accept” or “Conflict”) and all conflicts have a reason, this button will be enabled above the list of your review assignments:
If you have more than one Proposal Set, the Reviewer Tool will prompt you to return to the landing screen to select your other Proposal Set and enter its corresponding conflict decisions. Please note that all conflict decisions on ALL Proposal Sets must be made before any conflict decisions can be submitted.
Now the landing screen indicates that the conflict decisions have been made for one Proposal Set, but the other is still pending:
As a result, the “Submit conflict decisions” button is still not enabled.
Repeat the conflict declaration process for the second (and any other) Proposal Sets. Once all conflict decisions on all Proposal Sets are made, submit your conflict decisions using the button on the landing screen or above any individual Proposal Set assignment list.
Step 4: Review your assignments.
Once your conflict decisions have been submitted, you have entered the “Reviewing” stage of the process. The status of your Proposal Set(s) on the landing page will reflect this stage:
Within each Proposal Set assignment list, the unconflicted assignments will now be shown as “Ready to review”. Conflicted assignments will retain the status “Conflicted pending decision” until the JAO approves or rejects the submitted conflicts and, if necessary, assigns a replacement assignment to your Proposal Set. All approved conflicts will result in a replacement assignment; you will always have to accept 10 review assignments per Proposal Set.
An individual Proposal Set assignment list will appear as below. Note that the tool tip indicates the ranks that are currently possible to assign; in the example below, ranks 1-8 can be assigned while two assignments are "Conflicted pending decision". Once these two declared conflicts have been approved or rejected, there will be two assignments in their place, and the tool tip will indicate that the ranks possible to assign are now 1-10.
Be aware that you should expect a delay between the time that you submit your conflicts and when you receive your new assignments. Running the reassignment algorithm requires other reviewers to reject their assignments as well, and the quality of the reassignments improves as more assignments become available to reassign. Further, you may not be notified at the same time about conflicts that the JAO accepts and those that the JAO rejects; since accepted conflicts require a replacement proposal before you receive a notification, the delay may be longer before you learn of these decisions.
During the first week of the process (October 15-22), the JAO will collect conflict declarations from all reviewers and perform one batch reassignment at the end of that week. The quality of the reassignments improves as more assignments become available to reassign, which is why we aim to do this first reassignment all at once. You will not receive any replacement assignments until this first batch reassignment has been performed. After October 22, as reviewers assess their conflicts on the new assignments, smaller batch reassignments will be performed more often. It may take less than a day and as long as a few days to receive a replacement assignment in these cases.
Step 4.1: Assignments with no conflicts.
For assignments with the status “Ready to review”, please proceed to reviewing the proposal. Please see Step 4.3 for a description of this process.
Step 4.2: Assignments with conflicts.
If you have declared conflicts in your Proposal Set, they will be shown immediately upon submission of your conflicts as “Conflicted pending decision”, as described above. The JAO will handle your declared conflicts in one of two ways:
-
If the declared conflict is deemed to be sufficiently minor that it should not keep the Reviewer from performing their review, the declared conflict will be rejected. In this case, you will receive an email indicating that the assignment is ready to review. In the Reviewer Tool, the status of the assignment will change from “Conflicted pending decision” to “Ready to review”.
-
If the declared conflict is deemed to be sufficiently major, the assignment will be rejected. The status of the assignment will remain in the state “Conflicted pending decision” until a replacement assignment is available to the Reviewer. The JAO will collect conflicted assignments from all Reviewers and periodically shuffle them among Reviewers needing new assignments. Once your new assignment is available, it will show up in your Proposal Set list with the status “Pending”, as shown below. You will receive an email indicating that you should declare conflicts for your new assignments just as you did in Step 3. It is important to review your assignments and submit your conflicts as quickly and early in the process as possible in order for the JAO to ensure that all Reviewers have complete assignment lists.
Once your conflicts have been resolved, either by a conflict rejection by the JAO or with the acceptance of a new replacement assignment, please proceed to reviewing the proposal. Please see Step 4.3 for a description of this part of the process.
If you begin to review your assignments before all of your “Conflicted pending decision” assignments have their conflicts resolved or reassigned, keep in mind that your final rankings cannot be submitted until you have accepted (and reviewed) a complete assignment list.
Step 4.3: Entering your ranks and reviews.
To add reviews for those assignments with “Ready to review” status, just expand the view of the desired assignment. You will now see that the expanded view has a new tab called “Assessment”, where you can record your rank and review (“Comments to the PI”), as well as an optional field for any comments to the JAO. Next to the “Assessment” tab is the “Proposal information” tab, which displays more specific information about the proposal (as displayed during Step 3).
As stated above, the “Assessment” tab should be used to record your rank, expressed as “N of X”, where N is the Reviewer’s rank of this assignment out of X assignments in this Proposal Set.
-
For instance, if you judge this assignment to be the strongest proposal in this Proposal Set, and your Proposal Set has 10 assignments, the rank would be 1 of 10. Even if you have multiple Proposal Sets, it is necessary to rank the assignments relative to one another in a given Proposal Set.
- Note that the assessment panel indicates the number of assignments (X) that are currently possible to assign; in the example above, ranks 1-8 can be assigned while two assignments are "Conflicted pending decision". Once these two declared conflicts have been approved or rejected, there will be two assignments in their place, and the number of assignments will indicate that the ranks possible to assign are now 1-10.
-
Note that this is a relative rank, not an absolute score. Before you can submit the ranks and reviews for your Proposal Set, you must have an assignment with rank “1”, an assignment with rank “2”, an assignment with rank “3”, and so on up to X, the number of assignments in your Proposal Set.
-
Note that the red ! character next to the text field where you should enter your rank will be displayed if the field is empty or if you enter a non-integer character.
Two text fields are available to record your Comments to the PI and Comments to the JAO.
-
Please consult the criteria described here and here to help you review your assignments and compose your Comments to the PI.
-
Keep in mind that the JAO will not edit your comments before they are sent to the PI, so it is critical to keep your comments professional and courteous.
-
Note that the red ! character next to the text field where you should enter your review will be displayed if the comment is too short (less than 10 characters) or too long (more than 4000 characters).
The Comments to the JAO field is optional and only needed if you have technical concerns about the proposal or other private communication required. If you need advice during the distributed review process about how to handle a technical question, please contact pht@alma.cl.
Your ranks and review texts are auto-saved while you’re working (as long as you are connected to the internet), but they are not considered final until you click “Submit reviews”. Note that the auto-saving happens every few seconds, but if you enter text and navigate away before the auto-saving takes effect, your text will be lost.
Step 5: Submit your ranks and reviews.
Once you have completed your reviews and ranks for a given Proposal Set, you should submit them. The Reviewer Tool will allow you to submit your ranks and reviews one Proposal Set at a time. In any given Proposal Set page, the button to “Submit reviews” will be enabled once all assignments in that Proposal Set have a review of the appropriate length (i.e., more than 10 and less than 4000 characters) and a valid rank.
In this example, there is a duplicate rank in the list. In such a case, the tool will not validate until you fix the error, which is identified with by red ! marks. Once all ranks are unique for an individual Proposal Set, you will be able to submit your ranks and reviews for that Proposal Set.
Upon clicking “Submit reviews”, you will be taken to a final confirmation screen where you can review the text of all of your reviews one last time. The list is ordered per your ranking for each assignment.
If you would like to make further edits, please click “Back” to make changes. Otherwise, click “Confirm final submission of ranks and reviews” to submit your final reviews for this Proposal Set. You may not make any changes to your ranks and reviews for this Proposal Set after this point.
If you have another Proposal Set, navigate back to your landing page to complete the Reviews for your other Proposal Set(s). You can tell on the landing page which Proposal Sets have had all of their reviews completed by checking the status indicators on that page.
Section 6: Fill out the Reviewer Survey.
Upon submission of your final Proposal Set, you will be directed to a Reviewer Survey meant to capture your feedback on the Reviewer Tool and Distributed Peer Review process. Your thoughts are crucial to improving the tool and gauging the success of the Supplemental Call review process. Please take a few minutes to respond. Thank you!