Frequently Asked Questions

1. What important changes have been made to the proposal review process for Cycle 8 in 2021?

There are three important changes to the proposal review process in Cycle 8.

i) ALMA will adopt a distributed peer review process to scientifically review a majority of proposals submitted to the Cycle 8 Call for Proposals, in which one member of the proposal team on each submitted proposal will be responsible for reviewing ten proposals. 

ii) Review panels will evaluate a subset of proposals, specifically Large Programs and proposals requesting more than 25 hours on the 12-m array.

iii) All Cycle 8 proposals will be reviewed using a dual anonymous procedure, in which the reviewers will not see the names of the investigators of any proposal that they review. 

 

2. What is distributed peer review?

Distributed peer review is a process in which one member of the proposal team, either the Principal Investigator (PI) or a co-Investigator (co-I), commits to review ten other submitted proposals. Therefore, the best proposals are selected by peer review, but instead of having a small committee review a large number of proposals, a larger number of people each review a smaller number of proposals. In this manner, the review load is distributed among many peers, and any individual reviewer will have a lower workload, with more time to spend reviewing each assigned proposal.  

 

3. Are other observatories using distributed peer review?

Yes: Gemini’s Fast Turnaround proposal system uses distributed peer review and ESO initiated a pilot program in 2018, which received a very positive response from the community.

 

4. Why is ALMA using distributed peer review instead of the traditional panels to review a subset of the proposals on the Main Call?

Distributed peer review proved to be a viable review system for ALMA and its community after the success of the pilot program for the Cycle 7 Supplemental Call in 2019. The advantages that this review system provides include an increased involvement of the community in the review process and a significant reduction of the workload for each reviewer.  Based on the success of the pilot program, feedback from the community who participated in the pilot program, and advice from the ALMA Science Advisory Committee, the ALMA Board approved using distributed peer review for the smaller proposals in Cycle 8 while using traditional panel reviews for the larger proposals.

 

5. How will the distributed peer review process work for the Cycle 8 Call for Proposals?

Proposals requesting less than 25 hours on the 12-m array and less than 150 hours on the 7-m array will be evaluated through distributed peer review.  At proposal submission, the PI of such proposals must designate someone from the proposal team to be the reviewer. In most cases the PI will be the designated reviewer, but the PI may also designate a co-I. After the proposal deadline, the Proposal Handling Team (PHT) at the Joint ALMA Observatory (JAO) will assign ten proposals to the designated reviewer of each proposal. If reviewers have a conflict of interest with any of their assigned proposals, they can request a replacement proposal through the Reviewer Tool. Then during Stage 1, the reviewer will rank the ten proposals (1-10) in order of scientific priority, and write a brief review for each proposal. If ranks and reviews are not submitted by the time of the Stage 1 review deadline, the proposal on which the reviewer is acting as the designated reviewer will be rejected. After the Stage 1 review deadline, each reviewer will have the option to participate in a second stage (Stage 2) of the review process where the anonymized comments from the other reviewers of the same proposals will be made available. Reviewers can modify their own ranks and comments if desired in Stage 2.  Once this second stage is completed, the ranks from all reviewers of each submitted proposal will be combined to produce a global ranked list of proposals, which will be used to produce the observing queue after merging with the results of the panel process.

 

6. Who can I designate to be the reviewer for my submitted proposal? As a PI, how do I designate a reviewer?

Any PI and most co-Is on the proposal can be designated as the reviewer. If the PI does not have a PhD at the time of proposal submission (e.g., a student), the PI can still be the reviewer, but a mentor (who must have a PhD) must be identified at the time of the proposal submission. A PI may designate a co-I as the reviewer as long as the co-I has a PhD in astronomy or a closely related field. The reviewer and if needed, the mentor, must be designated in the Observing Tool (OT) at the time of proposal submission and cannot be changed after the proposal deadline.

 

7. Can a student (without a PhD) be a designated reviewer?

A student can be the designated reviewer if they are the PI of the proposal and also designates a mentor, who must have a PhD in astronomy or a closely related field. The mentor must be identified when the proposal is submitted. The mentor does not have to be a co-I on the student’s proposal. A student co-I is not eligible to be a designated reviewer.

 

8. What is the role of a mentor?

A mentor should provide guidance as needed to a student reviewer during the process, but there are no formal requirements. The mentor must abide by the same confidentiality requirements as the student reviewer. 

 

9. As a PI, is there anything else I should keep in mind when submitting my proposal?

PIs should ensure that the designated reviewer has an up-to-date email address in the Observing Tool. If the email is not up-to-date, please ask the reviewer to update their ALMA user profile through the ALMA Science Portal. The JAO will use email to communicate with your reviewer, so if the email address is out of date, there is a chance that your reviewer will not submit their reviews on time and your proposal will be rejected as a result. It is the responsibility of the PI and the designated reviewer to ensure the reviews are submitted on time.

Additionally, please encourage the reviewer to enter their fields of expertise in their ALMA Science Portal profile. This information is extremely important for the optimal assignment of the proposals that each person will review. 

 

10. How will proposals be assigned to reviewers?

To the extent possible, the JAO will assign proposals based on the expertise of the reviewer. A reviewer can indicate their expertise in their user profile on the ALMA Science Portal, where they can specify the scientific category/keywords that best match their expertise. If a reviewer does not complete their user profile, the category/keywords of the submitted proposal will be used for the assignment. The JAO will also avoid potential conflicts of interest by checking for frequent collaborations on submitted proposals. However, because of conflicts and the limited number of proposals submitted in some category/keywords, the JAO may also assign proposals in similar category/keywords, and on rare occasions, a proposal from a different category may be assigned. Keyword/category matches can be made most of the time.

Due to the nature of the assignment algorithm, it is very important that all reviewers add their expertise information to their profile on the ALMA Science Portal and keep it up-to-date.  This will allow the JAO to optimize the proposal assignments to each reviewer. 

 

11. What if I am participating in the distributed review process and I do not submit my ranks and reviews by the Stage 1 review deadline? If I submit the ranks for one proposal set but not the other, will both of my proposals be rejected, or only the one for which I failed to submit the ranks?

If a designated reviewer does not submit their reviews before the Stage 1 review deadline (see question 5 for information about Stage 1), the proposal for which they were identified as the designated reviewer will be rejected. If a user is the designated reviewer for more than one proposal, they will be assigned a distinct proposal set to review for each submitted proposal. If the reviewer submits ranks and reviews for one proposal set but not for another, only the submitted proposal(s) associated with unsubmitted ranks and reviews will be rejected.

 

12. Will my proposal really be rejected if I don't submit all my reviews before the Stage 1 deadline of the distributed review process?

Yes. 

 

13. For proposals not included in the distributed peer review process, how will the panel review process work for the Cycle 8 Main Call?

All Large Programs, and any proposals requesting more than 25 hours on the 12-m array, will be peer reviewed by volunteers from the community serving on traditional review panels with topical expertise. The JAO will assign each submitted proposal to a panel based on the scientific category selected by the PI. 

In the first stage of the panel review process, panelists will individually score each non-conflicted proposal assigned to them and will write a brief review. In the second stage, panelists will meet (virtually, in Cycle 8) to discuss each proposal assigned to their panels. For Large Programs, the panels will vote to decide which Large Programs should advance to the ALMA Proposal Review Committee (APRC) for further review. For the remaining proposals (i.e., proposals requesting between 25 and 50 hours on the 12-m array), the panelists will re-score the proposals based on the panel discussion and the average score will be used to determine the proposal rank. Panelists will finalize the consensus reports based on the individual comments and the discussion during the meeting. The ranked lists from each panel will be merged, and later used to create the observing queue.

Following the ALMA Review Panel meetings, the ALMA Proposal Review Committee (APRC) will meet to review the Large Programs selected by the panels. The APRC will then re-score the Large Programs and make a recommendation on which Large Programs to schedule.

 

14. What if I am the PI of several proposals that qualify for the distributed review process? Will I have to review proposals for each of my submitted proposals?

A PI may submit several proposals that qualify for the distributed peer review process. For each of those proposals, a PI will need to designate a reviewer. If you have submitted several proposals as PI, you do not have to be the designated reviewer on all (or any) of them. The designated reviewer for each proposal can be the PI or a co-I, and that person commits to review ten other proposals. If the same person is the designated reviewer associated with more than one proposal, then that person will review ten proposals for each respective proposal. For example, if a reviewer is the designated reviewer on behalf of 3 proposals, then that person should expect to review 30 other proposals.

In addition, the same PI may submit proposals requesting more than 25 hours of time on the 12-m array, and/or Large proposals, which will instead be reviewed by panels.

  

15. I submitted a proposal that requests more than 25 hours of time on the 12-m array, but is not a Large Program.  Do I still have to designate a reviewer?

Yes. The Observing Tool will automatically require a reviewer to be designated for any proposal that is not a Large proposal. For all proposals that request between 25 to 50 hours on the 12-m array, the PI is required to designate a reviewer, and the reviewer will not need to review proposals as long as the final time request confirmed by the JAO is more than 25 h on the 12-m array.

 

16. What is dual anonymous peer review, and why is it being implemented?  

ALMA is strongly committed to ensure that the proposal review process is as fair and impartial as possible. Analysis of the proposal rankings in previous cycles has identified systematics that may signify the presence of biases in the review process (Carpenter 2020). To ensure that the proposal review process is as fair and unbiased as possible, ALMA is beginning dual-anonymous review in Cycle 8. In a dual-anonymous review, the proposal team does not know the identity of the reviewers and the reviewers do not know the identity of the proposal team. While proposers will still enter their names and affiliations in the ALMA Observing Tool (OT), this information will not appear on the proposal cover sheet, nor in the tools used by the reviewers. It is the responsibility of the proposers to ensure anonymity is preserved when writing their proposals. Guidelines on preparing proposals will appear in the Cycle 8 Call for Proposals.

Large Programs have special procedures that will allow the APRC to see the identity of the proposal team, but only after the scientific review, rankings, and recommendations have been completed. Details on the review procedures for Large Programs are described in the Call for Proposals. 

 

17. Will ALMA provide guidelines about how to “anonymize” my proposal(s) for dual anonymous?

Yes! More details will be made available to PIs in the Call for Proposals.

 

18. What is the timeline and deadlines for the panel and distributed peer review processes? 

The schedule for the distributed peer review process will be outlined in detail in the Cycle 8 Call for Proposals, and the schedule for the panel review is being communicated with the selected panelists.  Broadly, the various stages of the review processes will take place in May and June, with results communicated with PIs by early August. This is all in preparation for the beginning of the Cycle 8 observations in October.

 

19. I have a question about the process that is not covered here. What should I do? 

Contact the Helpdesk and specify the Project Planning department. Maybe your question will end up on the FAQ!