Frequently Asked Questions
General
Q1: What important changes have been made to the proposal review process for Cycle 8 2021?
There are three important changes to the proposal review process in Cycle 8.
i) ALMA will adopt a distributed peer review process to scientifically review a majority of proposals submitted to the Cycle 8 2021 Call for Proposals, in which one member of the proposal team on each submitted proposal will be responsible for reviewing ten proposals.
ii) Review panels will evaluate a subset of proposals, specifically Large Programs and proposals requesting more than 25 hours on the 12-m array.
iii) All Cycle 8 2021 proposals will be reviewed using a dual anonymous procedure, in which the reviewers will not see the names of the investigators of any proposal that they review.
Q2: I have a question about the process that is not covered here. What should I do?
Contact the Helpdesk and specify the Project Planning department. Maybe your question will end up on the FAQ!
Q3: How will ALMA merge the results of distributed peer review and of panel review?
The main outcome of the proposal review process is a scientifically-ranked list of proposals. The inputs used to generate the ranked list are 1) the proposal rankings of proposal sets from individual reviewers in distributed peer review, 2) the ranked list of proposals (excluding Large Programs) within each individual panel, and 3) the ranked list of Large Programs recommended by the ALMA Proposal Review Committee (APRC).
Merging the distributed peer review rankings
In distributed peer review, each reviewer ranks their assigned proposals from 1 (strongest) to 10 (weakest). This ranking is essentially a series of choices by the reviewer, first choosing the top-ranked proposal from all proposals, then choosing the second-ranked item from the remaining proposals, and so on. The Plackett-Luce model (see Plackett 1975 and Luce 1977) provides a mathematical formalism to combine the choices from the individual reviewers to determine the consensus ranked list of all proposals. The Plackett-Luce model improves upon simply averaging the individual ranks in that it considers the strength of the proposals within a Proposal Set. For example, if a reviewer gives a poor rank to a proposal only because the Proposal Set contains a high number of excellent proposals (as determined by other reviewers), the Plackett-Luce algorithm accounts for this effect when computing the final rankings. Turner et al. (2020) is just one of many papers that present research on implementing the Plackett-Luce model.
Merging the panel rankings
Within a panel, the average score given by the reviewers will be used to determine the proposal rankings of each panel. The ranks from the individual panels are then merged to create a single ranked list. In order to do this, the rankings in each panel are first normalized by the total number of proposals within the panel, and then the rankings from all panels are sorted by the normalized rank. The sorted list of normalized ranks are then renumbered from 1 to N, where N is the total number of proposals in all panels. Any ties in the normalized ranks are broken by using the panel scores for the proposals, which are first normalized so that all panels have the same mean and standard deviation.
Merging the rankings for distributed peer review, panels, and Large Programs
The rankings from all review processes are then combined to produce a final ranked list that will be used to build an observing queue. The Large Programs recommended by the APRC will have top priority in the final ranked list. The rankings from the panels and distributed peer review are combined by first normalizing the ranked lists by the number of proposals in each review process and then sorting by the normalized rank.
Creating the observing queue
The ranked list of proposals only establishes the priority order to build the observing queue. Creating the observing queue takes into consideration the available time in a configuration and LST, the amount of time allocated to each executive, and the amount of time available based on historical weather conditions for a given observing band and observing frequency. It is during the queue-building stage that the priority grades (A, B, C) are assigned to the accepted proposals, and at this stage a proposal may be declined depending on the amount of telescope time available. Because numerous factors are considered in building the queue, it is possible for a high ranked proposal to be declined while a lower ranked proposal is accepted.
Approval of the review results
After the ranked list is generated and the observing queue is created, the ranked list and the priority grades are sent to the ALMA Director for approval. After the Director’s approval, the results are sent to representatives from East Asia, ESO, North America, and Chile for final approval. The JAO then notifies the Principal Investigators of the results.
Dual-Anonymous proposal review
Q1: What is dual-anonymous peer review, and why is it being implemented?
ALMA is strongly committed to ensuring that the proposal review process is as fair and impartial as possible. Analysis of the proposal rankings in previous cycles has identified systematics that may signify the presence of biases in the review process (Carpenter 2020). To ensure that the proposal review process is as fair and unbiased as possible, ALMA is beginning dual-anonymous review in Cycle 8 2021. In a dual-anonymous review, the proposal team does not know the identity of the reviewers and the reviewers do not know the identity of the proposal team. While proposers will still enter their names and affiliations in the ALMA Observing Tool (OT), this information will not appear on the proposal cover sheet, nor in the tools used by the reviewers. It is the responsibility of the proposers to ensure anonymity is preserved when writing their proposals.
Large Programs have special procedures that will allow the ALMA Proposal Review Committee (APRC) to see the identity of the proposal team, but only after the scientific review, rankings, and recommendations have been completed. Details on the review procedures for Large Programs are described in the Call for Proposals.
Q2: Am I still required to enter the names of myself and co-Investigators in the ALMA Observing Tool (OT)? Why?
Yes, you will still need to enter your name and the names of your co-Investigators in the ALMA OT, even when this information will not be accessible to the reviewers.
ALMA requires this information in the first stage of the proposal assignment process to prevent the assignment of a proposal to a reviewer with a clear conflict of interest. Later on, this information is required to associate the proposal, and eventual data if the proposal is accepted, to you and your co-Investigators.
Q3: How will reviewers identify conflicts of interest if they have no access to the list of proposers?
Institutional conflicts will be automatically identified based on user profiles of the investigators and the reviewers (to check and/or update your user profile please click here). Potential collaborative conflicts will also be identified based on the proposal submission history of the PI and the reviewers over previous and the current cycles. No proposal will be assigned to a reviewer if such an assignment would constitute any of these types of conflicts. In addition, reviewers will be able to manually declare conflicts on their assigned proposals that are not identified by our system, such as cases where the assigned proposal is in conflict with the reviewers own proposal. In such cases, the JAO will replace the conflicted proposal with a new one.
Q4: I would like to resubmit a proposal from a previous cycle. May I resubmit the same “Scientific Justification” PDF as before?
All proposals, including resubmissions, must comply with the anonymization rules. Since the dual-anonymous process is new to Cycle 8 2021, most likely you will need to modify the Scientific Justification following the dual-anonymous guidelines to ensure that it complies with the anonymization rules.
Q5: I would like to reference our previous ALMA proposal(s), because our Cycle 8 2021 proposal is an extension of that previous work. How can I reference proprietary ALMA data, or a previous incomplete ALMA project?
To properly reference proprietary ALMA data in an anonymized way, please use the corresponding project code or the phrase “obtained via private communication", but a name should not be specified since it could strongly imply who may be an investigator on the proposal.
Q6: What will happen to a proposal that is not fully anonymized?
Any proposals that appear to be in flagrant violation of the guidelines will be discussed internally between the Joint ALMA Observatory (JAO) Proposal Handling Team and the ALMA Director. If the violation is confirmed, the proposal will be rejected and will not be further reviewed. Less serious violations of the guidelines will be flagged by the JAO, but the proposal will continue to be reviewed. The JAO will provide proper feedback to the Principal Investigator if any violation is detected.
Q7: How can I know if I am following the guidelines on anonymity?
Guidelines are given in the dual-anonymous guidelines. If you have questions about making your proposal fully anonymized according to the guidelines, you can always contact your ALMA Regional Center (ARC) through the Helpdesk. Your ARC support staff will be able to help answer your question, or they will forward your question to the JAO Proposal Handling Team for additional assistance with the guidelines.
Q8: I’m an expert in a small, niche field, and many of the references relevant to my proposal are by members of our team. Even if I follow all of the guidelines to anonymize my proposal, I’m worried that my proposal may be identifiable. Could my proposal still be disqualified?
Your proposal will not be disqualified as long as you have followed the dual-anonymous guidelines to anonymize your proposal. The purpose of dual-anonymous is not to disregard your expertise or foundational work, but to guide the reviewer to focus and assess the science of a proposal rather than the group behind it. As long as the proposal is written in such a way that you do not explicitly identify yourself or your team, your proposal will be in compliance.
Q9: In my proposal, I would like to refer to data and/or software that are unpublished. How can we incorporate this information without disclosing our identity?
As described in the dual-anonymous guidelines, unpublished data or software can be referenced as "obtained via private communication". Unpublished data can also be referenced by a project code as long as it is not accompanied by the name of the PI.
Q10: Is the usage of the words "we" or "I" (or other identifying pronouns) prohibited in dual-anonymous peer review?
No, it is not entirely prohibited. First person pronouns (we, I, our, my) can be used to express the intent of the proposal itself, but not to identify the proposal authors. For example, you may use the phrases, “We plan to observe the CO molecular line…” or “Our preferred target is Target X due to…” or even, "Following the observational strategy employed in Perez et al. (2019), we plan to..." On the other hand, you should not use phrases like “We showed in Perez et al. (2019)...” that connect the selected pronouns to the proposal authors.
Q11: How do I show that I’m an expert equipped to work with the data, without revealing myself?
Your description of the scientific background, motivation, planned observations, and technical justification should express a level of expertise sufficient to convince the reviewers or panels, without needing to identify yourself personally.
Q12: Do proposals for Large Programs also need to follow the dual-anonymous guidelines?
Yes, the Scientific and Technical Justification of Large Programs must be fully anonymized in the same way as all proposals. In addition, Large Programs are required to include a one-page PDF file (the "management plan") that describes the roles of the proposal team and the available computing resources. The management plan should include the names of at least the key members of the proposal team.
Large Programs will be ranked based only on the scientific and technical justification. After the Large Programs have been scientifically ranked by the ALMA Proposal Review Committee (APRC), the APRC will be able to review the management plans. The APRC will be permitted to note any concerns on the qualifications of the management team to carry out the proposed science. The concerns will be forwarded to the ALMA Director for consideration. However, the scientific ranking of the Large Programs will remain the same. Further information of the review process for Large Programs can be found in the Proposer's Guide.
Q13: Where can I find more information about dual-anonymous peer review, as it has been applied and assessed in other proposal review processes for other facilities?
JWST, HST, other NASA science missions, and ESO, have already implemented dual-anonymous in their call for proposals. You can find more information, including additional links and references, in the following:
NASA → https://science.nasa.gov/researchers/dual-anonymous-peer-review
ESO → https://www.eso.org/sci/observing/phase1/dual-anonymous-guidelines.html
Distributed peer review
Q1: What is distributed peer review?
Distributed peer review is a process in which one member of the proposal team, either the Principal Investigator (PI) or a co-Investigator (co-I), commits to review ten other submitted proposals. Therefore, the best proposals are selected by peer review, but instead of having a small committee review a large number of proposals, a larger number of people each review a smaller number of proposals. In this manner, the review load is distributed among many peers, and any individual reviewer will have a lower workload, with more time to spend reviewing each assigned proposal.
Q2: Are other observatories using distributed peer review?
Yes: Gemini’s Fast Turnaround proposal system uses distributed peer review and ESO initiated a pilot program in 2018, which received a very positive response from the community.
Q3: Why is ALMA using distributed peer review instead of the traditional panels to review a subset of the proposals on the Main Call?
Distributed peer review proved to be a viable review system for ALMA and its community after the success of the pilot program for the Cycle 7 Supplemental Call in 2019. The advantages that this review system provides include an increased involvement of the community in the review process and a significant reduction of the workload for each reviewer. Based on the success of the pilot program, feedback from the community who participated in the pilot program, and advice from the ALMA Science Advisory Committee, the ALMA Board approved using distributed peer review for the smaller proposals in Cycle 8 2021 while using traditional panel reviews for the larger proposals.
Q4: How will the distributed peer review process work for the Cycle 8 2021 Call for Proposals?
Proposals requesting less than 25 hours on the 12-m array and less than 150 hours on the 7-m array will be evaluated through distributed peer review. At proposal submission, the PI of such proposals must designate someone from the proposal team to be the reviewer. In most cases the PI will be the designated reviewer, but the PI may also designate a co-I. After the proposal deadline, the Proposal Handling Team (PHT) at the Joint ALMA Observatory (JAO) will assign ten proposals to the designated reviewer of each proposal. If reviewers have a conflict of interest with any of their assigned proposals, they can request a replacement proposal through the Reviewer Tool. Then during Stage 1, the reviewer will rank the ten proposals (1-10) in order of scientific priority, and write a brief review for each proposal. If ranks and reviews are not submitted by the time of the Stage 1 review deadline, the proposal on whose behalf the reviewer is acting as the designated reviewer will be rejected.
After the Stage 1 review deadline, each reviewer with on-time Stage 1 submissions will have the option to participate in a second stage (Stage 2) of the review process. In this stage, the anonymized comments from the other reviewers of the same proposals will be made available. Reviewers can modify their own ranks and comments if desired in Stage 2. Once this second stage is completed, the ranks from all reviewers of each submitted proposal will be combined to produce a global ranked list of proposals, which will be used to produce the observing queue after merging with the results of the panel process.
Q5: Who can I designate to be the reviewer for my submitted proposal? As a PI, how do I designate a reviewer?
Any PI and most co-Is on the proposal can be designated as the reviewer. If the PI does not have a PhD at the time of proposal submission (e.g., a student), the PI can still be the reviewer, but a mentor (who must have a PhD) must be identified at the time of the proposal submission. A PI may designate a co-I as the reviewer as long as the co-I has a PhD in astronomy or a closely related field. The reviewer and if needed, the mentor, must be designated in the Observing Tool (OT) at the time of proposal submission and cannot be changed after the proposal deadline.
Q6: Can a reviewer split the workload with other co-authors?
No. Reviewers are committed to review all ten proposals assigned to the proposal set associated with the proposal on which they are the designated reviewer. The review of these assignments cannot be shared with others. If a PI submits many proposals, then for each proposal they can select one of their co-authors (with a PhD) to be the designated reviewer, and that person will be responsible for reviewing the respective proposal set. Thus, in this way the workload of reviewing many proposal sets does not have to rest with a single reviewer unless they choose to do so.
Q7: Can a student (without a PhD) be a designated reviewer?
A student can be the designated reviewer if they are the PI of the proposal and also designate a mentor, who must have a PhD in astronomy or a closely related field. The mentor must be identified when the proposal is submitted. The mentor does not have to be a co-I on the student’s proposal. A student co-I is not eligible to be a designated reviewer.
Q8: What is the role of a mentor?
A mentor should provide guidance as needed to a student reviewer during the process, but there are no formal requirements. The mentor must abide by the same confidentiality requirements as the student reviewer.
For more information about the role of a mentor please vcheck the “Guidelines for Mentors”.
Q9: Who will see my proposal?
Each proposal will be sent to ten reviewers. In the case of a student reviewer, the corresponding mentor may also see a proposal.
Q10: How much expertise will the reviewers who are reviewing my proposal have?
Each submitted proposal will receive ten reviews. Most reviewers will have a PhD, and student reviewers (without a PhD) will be required to designate a mentor with a PhD to assist with the reviews. Reviewers will preferentially be assigned proposals with similar proposal keywords and the same proposal category to match the expertise of the reviewer.
Q11: As a PI, is there anything else I should keep in mind when submitting my proposal?
PIs should ensure that the designated reviewer has an up-to-date email address in the Observing Tool. If the email is not up-to-date, please ask the reviewer to update their ALMA user profile through the ALMA Science Portal. The JAO will use email to communicate with your reviewer, so if the email address is out of date, there is a chance that your reviewer will not submit their reviews on time and your proposal will be rejected as a result. It is the responsibility of the PI and the designated reviewer to ensure the reviews are submitted on time.
Additionally, please encourage the reviewer to enter their fields of expertise in their ALMA Science Portal profile. This information is extremely important for the optimal assignment of the proposals that each person will review.
Q12: How will proposals be assigned to reviewers?
To the extent possible, the JAO will assign proposals based on the expertise of the reviewer. To match proposal assignments to a reviewer's expertise, reviewers can specify their areas of expertise (which is highly recommended!) in the user profile accessible on the ALMA Science Portal (LINK). The more category/keyword pairs that a reviewer specifies, the more likely it is that they will be assigned proposals with their expertise. If a reviewer does not specify their expertise, they will be assumed to be experts in the category/keyword(s) of their submitted proposal.
It is still possible, however, that a reviewer cannot be assigned proposals entirely within their field(s) of expertise. In these cases, the Proposal Handling Team has created lists of "similar keywords", both within a single scientific category and between scientific categories. The Proposal Handling Team will attempt to assign proposals in these similar category/keywords, or in the same category as the submitted proposal.
In uncommon instances, a reviewer may still be assigned a proposal in a category other than that of the submitted proposal or their area(s) of expertise. For instance, PIs submitting proposals in keywords that receive few proposals overall are less likely to be assigned proposals that match their expertise; there are just not many proposals in those keywords, and the reviewers in those keywords tend to be conflicted on many of those proposals.
Q13: Is lack of expertise a reason to declare a conflict in the review of a proposal? That is, if I am a total outsider to the field, can I declare a conflict when reviewing a proposal?
No, a lack of expertise with a given proposal scientific category and keyword is not a reason to declare a conflict. Reviewers are asked to rank their assigned proposals based on scientific merit. PIs should always try to make the science case clear even to non-experts.
However, to further assist reviewers who may not consider themselves expert on a given assignment, Distributed Peer Review contains a "Stage 2" step where reviewers can view the comments (but not the ranks) of other reviewers and modify their own rankings and reviews if they choose. In this way, reviewers who are possibly less experienced in a given science area, or who may have missed important information, can take the comments of other reviewers into consideration in forming their final ranks and reviews.
Q14: As a reviewer, if I declare a conflict with one of my assignments, how long should it take to receive a new assignment in its place?
During the first week of the process (May 6-13), the JAO will collect conflict declarations from all reviewers and perform one batch reassignment at the end of that period. The quality of the reassignments improves as more assignments become available to reassign, which is why the aim is to do this first reassignment all at once. Reviewers will not receive any replacement assignments until this first batch reassignment has been performed. After May 13, as reviewers assess their conflicts on the new assignments, smaller batch reassignments will be performed more often. It may take less than one working day and as long as a few working days to receive a replacement assignment in these cases. While the reassignments are done you can start to review those proposals for which you do not have a conflict of interest with.
Q15: What if I am participating in the distributed review process and I do not submit my ranks and reviews by the Stage 1 review deadline? If I submit the ranks for one proposal set but not the other, will both of my proposals be rejected, or only the one for which I failed to submit the ranks?
If a designated reviewer does not submit their reviews before the Stage 1 review deadline, the proposal for which they were identified as the designated reviewer will be rejected. If a user is the designated reviewer for more than one proposal, they will be assigned a distinct proposal set to review for each submitted proposal. If the reviewer submits ranks and reviews for one proposal set but not for another, only the submitted proposal(s) associated with unsubmitted ranks and reviews will be rejected.
Q16: Will my proposal really be rejected if I don't submit all my reviews before the Stage 1 deadline of the distributed review process?
Yes.
Q17: What if I am the PI of several proposals that qualify for the distributed review process? Will I have to review proposals for each of my submitted proposals?
A PI may submit several proposals that qualify for the distributed peer review process. For each of those proposals, a PI will need to designate a reviewer. If you have submitted several proposals as PI, you do not have to be the designated reviewer on all (or any) of them. The designated reviewer for each proposal can be the PI or a co-I, and that person commits to review ten other proposals. If the same person is the designated reviewer associated with more than one proposal, then that person will review ten proposals for each respective proposal. For example, if a reviewer is the designated reviewer on behalf of 3 proposals, then that person should expect to review 30 other proposals.
In addition, the same PI may submit proposals requesting more than 25 hours of time on the 12-m array, and/or Large proposals, which will instead be reviewed by panels.
Q18: During Stage 2 will I have the option to interact with other reviewers?
No. During Stage 2 there will be no interaction among the reviewers, other than the ability to see others’ comments that were written in Stage 1. The purpose of this stage is to help reviewers identify any important issue that they may have overlooked initially. This can be done by reading the reviews written by other reviewers who have different experiences and points of view.
Q19: How should reviewers determine the rank for a given assignment? Is this a score or a rank?
The ranks submitted for a set of reviews are relative ranks, not absolute scores. As a result, a reviewer should rank their assignments from 1 to N, where N is the number of assignments in a Proposal Set. For example, the strongest proposal should be assigned rank=1, the second-best proposal rank=2, etc…, with the weakest proposal assigned rank=N. The ranks within a Proposal Set may not be duplicated.
Q20: What feedback will PIs receive on their submitted proposals?
After the proposals have been reviewed and the JAO constructs the Cycle 8 2021 observing queue, PIs will be sent their priority grade as well as the proposal ranks and reviews, verbatim, from each designated reviewer. The reviewer names will remain anonymous.
Q21: What is the timeline and deadlines for the panel and distributed peer review processes?
The schedule timeline for the distributed peer review process can be found here.
Q22: As a reviewer, how will I receive information about the review process?
Once the review process starts, all official communication between the JAO and the reviewer will be done through the reviewer's email address that is registered in their ALMA account. For smooth communication during the review process, it is crucial that the information in your ALMA account (email address, institution, etc.) is up to date. PIs and reviewers are urged to check that emails from the JAO Proposal Handling Team with address ph1m_noreply@alma.clare not in their SPAM folder. General questions about the process can be submitted to the ALMA Helpdesk.
Q23: What browsers are supported for the Reviewer Tool used in distributed peer review?
The Reviewer Tool works well with Chrome and Firefox, but we have identified some minor issues with Safari. If you use Safari to run the Reviewer Tool and encounter a technical issue, please try reloading the browser and/or try a different browser. If these workarounds do not solve your problem, please contact the ALMA PHT (pht@alma.cl).
Panel review
Q1: For proposals not included in the distributed peer review process, how will the panel review process work for the Cycle 8 2021 Main Call?
All Large Programs, and any proposals requesting more than 25 hours on the 12-m array, will be peer reviewed by volunteers from the community serving on traditional review panels with topical expertise. The JAO will assign each submitted proposal to a panel based on the scientific category selected by the PI.
In the first stage of the panel review process, panelists will individually score each non-conflicted proposal assigned to them and will write a brief review. In the second stage, panelists will meet (virtually, in Cycle 8) to discuss each proposal assigned to their panels. For Large Programs, the panels will vote to decide which Large Programs should advance to the ALMA Proposal Review Committee (APRC) for further review. For the remaining proposals (i.e., proposals requesting between 25 and 50 hours on the 12-m array), the panelists will re-score the proposals based on the panel discussion and the average score will be used to determine the proposal rank. Panelists will finalize the consensus reports based on the individual comments and the discussion during the meeting. The ranked lists from each panel will be merged, and later used to create the observing queue.
Following the ALMA Review Panel meetings, the ALMA Proposal Review Committee (APRC) will meet to review the Large Programs selected by the panels. The APRC will then re-score the Large Programs and make a recommendation on which Large Programs to schedule.
Q2: I submitted a proposal that requests more than 25 hours of time on the 12-m array, but is not a Large Program. Do I still have to designate a reviewer?
Yes. The Observing Tool will automatically require a reviewer to be designated for any proposal that is not a Large proposal. For all proposals that request between 25 to 50 hours on the 12-m array, the PI is required to designate a reviewer, and the reviewer will not need to review proposals as long as the final time request confirmed by the JAO is more than 25 h on the 12-m array.
Q3: The person who is designated as reviewer on a proposal that requests more than 25 hours of time on the 12-m array would need to review proposals of the distributed peer review process?
No.
Return to the main ALMA Proposal Review page