🔍

Science Highlight

 

 

More...
news!

Frequently Asked Questions

General

Q1: What important changes have been made to the proposal review process for Cycle 9?

You can find the major changes introduced in Cycle 9 here.

 

Q2: I have a question about the process that is not covered here. What should I do?

Submit a Helpdesk ticket under the Project Planning department.

 

Q3: How will ALMA determine the overall proposal rankings?

The main outcome of the proposal review process is a scientifically ranked list of proposals. The inputs used to generate the ranked list are 1) the proposal rankings of Proposal Sets from individual reviewers in distributed peer review, and 2) the ranked list of Large Programs recommended by the ALMA Proposal Review Committee (APRC).

Distributed peer review rankings

In distributed peer review, each reviewer ranks their assigned proposals from 1 (strongest) to 10 (weakest). The rankings from individual reviewers will be combined to form an overall ranked list. For a given proposal, the highest and lowest ranks will be dropped. The remaining ranks will be averaged.  The sorted list of average ranks will then determine the overall scientific ranked list of proposals.

Creating the observing queue

The ranked list of proposals only establishes the priority order to build the observing queue.  The Large Programs recommended by the APRC have first priority in building the queue, but they are limited to how much time they can fill in a configuration/LST as described in the Proposer’s Guide. Creating the observing queue also takes into consideration the available time in a configuration and LST, the amount of time allocated to each executive, and the amount of time available based on historical weather conditions for a given observing band and observing frequency. It is during the queue-building stage that the priority grades (A, B, C) are assigned to the accepted proposals, and at this stage a proposal may be declined depending on the amount of telescope time available. Because numerous factors are considered in building the queue, as described above, it is possible for a high ranked proposal to be declined while a lower ranked proposal is accepted.

Approval of the review results

After the ranked list is generated and the observing queue is created, the ranked list and the priority grades are sent to the ALMA Director for approval. After the Director’s approval, the results are sent to representatives from East Asia, Europe, North America, and Chile for final approval. Principal Investigators are then notified of the results.

 

Q4: What is the list of conflicts of interest and for what it is used?

Reviewers and mentors have the option to provide a list of investigators for whom they have a major conflict of interest. This list can be provided in their user profile through the ALMA Science Portal. Reviewers will not be assigned a proposal in which the PI, a co-PI, or one of the co-Is is in the list of conflicts of interest provided by the reviewer or their mentor.

 

Q5: How do I know if I have a conflict of interest with someone or not?

To identify your conflicts of interest please follow the guidelines described here.

 

Q6: If I am a reviewer, or mentor, is it mandatory for me to provide a list of conflicts of interest?

No, providing a list of conflicts of interest is optional for all ALMA users.

 

Q7: If I am a reviewer, or mentor, what will happen if I do not provide a list of conflicts of interest?

If a reviewer does not provide the list of their conflicts of interest, a list of potential conflicts of interest will be identified based on the proposal submission history of the reviewer, as described in the conflict criteria.

 

Dual-Anonymous proposal review

Q1: What is dual-anonymous peer review, and why has it been implemented? 

ALMA is strongly committed to ensuring that the proposal review process is as fair and impartial as possible. Analysis of the proposal rankings in previous cycles has identified systematics that may signify the presence of biases in the review process (Carpenter 2020). To ensure that the proposal review process is as fair and unbiased as possible, the ALMA proposal review process follows a dual-anonymous system. In a dual-anonymous review, the proposal team does not know the identity of the reviewers and the reviewers do not know the identity of the proposal team. While proposers will still enter their names and affiliations in the ALMA Observing Tool (OT), this information will not appear on the proposal cover sheet, nor in the tools used by the reviewers. It is the responsibility of the proposers to ensure anonymity is preserved when writing their proposals.

Large Programs have special procedures that will allow the ALMA Proposal Review Committee (APRC) to see the identity of the proposal team, but only after the scientific review, rankings, and recommendations have been completed. Details on the review procedures for Large Programs are described in the Call for Proposals.

 

Q2: Am I still required to enter the names of myself and co-I in the ALMA Observing Tool (OT)? Why?

Yes, you will still need to enter your name and the names of your co-I in the OT, even though this information will not be accessible to the reviewers.

ALMA requires this information in the first stage of the proposal assignment process to prevent the assignment of a proposal to a reviewer with a clear conflict of interest. Later on, this information is required to associate the proposal, and eventual data if the proposal is accepted, to you and your co-I.

 

Q3: How will reviewers identify conflicts of interest if they have no access to the list of proposers?

The Proposal Handling Team (PHT) at the Joint ALMA Observatory (JAO) will identify conflicts of interest based on the list of conflicts of interest provided by the reviewers in their user profiles (to check and/or update your user profile please click here). Reviewers will not be assigned a proposal in which a PI, co-PI, or co-I is in this list. If a reviewer does not provide this information, potential collaborative conflicts will be identified based on the proposal submission history of the reviewer. In addition, reviewers will be able to manually declare conflicts on their assigned proposals that are not identified by our system, such as cases where the assigned proposal is in conflict with the reviewer's own proposal. In such cases, the PHT will replace the conflicted proposal with a new one.

 

Q4: I would like to resubmit a proposal from a previous cycle.  May I resubmit the same “Scientific Justification” PDF as before?   

All proposals, including resubmissions, must comply with the anonymization rules. PIs should verify that the resubmitted proposal complies with the current  dual-anonymous guidelines.

 

Q5: I would like to reference our previous ALMA proposal(s), because our Cycle 9 proposal is an extension of that previous work.  How can I reference proprietary ALMA data, or a previous incomplete ALMA project?

To properly reference proprietary ALMA data in an anonymized way, please use the corresponding project code without referencing it as it is your own, or use the phrase “obtained via private communication". Note that a name should not be specified since it could strongly imply who may be an investigator on the proposal. Further information and examples can be found in the dual-anonymous guidelines.

 

Q6: What will happen to a proposal that is not fully anonymized?

Any proposals that appear to be in significant violation of the guidelines may be rejected. Any such violations will be reviewed by the Proposal Handling Team and the final decision will be made by the ALMA Director.

 

Q7: How can I know if I am following the guidelines on anonymity?

Guidelines are given in the dual-anonymous guidelines. If you have questions about making your proposal fully anonymized according to the guidelines, you can contact your ALMA Regional Center (ARC) through the Helpdesk . Your ARC support staff will be able to help answer your question, or they will forward your question to the Proposal Handling Team for additional assistance with the guidelines.

 

Q8: I’m an expert in a small, niche field, and many of the references relevant to my proposal are by members of our team.  Even if I follow all of the guidelines to anonymize my proposal, I’m worried that my proposal may be identifiable.  Could my proposal still be disqualified?

Your proposal will not be disqualified as long as you have followed the dual-anonymous guidelines  to anonymize your proposal. The purpose of dual-anonymous is not to disregard your expertise or foundational work, but to guide the reviewer to focus and assess the science of a proposal rather than the group behind it. As long as the proposal is written in such a way that you do not explicitly identify yourself or your team, your proposal will be in compliance.

 

Q9: In my proposal, I would like to refer to data and/or software that are unpublished. How can we incorporate this information without disclosing our identity?

As described in the dual-anonymous guidelines, unpublished data or software can be referenced as "obtained via private communication".  Unpublished data can also be referenced by a project code as long as it is not accompanied by the name of the PI, or is not referenced as it is your own.

 

Q10: Is the usage of the words "we" or "I" (or other identifying pronouns) prohibited in dual-anonymous peer review?

No, it is not entirely prohibited. First person pronouns (we, I, our, my) can be used to express the intent of the proposal itself, but not to identify the proposal authors.  For example, you may use the phrases, “We plan to observe the CO molecular line…” or “Our preferred target is Target X due to…” or even, "Following the observational strategy employed in Perez et al. (2019), we plan to..." On the other hand, you should not use phrases like “We showed in Perez et al. (2019)...” that connect the selected pronouns to the proposal authors. 

 

Q11: How do I show that I’m an expert equipped to work with the data, without revealing myself?

Your description of the scientific background, motivation, planned observations, and Technical Justification should express a level of expertise sufficient to convince the reviewers or panel, without needing to identify yourself personally.

 

Q12: Do proposals for Large Programs also need to follow the dual-anonymous guidelines?  

Yes, the Scientific and Technical Justification of Large Programs must be fully anonymized in the same way as all proposals. In addition, Large Programs are required to include a one-page PDF file (the "management plan") that demonstrates the proposal team has the resources available to carry out the proposal in a timely fashion. The management plan should include the names of at least the key members of the proposal team.

Large Programs will be ranked based only on the Scientific and Technical Justification. After the Large Programs have been scientifically ranked by the ALMA Proposal Review Committee (APRC), the APRC will review the management plans. The APRC will be permitted to note any concerns on the qualifications of the management team to carry out the proposed science. The concerns will be forwarded to the ALMA Director for consideration. However, the scientific ranking of the Large Programs will remain the same. Further information of the review process for Large Programs can be found in the Proposer's Guide.

                                                           

Q13: Where can I find more information about dual-anonymous peer review, as it has been applied and assessed in other proposal review processes for other facilities?

JWST, HST, plus other NASA science missions, and ESO, have already implemented dual-anonymous in their call for proposals. You can find more information, including additional links and references, in the following:

JWST → https://jwst-docs.stsci.edu/jwst-opportunities-and-policies/jwst-call-for-proposals-for-cycle-1/jwst-cycle-1-anonymous-proposal-review

HST →https://outerspace.stsci.edu/display/APRWG/Recommendations+of+the+Working+Group+on+Anonymizing+Proposal+Reviews

NASA → https://science.nasa.gov/researchers/dual-anonymous-peer-review

ESO → https://www.eso.org/sci/observing/phase1/dual-anonymous-guidelines.html

 

Distributed peer review

Q1: What is distributed peer review?

Distributed peer review is a process in which one member of the proposal team, either the PI or a co-I, commits to review ten other submitted proposals. Therefore, the best proposals are selected by peer review, but instead of having a small committee review a large number of proposals, a larger number of people each review a smaller number of proposals. In this manner, the review load is distributed among many peers, and any individual reviewer will have a lower workload, with more time available to review each assigned proposal. 

 

Q2: Are other observatories using distributed peer review?

Yes: Gemini’s Fast Turnaround proposal system uses distributed peer review and ESO initiated a pilot program in 2018 for La Silla Paranal Observatory, which received a very positive response from the community.

 

Q3: What if I am the PI of several proposals that qualify for the distributed review process? Will I have to review proposals for each of my submitted proposals?

A PI must designate a reviewer for each proposal that will be reviewed through the distributed peer review process. The designated reviewer for each proposal can be the PI or a co-I, and that person commits to review ten other proposals. If the same person is the designated reviewer associated with more than one proposal, then that person will review ten proposals for each respective proposal. For example, if a reviewer is the designated reviewer on behalf of two proposals, then that person should expect to review twenty proposals.

While it is possible to be designated as reviewer for more than one proposal, it is important to keep the workload to a manageable level. It is strongly recommended that a reviewer commits to review at most thirty proposals.

 

Q4: Who can I designate to be the reviewer for my submitted proposal? As a PI, how do I designate a reviewer?

Any PI and most co-Is on the proposal can be designated as the reviewer. If the PI does not have a PhD at the time of proposal submission (e.g., a student), the PI can still be the reviewer, but a mentor (who must have a PhD) must be identified at the time of the proposal submission. A PI may designate a co-I as the reviewer as long as the co-I has a PhD in astronomy or a closely related field. The reviewer and if needed, the mentor, must be designated in the Observing Tool at the time of proposal submission.

 

Q5: Can a reviewer split the workload with other co-investigators?

No. Reviewers are committed to review all ten proposals assigned to the Proposal Set associated with the proposal on which they are the designated reviewer. The review of these assignments should be carried out by the same person, as this ensures uniform and consistent scientific ranking and review. If a PI submits many proposals, then for each proposal they can select one of their co-Is (with a PhD) to be the designated reviewer, and that person will be responsible for reviewing the respective Proposal Set. Thus, in this way the workload of reviewing many Proposal Sets does not have to rest with a single reviewer unless they choose to do so.

 

Q6: Can a student (without a PhD) be a designated reviewer?

A student can be the designated reviewer if they are the PI of the proposal. In this case, the student has to also designate a mentor, who must have a PhD in astronomy or a closely related field. The mentor must be identified in the Observing Tool (OT) when the proposal is submitted. The mentor does not have to be a co-I on the student’s proposal. A student co-I is not eligible to be a designated reviewer.

 

Q7: What is the role of a mentor?

A mentor provides guidance as needed to a student reviewer during the process. This includes providing advice to the student on the scientific merit of the proposals, and on writing the comments to the PI. The mentor must abide by the same confidentiality requirements as the student reviewer.

For more information about the role of a mentor please check the Guidelines for Mentors.

 

Q8: Who will see my proposal?

Each proposal will be sent to ten reviewers. In the case of a student reviewer, the corresponding mentor will also see the proposal.

 

Q9: What is the expertise level of the reviewers?

Each submitted proposal will receive ten reviews. Most reviewers will have a PhD, and student reviewers (without a PhD) will be required to designate a mentor with a PhD to assist with the reviews. Reviewers will preferentially be assigned proposals with the same keywords and categories as their expertise.

 

Q10: As a PI, is there anything else I should keep in mind when submitting my proposal?

PIs should ensure that the designated reviewer has an up-to-date email address in the Observing Tool (OT). If the email is not up-to-date, please ask the reviewer to update their ALMA user profile through the ALMA Science Portal. The Proposal Handling Team will use email to communicate with your reviewer, so if the email address is out of date, there is a chance that your reviewer will not submit their reviews on time and your proposal will be rejected as a result. It is the responsibility of the PI and the designated reviewer to ensure the reviews are submitted on time.

Additionally, please encourage the reviewer to enter their fields of expertise and conflicts of interest in their ALMA Science Portal profile. This information is extremely important for the optimal assignment of the proposals that each person will review.

 

Q11: How will proposals be assigned to reviewers?

To the extent possible, the Proposal Handling Team (PHT) will assign proposals based on the expertise of the reviewer. To match proposal assignments to a reviewer's expertise, reviewers can specify their areas of expertise (which is highly recommended!) in the user profile accessible on the ALMA Science Portal . If a reviewer does not specify their expertise, they will be assumed to be experts in the category/keyword(s) of their submitted proposal.

It is sometimes necessary to assign proposals to reviewers that do not match their specified keywords. In these cases, the PHT will first attempt to assign proposals in the same scientific category as the reviewer’s expertise. In rare instances, a reviewer may be assigned a proposal in a category outside their expertise.

 

Q12: Is lack of expertise a reason to declare a conflict in the review of a proposal? That is, if I am a total outsider to the field, can I declare a conflict when reviewing a proposal?

No, a lack of expertise with a given proposal scientific category and keyword is not a reason to declare a conflict. Reviewers are asked to rank their assigned proposals based on scientific merit. PIs should always try to make the science case clear even to non-experts.

To further assist reviewers who may not consider themselves expert on a given assignment, distributed peer review contains a "Stage 2" step where reviewers can view the comments (but not the ranks) of other reviewers and modify their own rankings and reviews if they choose. In this way, reviewers who are possibly less experienced in a given science area, or who may have missed important information, can take the comments of other reviewers into consideration in forming their final ranks and reviews.

 

Q13: As a reviewer, if I declare a conflict with one of my assignments, how long should it take to receive a new assignment in its place?

During the first week of the process (May 04-11), the Proposal Handling Team will collect conflict declarations from all reviewers and perform one batch reassignment at the end of that period. The quality of the reassignments improves as more assignments become available to reassign, which is why the aim is to do this first reassignment all at once. Reviewers will not receive any replacement assignments until this first batch reassignment has been performed. After May 11, as reviewers assess their conflicts on the new assignments, smaller batch reassignments will be performed more often. It may take less than one working day and as long as a few working days to receive a replacement assignment in these cases. While the reassignments are done you can start to review those proposals for which you are not conflicted.

 

Q14: What should I do if I identify a new conflict of interest after I already declared them?

If a designated reviewer realizes they have a conflict on a proposal after submitting their conflict decisions, they should contact the Proposal Handling Team (pht@alma.cl).

 

Q15: How much time does it take to review a Proposal Set?

Reviewing a Proposal Set takes time. First the reviewer has to go through all assignments to identify possible conflicts of interest. Then, the reviewer has to read each proposal carefully, prepare the comment to the PI, and rank the proposals in the Proposal Set. Once all proposals have been reviewed, and ranked, it is recommended to go through the comments and ranks in case something needs to be adjusted. Reviewers can anticipate spending one to two days reviewing a single Proposal Set. It is highly recommended for reviewers to start the review process soon after they receive their Proposal Sets.

 

Q16: What if I am participating in the distributed review process and I do not submit my ranks and reviews by the Stage 1 review deadline? If I submit the ranks for one Proposal Set but not the other, will both of my proposals be rejected, or only the one for which I failed to submit the ranks?

If a designated reviewer does not submit their reviews before the Stage 1 review deadline, the proposal for which they were identified as the designated reviewer will be rejected. If a user is the designated reviewer for more than one proposal, they will be assigned a distinct Proposal Set to review for each submitted proposal. If the reviewer submits ranks and reviews for one Proposal Set but not for another, only the submitted proposal(s) associated with unsubmitted ranks and reviews will be rejected.

 

Q17: Will my proposal really be rejected if I don't submit all my reviews before the Stage 1 deadline of the distributed review process?

Yes.

 

Q18: During Stage 2 will I have the option to interact with other reviewers?

No. During Stage 2 there will be no interaction among the reviewers, other than the ability to see other comments that were written in Stage 1. The purpose of this stage is to help reviewers identify any important issue that they may have overlooked initially.  This can be done by reading the reviews written by other reviewers who have different experiences and points of view.

 

Q19: How should reviewers determine the rank for a given assignment? Is this a score or a rank?

The ranks submitted for a set of reviews are relative ranks, not absolute scores. As a result, a reviewer should rank their assignments from 1 to 10. For example, the strongest proposal should be assigned rank=1, the second-best proposal rank=2, etc…, with the weakest proposal assigned rank=10. The ranks within a Proposal Set may not be duplicated.

 

Q20: What feedback will PIs receive on their submitted proposals?

After the proposals have been reviewed and the priority grades are assigned, PIs will be sent their priority grade as well as the proposal ranks and reviews, verbatim, from each designated reviewer. The reviewer names will remain anonymous.

 

Q21: What is the timeline and deadlines for the distributed peer review processes?

The schedule timeline for the distributed peer review process can be found here.

 

Q22: As a reviewer, how will I receive information about the review process?

Once the review process starts, all official communication between the Proposal Handling Team and the reviewer will be done through the reviewer's email address that is registered in their ALMA account. For smooth communication during the review process, it is crucial that the information in your ALMA account (email address, institution, etc.) is up to date. PIs and reviewers are urged to check that emails from the Proposal Handling Team with address ph1m_noreply@alma.cl are not in their SPAM folder. General questions about the process can be submitted to the ALMA Helpdesk.

 

Q23: What browsers are supported for the Reviewer Tool used in distributed peer review?

The Reviewer Tool works well with Chrome and Firefox, but we have identified some minor issues with Safari. If you use Safari to run the Reviewer Tool and encounter a technical issue, please try reloading the browser and/or try a different browser. If these workarounds do not solve your problem, please contact the Proposal Handling Team (pht@alma.cl).

 

Large Programs

Q1: How are Large Programs reviewed?

Large Programs will be reviewed by a single panel of experts, the ALMA Proposal Review Committee (APRC). In the first stage of the panel review process, all members of the APRC will individually score each non-conflicted Large Program and will write a review. In the second stage, panelists will meet to discuss and re-score the Large Programs, and make a recommendation on which Large Programs to schedule. Panelists will also finalize the consensus reports based on the individual comments and the discussion during the meeting.

Additionally, to gain further expertise, Large Programs will also be reviewed by external reviewers. These reviewers will provide written reports on the Large Programs, which will be considered during the APRC meeting, but will not participate in the meeting.


Return to the main ALMA Proposal Review page