🔍

Science Highlight

 

 

More...
news!

Frequently Asked Questions

Frequently Asked Questions

1. If my ACA proposal is not accepted in the Cycle 7 Main Call, can I submit the same proposal to the Cycle 7 Supplemental Call?

Yes. PIs will be able to update their proposals based on comments received from the Cycle 7 review panels before submitting to the Supplemental Call.

2. What is distributed peer review?

Distributed peer review is a process in which users who submit proposals also commit to reviewing the other proposals. Therefore, the best proposals are still selected by peer review, but instead of having a small committee review a large number of proposals, a large number of people each review a small number of proposals. In this way, the review load is distributed among many peers, and any individual reviewer will have a lower workload and more time to spend reviewing each assigned proposal.

3. How will the Cycle 7 Supplemental Call distributed peer review process work?

At proposal submission, the PI must designate someone from the proposal team to be the reviewer. Normally the PI will be the designated reviewer, but the PI may also designate a co­investigator, as described in the following frequently-­asked­-questions. After the proposal deadline, the Joint ALMA Observatory (JAO) will assign ten proposals to each designated reviewer. If reviewers have a conflict of interest with any of their assigned proposals, they can request a replacement proposal through the Reviewer Tool. The reviewer will then rank the ten proposals (1-­10) and write a brief review of each one. The ranks from all reviewers of each submitted proposal will be averaged to produce a global ranked list of proposals, which will be used to produce the observing queue.

4. Who can I designate to be the reviewer for my submitted proposal? As a PI, how do I designate a reviewer?

Any PI and most co­-investigators on the proposal can be designated as the reviewer. If the PI does not have a PhD at the time of proposal submission (e.g., a student), the PI can still be the reviewer, but a mentor (who must have a PhD) must be identified at the time of the proposal submission through the Observing Tool (OT). If the PI designates a co-I as the reviewer, the co-I must have a PhD in astronomy or a closely related field.

The reviewer must be designated in the Observing Tool at the time of proposal submission and cannot be changed after the proposal deadline.

5. Can a student (without a PhD) be a designated reviewer?

A student can be the designated reviewer if s/he is the PI of the proposal and also designates a mentor, who must have a PhD in astronomy or a closely related field. The mentor must be identified when the proposal is submitted. The mentor does not have to be a co­-investigator on the student’s proposal. A student co­-investigator is not eligible to be a designated reviewer.

6. What is the role of a mentor?

A mentor should provide guidance as needed to a student reviewer during the process, but there are no formal requirements. The mentor must abide by the same confidentiality requirements as the student reviewer. For more details, please check the Guidelines for Mentors page.

7. As a PI, is there anything else I should keep in mind when submitting my proposal?

In addition to specifying a reviewer to participate in the distributed review process when you submit your proposal, please also ensure that the email address for the reviewer you have selected has an up-to-date email address in the Observing Tool. (If it is not up to date in the ALMA User Database, please ask them to update their ALMA user profile information.) The JAO will use email to communicate with your reviewer, so if the address is out of date, there is a chance that your reviewer will not submit their reviews on time and your proposal will be rejected as a result.

Further, consider that the assignment algorithm uses the keywords you select for your submitted proposal to be the "fields of expertise" of the designated reviewer of that proposal. This means that barring any conflict of interest, the algorithm will try its best to match review assignments with similar keywords to your designated reviewer, and the algorithm will attempt to assign your submitted proposal to reviewers with similar keywords. Thus please be careful in choosing these subcategories for your proposal, and keep in mind that two keywords may better describe your proposal rather than just one.

8. Who will see my proposal?

Each proposal will be sent to ten reviewers. In cases with student reviewers, the corresponding mentors may also see your proposal.

9. How much expertise will the reviewers who are reviewing my proposal have?

Each submitted proposal will receive ten reviews. Most reviewers will have a PhD, and student reviewers will be required to designate a mentor with a PhD to assist with the reviews. Reviewers will preferentially be assigned proposals with similar proposal keywords and the same proposal category to match the expertise of the reviewer with their assignments. PI feedback on all reviews received will be solicited after the review process has ended to address this point, among others.

10. What if I am the PI of several proposals to the Supplemental Call? Will I have to review proposals for each of my submitted proposals?

A PI will need to designate a reviewer for each submitted proposal. The designated reviewer can be the PI or another co­-investigator. If you have several submitted proposals as PI, you do not have to be the designated reviewer on all (or any) of them.

11. How should reviewers determine the rank for a given assignment? Is this a score or a rank?

The ranks submitted for a set of reviews are relative ranks, not absolute scores. As a result, a reviewer should rank their assignments from 1 to N, where N is the number of assignments in a Proposal Set. For example, the best proposal should be assigned rank=1, the second best proposal rank=2, etc…, with the poorest proposal assigned rank=N. The ranks within a Proposal Set may not be duplicated.

12. What if I do not submit my ranks and reviews by the review deadline? If I submit the ranks for one proposal set but not the other, will both of my proposals be rejected, or only the one for which I failed to submit the ranks?

If a designated reviewer does not submit his/her reviews before the review deadline, the proposal for which they were identified as a reviewer will be rejected. If a designated reviewer has several proposal sets and submits ranks for one but not for another, only the submitted proposal(s) associated with unsubmitted ranks and reviews will be rejected. It is imperative that the reviews are submitted by this deadline since the proposal ranks will be computed and observing queue built immediately following the deadline.

13. Will my proposal really be rejected if I don’t submit all of my reviews on time?

Yes.

14. How are conflicts of interest determined in the Supplemental Call?

Conflicts of interest will be defined using a similar process as in the Main Call. The review assignment algorithm considers proposals submitted in the Cycle 7 Main Call and the Cycle 7 Supplemental Call, and it avoids review assignments in which the reviewer(/mentor, if applicable) would be considered to have a major conflict with the PI of the proposal for review. In addition, reviewers will be able to declare additional conflicts on their assigned proposals and have such conflicted assignments replaced with other proposals. Refer to the guidelines for Conflict Criteria for more information.

15. As a reviewer, if I declare a conflict with one of my assignments, how long should it take to receive a new assignment in its place?

During the first week of the process (October 15-22), the JAO will collect conflict declarations from all reviewers and perform one batch reassignment at the end of that week. The quality of the reassignments improves as more assignments become available to reassign, which is why we aim to do this first reassignment all at once. You will not receive any replacement assignments until this first batch reassignment has been performed. After October 22, as reviewers assess their conflicts on the new assignments, smaller batch reassignments will be performed more often. It may take less than one working day and as long as a few working days to receive a replacement assignment in these cases.

16. Will the author lists be randomized in the Supplemental Call peer review process?

Starting with the Cycle 7 Main Call, ALMA will utilize a randomized investigator list during the peer review process to reduce potential sources of bias. The Cycle 7 Supplemental Call will use the same randomization as the Main Call during the peer review process.

17. How will the accepted proposals be determined?

As in the Main Call, the process to determine which proposals are accepted involves creating a ranked list of all proposals; in the Cycle 7 Supplemental Call, this list is generated using the average rankings for each proposal. The JAO will then use the global ranked list, the time available per receiver band and LST, and regional balance across the ALMA regions to determine which proposals fit within the observing queue (with priority Grade “C”).

18. As a PI, how will I receive feedback on my submitted proposal?

After the JAO determines which proposals fit within the time available, PIs will be sent their priority grade as well as the proposal ranks and reviews, verbatim, from each designated reviewer. The reviewer names will remain anonymous.

19. How will reviews be assigned to reviewers?

The scientific category and keywords of the submitted proposal will be used to determine the starting review assignments for each reviewer. If there are not enough submitted proposals in that category/keyword combination without conflicts, assignments from similar categories (i.e., Category 1 and 2, or Category 3 and 4) may also be used. If there are still not enough submitted proposals in similar categories, then all categories may be used.

20. I have been assigned a review assignment outside of my submitted proposal’s scientific category. What should I do?

The scientific category and keywords of the submitted proposal will be used to determine the starting review assignments for each reviewer. If there are not enough submitted proposals in that category/keyword combination without conflicts, assignments from similar categories (i.e., Category 1 and 2, or Category 3 and 4) may also be used. If there are still not enough submitted proposals in similar categories, then all categories can be used. Please assess the science case of the submitted proposal to the best of your ability.

21. What operating systems are supported for the Reviewer Tool?

Safari, Chrome, Firefox, and standard tablet operating systems. For tablets, we recommend running the browser in full screen mode for improved usability.

22. As a reviewer, how will I receive information about the review process?

General questions about the process can be submitted to the ALMA Helpdesk, but once the review process starts, all official communication between the JAO and the reviewer will be done through the reviewer's email address that is registered in her/his ALMA account. As a result, for smooth communication during the review process, it is crucial that the information in your ALMA account (email address, institution, etc.) is up to date and that emails from the JAO are not ending up in your SPAM folder.

23. Are other observatories using distributed peer review systems?

Yes: Gemini’s Fast Turnaround proposal system uses distributed peer review ( https://www.gemini.edu/sciops/observing-gemini/proposal-routes-and-observing-modes/fast-turnaround ) and ESO initiated a pilot program in 2018 ( https://www.eso.org/sci/publications/announcements/sciann17130.html ), which received a very positive response from the community ( https://www.eso.org/sci/publications/announcements/sciann17141.html ).

24. Why is ALMA using distributed peer review instead of the traditional panels to review the Supplemental Call?

ALMA would not be able to offer a Supplemental Call using a panel-­based review model since it would require considerable resources from the JAO as well as the community. Distributed peer review provides a viable means to retain the benefits of peer review while maximizing the opportunity for the community to use the ACA.

25. Will ALMA continue to have Supplemental Calls?

ALMA will monitor the proposal pressure on the ACA in the Main Call to determine if a Supplemental Call will be needed in future Cycles.

26. Will ALMA continue to use distributed peer review?

The Cycle 7 Supplemental Call represents a pilot program for distributed peer review in ALMA, and we plan to learn from the experience and gather feedback from the community before considering using the system in future cycles.

27. Are TP-only proposals acceptable for the Supplemental Call?

No, though users can resubmit approved proposals from previous cycles where only TP observations remain to be executed.

28. I have a question about the process that is not covered here. What should I do?

Contact the Helpdesk and specify the Project Planning department. Maybe your question will end up on the FAQ!

 


 

Back to Supplemental Call Main Page.