FAQ for the EMNLP 2021 Decision Process
In this blog post, we provide more details on the process we followed with FAQ (adapted from the ACL-IJCNLP 2021 blog post) to answer to a wide range of questions authors may have. Please read this carefully, as most questions we get by email are answered here.
Q1.What types of decisions were made for papers?
Following the process from recent EMNLP and ACL conferences, the Programme Committee (PC) chairs based their decisions on the recommendations from technical tracks, combined with the recommendations from the Ethics Committee (EC) chairs, which led to five categories for the accept/reject decisions:
1. Accept-to-Main-Conference
2. Conditional-Accept-to-Main-Conference
3. Accept-to-Findings
4. Conditional-Accept-to-Findings
5. Reject
Q2. What was the process followed by the EC?
Reviewers were asked to flag papers that could have ethical issues (check the Ethics Policy section in our CFP. 272 papers were flagged as such. For any paper that the technical reviewing committee expressed ethical concerns, the paper was sent to the EC (203 papers). The EC went over the papers to determine whether a full EC review would be required. If so, the paper received one or two ethics reviews from additional reviewers. The ethics reviews are treated differently from technical reviews, and they do NOT impact the overall scores of the paper. For any paper that is recommended to be accepted into the main conference or Findings based on technical reviews and that had been referred to the EC, the EC chairs have recommended one of the following to the PC chairs: (a) accept (12 EMNLP, 11 Findings), (b) conditional accept (the ethical issues must be addressed in the camera-ready version; 17 EMNLP, 20 Findings), and (c) reject due to ethical issues (1 paper).
Q3. What will happen to the conditionally accepted papers?
All conditionally accepted papers have comments regarding ethics in the meta-review, as well as one or two additional reviews that will appear at the bottom of all other reviews. These reviews state the changes needed to be made by the authors in the camera-ready version. The authors need to submit the camera-ready version by September 9, 2021, the same deadline for the papers without conditions. In addition, the authors need to provide a short document that explains how they have made the changes requested by the EC meta-reviews. The document can be either a plain text file or a PDF file, and needs to be uploaded to the Final Submission Page. This explanatory document will not be made public; it is just to expedite the conditional acceptance workflow for the EC’s review process. The EC chairs will go over this document and check the camera-ready version to determine whether the required changes have been included in the camera-ready version. If so, the condition will be removed, the paper will be accepted to the main conference or Findings, and the status of having been conditionally accepted will not be publicly visible. If not, the paper will be rejected. If you have any questions about EC reviews or the changes required by EC, you can contact the EMNLP 2021 Ethics Chairs (Margot Mieskes: margot.mieskes at h-da.de and Christopher Potts: cgpotts at stanford.edu).
Q4. The AC recommends ‘Accept’. Why is my paper still rejected?
AC recommendation is just one of many factors used in the final decision. For instance, some ACs disagree with reviewers: they may recommend Accept when some reviewers insist on the paper being rejected, or the other way around. PCs also look at other factors such as author rebuttal, three reviews, the discussion in the discussion board, SAC recommendations, EC recommendations (for flagged papers), and the submission itself.
Q5. My overall recommendation scores are 4, 3.5, 3.5. Why is my paper still rejected (or accepted to Findings, but not to the main conference)?
Review scores are just one of many factors considered by PCs. For instance, some ACs point out that in some reviews, the overall recommendation scores do not match the reviews (e.g., a largely negative review with a score of 3.5), but reviewers do not change their reviews or overall recommendation scores for various reasons, even when prompted by ACs. In that case, PCs read reviews, AC recommendation and SAC recommendation in order to reach a final decision.
Q6. My reviews are largely positive, but the paper is rejected due to an ethical review. Does EC have the one-vote veto right?
No. EC recommendation is only one piece of information used in the final decision. The PCs make the final decision.
Q7. Where can we see revised reviews (or comments to our rebuttal)?
The reviews sent to authors on August 25 are the revised reviews. If reviewers had some comments to the rebuttal that they wanted the authors to see, they would include that in their revised reviews.
Q8. How come the reviews remain the same after the rebuttal?
Many reviewers used the discussion board at START and, after the reviewer/AC discussion, they decided to keep their original reviews because they were not convinced by the authors’ rebuttal.
Q9. After the rebuttal, some overall recommendation scores decreased. Why?
This is likely to be due to the reviewer/AC discussion. Some reviewers’ opinions could change due to the discussion or after reading authors’ rebuttal, so the scores can go up or down in the revised reviews. The change of the scores shows that the reviewers have read the author rebuttal. Reviewers were asked to revise their reviews to reflect their new opinion of the paper, but that might not always happen. Unfortunately, we cannot make the information in the discussion board available to the authors due to the nature of double-blind reviews.
Q10. Some reviewers (or the AC) misunderstood our paper. Can the decision be reconsidered?
If some (but not all) reviewers or the AC misunderstood your paper, the overall recommendation scores tended to vary a lot among reviewers or AC’s meta review would disagree with the three reviews. SACs and PCs paid special attention to such papers, and the final decision is the result of considering all the input including AC and SAC recommendations and the discussion board. We will not reconsider decisions unless there is a technical issue, such as the meta-review being for a different paper.
Q11. The AC raised some issues that were not found in the three reviews, which we could not address in the rebuttal period. We disagree with those points.
Diligent ACs often read the papers, especially when reviewers disagree. ACs can therefore point out issues that were not identified by the reviewers. Given their expertise in the area, AC recommendation was one of the factors for the final decision.
Q12. We received new reviews after the rebuttal process, and therefore could not respond to them.
Some reviewers were very late or didn’t complete their reviews before the deadline for rebuttals. We had to recruit additional (emergency) reviewers to help ensure that each paper would have at least 3 reviews before the end of the rebuttal process. This was achieved in the vast majority of cases, and in the few other cases we extended the rebuttal period to give the authors a chance to respond. However, in some cases the original reviewers ended up completing the reviews and submitting them after the rebuttal period. We did keep the fact that some authors did not get a chance to respond to these additional reviews in mind when making final decisions.
Q13. Some reviewers raised new issues in the revised reviews which we have no chance to respond to. Had they mentioned that in their original reviews, we could have addressed their concern.
It is not uncommon for new issues to be raised during the reviewers/AC discussion period, and reviewers can include such issues in their revised reviews. Given the short review cycle, it is impossible to have another round of author rebuttal after the discussion period. Nevertheless, as mentioned in Section 1, reviews are just one of many factors considered in the final decision.
Q14. My paper is marked as accepted to the main conference (or the Findings), but it includes ethical reviews and an ethical meta-review. Does that mean my paper is conditionally accepted instead?
If a flagged paper was determined by the EC that a full ethical review was necessary, the paper would get one or ethical reviews. If such a paper was accepted to the main conference or Findings, an ethical meta review would be added. Thus, having those reviews does not mean that the paper is conditionally accepted. If you want PCs to double-check for you, please email PCs with your submission ID.
Q15. Now that my paper has been accepted to the Findings, can I post it to arXiv.org and my personal website now?
Yes.
Q16. Can I make changes to the title in the camera-ready version?
Yes, changes to the title are allowed.
Q17. Where can we find the templates for the camera-ready version? Should my paper have an appendix?
You should use the same style file as for the initial submission, all you need is to remove the "review" option from the line \usepackage[review]{emnlp2021} to generate the final version. You can still have an appendix, but it should be added to the same PDF as the main paper.
Q18. I don’t see the final submission page. Why?
The Final Submission Page will be open by September 3rd, 2021 for authors to start uploading their camera-ready version.
Q19. What is the page limit for the camera-ready version of the paper?
You will have one extra page for that, 9 pages for long, and 5 pages for short papers, plus unlimited space for references and the impact statement. The same limit applies for Findings papers.
Q20. What criteria are used to determine whether a paper should be accepted to the main conference or Findings? Why some Findings papers have overall scores higher than some Main Conference papers?
Papers recommended for possible acceptance were ranked based on recommendations from reviewers, ACs and SACs. With the goal of keeping the standard acceptance rate (around 20-25%), these papers were then grouped into two categories: Accept for EMNLP and Accept for Findings, based on various criteria, such as substance and quality of contribution. Findings papers represent solid work that couldn’t be included in the main conference due to the low acceptance rate. Once more, the scores are only one of the many sources of information used for the decision.
Q21. Does Findings have its own ISBN or DOI?
We expect that Findings will grow with future conferences, and become a recognised and respected publication venue within the field. Currently, we do not have Findings indexed in Scopus and Web-of-Science etc. This is a complex process that can only be done after a few years from the start of the publication. We note that Findings is a peer-reviewed, competitive publication outlet, which should have significantly more weight than workshops or arXiv publications. In addition, many workshops are non-archival, whereas Findings is.
Of course, authors have the choice to withdraw from Findings and submit to another conference, or ARR, if they chose to do so. However, the fact that the paper only narrowly made it for EMNLP this time does not guarantee it will make it for a main conference if resubmitted, given that it will be reviewed independently, by another set of reviews, ACs, SACs and PCs.
Q22. Will Findings papers have some form of presentation / poster session in the main conference?
We will follow the procedure introduced by ACL-IJCNLP 21, where papers will be given a short slot for presentation at the main conference. An email with details will be sent to all authors soon.
Q23. Will Findings papers be included in the proceedings of the main conference?
No. There will be a separate volume for Findings papers, which will be included in the ACL anthology under “Findings of EMNLP”.
Q24. Will my paper be presented as Oral or Poster?
We will be in touch soon with further details for each paper.
Q25. The conference will be in a hybrid format. What does it mean for my paper?
The conference will have both in person and remote participation, both for authors and participants in general. Authors will have the choice to choose between presenting a paper in person or remotely. In all cases, talks will be pre-recorded as a back-up option. The details on the format are being finalised with Underline, we will let authors know soon. Make sure to fill out the form to indicate your preference.
Q26. My paper was submitted via ARR and accepted for EMNLP or Findings. Will it be treated differently?
No, the paper will be published in the respective proceedings as any other EMNLP/Findings paper. The only difference will be the final submission system, which for ARR papers will be OpenReview. The format of the ARR papers should be the same as EMNLP, so please use the EMNLP style files and instructions. The number of content pages is also the same: 9 for long papers, 5 for short papers.
Q27. I haven’t received my notification. How can I find out about the status of my paper?
Many emails bounced, probably because some authors’ mail servers do not accept such emails. Please log on to Softconf and click on your paper, click ‘Your Reviews’ and you will see not only the reviews but the status at the very top of the page.