Double Decker Peer Review

In computer science, conferences are often the final publication venue for research. However, conference papers are not always reviewed very carefully in terms of objective criteria like correctness, and the focus of the review is often on subjective criteria such as interestingness and perceived impact. Moreover, sub-fields such as Machine Learning (and AI) have recently been reeling with the problem of a large number of submissions and not enough reviewers, thereby causing reviewer fatigue, noisy reviews, and some feelings of consternation among authors.

We instead suggest the following “double decker” structure for our conferences:

Conventional deck: This is identical to the review and acceptance in conferences today.
Online deck: If a paper is not accepted to the conventional deck of the conference due to low ratings in subjective criteria (such as interestingness or envisaged impact), but paper passes the objective criteria (such as correctness), then give the authors an option to post it as “Accepted for [conference name] online”. One may keep a low but non-zero threshold on the subjective criteria for papers to be accepted to the online deck.

Along with the double decker structure, there should be a greater emphasis on checking correctness of the results in conference peer review. This applies particularly to theory-focused papers where we can have some reviewers dedicated to checking correctness and other (possibly more experienced) reviewers can evaluate other criteria such as envisaged impact. The assignment of reviewers should be done in a manner that allows for such reviewing.

We think this double-decker approach can help mitigate the following issues:

– High subjectivity and noise in subjective criteria

Conferences often ask reviewers to provide their perceived impact of the paper and use this information for their final decisions. This criteria is often very hard to judge [1,2], and more so if reviewers do not have much experience in the area (e.g., graduate students). Additional criteria such as novelty and significance [3] also have a non-trivial influence on the decision but can also be highly subjective. The proposed changes of having the reviewers focus on specific criteria, and allowing for an online accepted deck with less emphasis on subjective criteria, can help make this process less noisy.

– Researchers’ time and energy in the resubmission cycle

A large number of papers rejected from a conference are then submitted to the next relevant conference. And this cycle goes on. In the present review setup, such a process may be somewhat of a necessity since: (a) authors may feel that the noise and reviewers’ subjective opinions were the cause of the rejection; and (b) authors may find it important to receive the “peer reviewed” stamp. The reviews from a conference sometimes helps in improving the quality of the paper, but some other times only express a subjective opinion regarding the perceived impact etc. In case of the latter, for authors who are happy with the online version, it will save time and effort by avoiding the resubmission process. The proposed change thereby breaks the resubmission cycle to some extent.

– High reviewing load

Breaking of the resubmission cycle will naturally lead to a reduction in the workload of reviewers.

– Reduced checking of correctness

An important objective of peer review is “to control the dissemination of research data to ensure that unwarranted claims, unacceptable interpretations or personal views are not published without prior expert review” [4]. However, a significant emphasis today has shifted to using and evaluating subjective criteria. The proposed change will hopefully put some more emphasis back on checking for correctness.

On a related note, the interdisciplinary journal PLOS ONE follows a similar principle, with the criteria for publication as “Often a journal’s decision not to publish a paper reflects an editor’s opinion about what is likely to have substantial impact in a given field. These subjective judgments can delay the publication of work that later proves to be of major significance. PLOS ONE will thoroughly peer-review your submissions and publish all papers that are judged to be technically rigorous.”

– Uncertainty in reviewing of follow-up work

Reviewer comments of the following form are not uncommon: “This submitted paper uses Theorem X from paper Y which is only on arXiv and hence not peer reviewed. Then how do we trust the correctness of that theorem, and hence how do we trust the correctness of this submitted paper?” The proposed procedure will assuage such reviewer concerns.

– Disadvantage to interdisciplinary papers

It is known that interdisciplinary work often has much tougher time getting published [5]. A part of this reason may be the non-trivial emphasis on “relevance” [3]. A lower bar on relevance for the online version can help interdisciplinary papers.

– Conference travel expenses

Not everyone may be able to afford to travel to a conference. Other publication venues are journals and arXiv, but journals have a much longer review period and arXiv is not considered “peer reviewed”. The option of an online deck in the conference will offer another point on this tradeoff: a low turnaround time along with a stamp of being peer reviewed.

– Radical changes

Finally, a note on feasibility. The proposed change is much less radical with respect to today’s peer-review system as compared to some other suggestions (such as an arXiv-only model [6]).