Gender Distributions of Paper Awards

with Jingyan Wang.

In this post, we evaluate the gender distribution of best paper awardees in various top CS conferences. The inspiration to do so comes from an interview of Andrea Goldsmith in the IEEE Information Theory Society (ITSoc) newsletter [1] in which she lays out some interesting statistics:

“Going by the names of authors, it seems that of the 64 papers that have won the ITSoc paper award, not a single one has a female author. Similarly, it appears that not a single female student has won the ISIT student paper award. Only five women have been elevated to IEEE Fellow through ITSoc, which is quite a small number given that approximately 3–5 members have been elevated annually to Fellow through ITSoc going back many decades. Finally, only one of the nine Padovani lecturers, who are selected as role models for current ITSoc graduate students, has been female. In my own experience serving on the ITSoc awards committees and Fellows committee, I rarely see women nominated for society awards and honors. When they are my sense is that their research, achievements, and impact are judged more harshly than that of the men. Perhaps that is why women are not well represented among the recipients of ITSoc’s highest honors and awards.”

In this blog post, we ask a similar question for other venues. Specifically, we compile data on the fraction of women authors in award-winning papers in a number of top conferences in Computer Science in this decade (2010-2018). The following plot shows the percentage of women authors among all authors in award-winning papers:

The following plot shows the percentage of award-winning papers with at least one woman author:

For comparison, all except two award-winning papers (one from ACL and one from FOCS, both single-authored) have at least one male author.

The numbers are quite striking especially in venues on the left side of these plots. It is also important to note that the conferences which have a healthier distribution in a relative sense (those towards the right of the plots), still have the number of award-winning women authors less than 20%. Also interestingly, comparing conferences on similar topics, we see that STOC and FOCS are almost identical in these plots; on the other hand NeurIPS and ICML are significantly different (p<0.05 for in percentage of women authors in award-winning papers; Fisher’s two-sided exact test).

We do hope that the compilation of this data in the post will at least spur some conversation about this topic. Two specific topics of discussion are:

1. Evaluation: It would be of interest to compare with the distribution of genders among the submitted papers, accepted papers, and papers nominated for the awards at these venues. Can we identify the main source(s) of this discrepancy in the peer-review pipeline — whether it is in the submissions itself, paper acceptance decisions in reviews, nominations for the awards, or in the final award decisions?

2. Transparency: The process of determining the awards is often not clear. For instance, in the conferences which adopt a double-blind policy, are the author identities visible to the award committee? How is the committee determined, and what criteria do they use? Finally, if these conferences release some details on why a certain paper was chosen for the award, it will not only provide some criteria for the award but also help motivate and guide budding researchers. For example, the award selection process along with remarks on the award-winning papers for ACL 2017 is described in the program chairs’ blog.

Some fine print:

• We consider the primary best paper award for each conference in our evaluation (in particular, the Best Long Paper for ACL, and the best paper in the research track for KDD). For ISIT which does not give out a best paper award, we consider the best student paper award. NeurIPS gave out best student paper awards until 2012 and then switched to best paper awards — we consider both awards for NeurIPS.
• The gender annotation was done using a procedure similar to that followed by [2]. There were some instances where the annotation could not be done using this method, including one author each in PLDI and S&P, 4 in SIGCOMM, and 5 in OSDI.
• A confounding factor in the plot on papers with at least one woman author is the size of the author list across different conferences. For example, certain conferences (such as OSDI) have papers with larger numbers of authors.

[Update Jun 18, 2019: CVPR just announced their best paper award for 2019, which went to a paper (Shumian Xin et al.) with a woman first author]

References

[1] “IEEE Information Theory Society Newsletter.” 2018. https://www.itsoc.org/publications/newsletters/march-2018-issue/at_download/file

[2] “Reviewer bias in single-versus double-blind peer review,” A. Tomkins, M. Zhang, and W. Heavlin. Proceedings of the National Academy of Sciences, 2017.