by Adrienne Porter Felt
Peer reviewers wield substantial power in the computer science research community. They set the standards for research, guide the tastes of their fields, and ultimately decide what work has value. They are also overwhelmingly male.
Whenever I take my seat at a program committee meeting, I look around the room at the other reviewers. I see respected colleagues and friends, but few of the reviewers are women. Where are our missing sisters?
Reviewers are the gatekeepers of computer science research. To be taken seriously, research must be presented at one of a few dozen conferences. Program committee members review submissions and decide which work to accept. This is an inherently human process: reviewers use their own judgment to rate the quality and interest of submitted work. As with any human process, bias can creep into the system.
Do other people know that these gatekeepers are mostly men? In case this is not already broad knowledge, I calculated the gender breakdown of top computer science program committees. I selected seven top conferences, representing different fields of computer science: AAAI (Artificial Intelligence), CCS (Security), POPL (Programming Languages), SIGCOMM (Communication), VLDB (Databases), ASPLOS (Architecture), and SIGGRAPH (Graphics). Each conference lists their reviewers** on the website. I identified reviewers' genders by finding their preferred pronouns in professional homepages, online biographies, and press releases.
The gender breakdown of seven top conference program committees in 2017.
Of these seven conferences, the one with the most women is 25% women. The two worst conferences (AAAI and CCS) had only 12% women. On average, 81% of a program committee is men.
Given the composition of program committees, most researchers can expect their work to be reviewed primarily or only by men. (A typical paper receives three reviews, and women make up less than a third of reviewers.)
A visualization of what the CCS 2017 program committee would look like if they sat together in a room. Men are shown as blue dots, and women are shown as gray dots. The 148-person committee --- including three chairs --- had 18 women.
These statistics are appalling but not wholly surprising. The lack of women reviewers has been noted in other scientific fields. An analysis of Frontiers journals found that "women are underrepresented in the peer-review process, and that editors of both genders operate with substantial same-gender preference (homophily) when appointing reviewers." A similar study of the American Geophysical Union's journals found that women were used as reviewers less than expected based on their membership and authorship; they attributed the discrepancy to "authors and editors, especially male ones, suggesting women as reviewers less often, and a slightly higher decline rate among women in each age group when asked."
Why gender matters in peer review
Scientific fields are steered by the subjective tastes and standards of program committees. Since committees are predominantly male, one can conclude that scientific fields are steered by the tastes and standards of men.
In most cases, men and women likely have similar opinions about research. Gender does not play a role when designing an appropriate experiment to compare two compression algorithms. However, gender can play a role.
Imagine that program committees were predominantly women. In this world, would conferences publish papers that contain a photo of a Playboy centerfold? Predominantly male committees have allowed it in the field of image processing since the '70s. As a result, the image has made its way into high school classrooms. Despite objections from women professors and students, conferences and journals continue to accept papers with this image.
Fields of computer science often intersect with social issues. Top conferences feature papers on socially relevant topics like Bitcoin performance, pacemaker security, and secure communication for journalists. However, I can find relatively little work addressing the role of technology in domestic violence (with one recent exception, written mostly by women). Would domestic violence, stalking, and intimate partner abuse be considered "hot" topics if more reviewers were women***? Presumably, there are certain topics and social issues that interest more women than men (or vice versa).
Further, peer review suffers from unconscious sexism against women. (One study found that peer reviewers overestimate male achievements and underestimate female performance, concluding that "peer reviewers cannot judge scientific merit independent of gender.") A committee of mostly men might unintentionally be more sexist than a gender-balanced committee. Unfortunately, I don't think that gender-balanced committees will completely solve this problem. Women reviewers can also be biased against women authors. However, increasing representation in program committees might be one of several steps taken to rectify the problem.
What should we do about it?
I have grown accustomed to being the only woman in a room. I assume that my male colleagues have similarly grown accustomed to not seeing women in meetings. We must shed our learned callousness. We must refuse to accept that the gender imbalance in peer review is inevitable.
The most obvious step is to invite more women to sit on program committees. However, this should be done carefully. To quote Sandra Knapp:
"Women scientists are often called upon to sit on committees or serve on interview panels, in part to ensure the representation of women in key institutional roles. This, however, can lead to a vicious cycle of only a few women doing a disproportionate amount of community service, which eats into research time, limiting promotion prospects and so on. This spiral can be hard to break."
Instead of asking the same small pool of women to review extra papers, I instead recommend finding new women to sit on program committees. Women in industry might have deep expertise in a subject matter despite not publishing often. Women can also supply expertise from related fields with more women (for example, journalists, lawyers, or psychologists might be highly qualified to review papers in some fields).
We should also recognize and address the lack of racial diversity in program committees. I have never been on a program committee with more than one Black reviewer, and I have been on several with no Black reviewers. Latinxs are also underrepresented. Unfortunately, I was unable to gather statistics about race and ethnicity. A more racially diverse program committee will have a broader set of tastes and interests than a homogenous one.
Nothing will change unless program chairs realize that they are falling short. The next time that you speak to (or are) a program chair, ask them about their efforts to improve gender balance and racial diversity among the committee.
**For AAAI, I used the "senior" program committee.
***Although these issues affect both genders, they disproportionately affect women.
