Attention Higher Education Faculty and Professionals: We Need to Take Peer Review More Seriously

A significant part of my job as a professor is to write research-based manuscripts, grant applications, and conference proposals. My success in publishing journal articles, presenting papers, and securing extramural funding will help to determine that my academic freedom and due process rights are protected through the granting of tenure. What this means is that I spend a good deal of time contending with peer review. Simply put, my career prospects are frequently predicated on peers’ assessments of my work. This makes peer review deeply important and personal to me.

Unfortunately, my experience with peer review has been decidedly mixed and getting worse. No, it hasn’t been negative because I'm bitter that reviewers don’t like my work. My mentor taught me early on that the best scholars take critique gracefully, so when reviewers sincerely raise questions about my work, I tend to give the comments significant thought. No, my experience with peer review has not been negative because reviewers have been rude or even nasty (though I have had a few borderline instances of this). My problem with peer review is that people don’t seem to take it seriously enough. Moreover, there are too few organizational checks in many review processes to prevent shoddy reviews from being sent out.

I can only speak to my own experience, which is based in the field of higher education. In the past year, I have received: 

•        One review for a journal manuscript that was a single sentence. This was after a 6-month wait for a first-round review.
•       One review for a conference proposal in which the reviewer indicated “not applicable” for most of the categories and offered as written feedback: “This is his total research”.
•      One review for a journal manuscript where the reviewer inexplicably conflated the theoretical framework and methods. (We used a theory that was derived from a long-term study that collected data at over 100 liberal arts and comprehensive universities. The reviewer thought we should expand the number of institutions in our study to include community colleges!)
•     One review for a conference proposal in which the reviewer felt they could not evaluate the conclusions and deemed the paper would not be complete because data had only been collected at 2/3 institutional sites. (We offered preliminary findings based on the 25 interviews that had already been conducted at the two sites.) 

It would be easy to simply shrug these experiences off as byproducts of an imperfect but generally functional system. You might also simply label me a poor loser, and I’ll own up to that. However, I think there are larger issues at play—issues at the individual and organizational level.

At the individual level, I suspect that there are a range of conditions and characteristics leading to reviews of poor quality.

#1 The just-in-time reviewer – This is the reviewer that leaves just enough time before the deadline to quickly scan your paper and offer a few generic but largely unhelpful comments. I also put in this category the “always too busy” reviewer, who probably should not have agreed to take on this responsibility but did and then found themselves simply too busy to give it due consideration.

#2 The unprepared reviewer – This is the reviewer who has been asked to review a paper they don’t understand for one reason or another. Perhaps it uses a research method with which they are not particularly well versed. Perhaps it is a graduate student or new professional who simply hasn’t learned how to conduct a review properly.

#3 The ne’er satisfied reviewer – This is the reviewer who will never, under any circumstances, concede that someone else’s work has value. Alternatively, this reviewer has been burned by enough reviewers that they use peer review to settle scores.

#4 The neglectful editor – I’ve never been an editor, so I’m speculating here. However, many of the terrible reviews I’ve received have ostensibly been seen and okayed by an editor. If you are an editor, and you see that a reviewer patently misunderstood the study or provided nothing but a few words of feedback, you are part of the problem.

At the organizational level, I see a host of problems that facilitate shoddy peer review. Again, my comments here are specific to the field of higher education.

#1 Too few scholarly associations and journals – In the United States there is basically one, maybe two, scholarly associations dedicated to the study of higher education broadly. Yes, there are a host of professional organizations for a variety of administrative functional areas. However, these organizations don’t always care much for scholarly research, and if you are interested in things outside of student affairs, you have relatively few conferences from which to choose. Similarly, there are just a few generalist peer-reviewed journals. The result is intense competition to have papers accepted in these venues. I have heard (and am even guilty) of submitting absurd numbers of proposals to conferences in the hopes something will land. Specific to peer review, this means that reviewers are being bombarded with a huge number of papers for a small number of slots. It’s a recipe for low-quality peer review.

#2 Small boxes, narrow perspectives – Many of the higher education conferences force you to place your work within a category: students, faculty, policy, and so forth. There are many, many research topics, however, that simply do not fit neatly in one or any of the boxes. As a result, reviewers are sometimes perplexed about whether to accept the paper or reject it for lack of “fitness”. In the same vein, the rubrics that are used to evaluate papers for conferences tend to adhere to very traditional structures of scholarly presentation: purpose statement, literature review, theoretical framework, methods, findings, significance. If you happen to be working on something just a little outside of the norm, you inevitably receive lower scores. Lastly, the field of higher education is besieged by an infatuation with large datasets, both quantitative and qualitative. I can’t tell you how many times I’ve seen: “This is a great study, but data is from just one institution, so we can’t do anything with it.” Recycling tropes of such narrow epistemological orientations lead to a devaluing of many types of research through the peer review process.

#3 Lack of education – We frequently fail to teach graduate students how to conduct professional peer review. This means they learn how to conduct peer review through their own experiences with peer review, thus perpetuating a cycle.

There are likely a few other reasons at both the individual and organizational level that have led, in my opinion, to the degradation of peer review. Thankfully, I think there are some concrete steps we, as a field, can take to address the problem.

First, I think editors need to play a more active role in vetting reviewers and reviews. The best journal I’ve had the pleasure of working with seemingly utilized a team of talented editors who had clearly both read my paper and the reviewers’ comments. The result was excellent feedback and stronger manuscript. Although it sounds extreme, if you are part of an editorial team sending out shoddy reviews, I think you should think long and hard about the integrity of your enterprise.

Second, we need some visionary young faculty and professionals to dream up new venues for presentation and publication. We desperately need more generalist venues for higher education scholarship, but also alternative venues that don’t so strictly adhere to convention.

Third, we need to orient our graduate students to the peer review process: why it matters, how it works, and what a professional review looks like. I think it is wonderful to include graduate students in conference review processes, but we have to ensure that they are prepared to take on the responsibility. 

Lastly, and most importantly, we need to respect peer review. This entails giving peers’ work the time it deserves and providing thoughtful, constructive feedback. It means not taking on too many commitments, diluting the attention you can give to someone’s paper or proposal. If you have ever complained about a peer review, be a part of the solution by ensuring your own reviews are high quality. In other words, you don’t get to complain about a review one day, then conduct a shoddy review the next day.

Professionally, much of my work lives and dies on the basis of peer review. And I’m not alone here. So, I’m calling on higher education faculty and professionals to take peer review seriously.