A significant part of my job as a professor is to write
research-based manuscripts, grant applications, and conference proposals. My success
in publishing journal articles, presenting papers, and securing extramural
funding will help to determine that my academic freedom and due process rights
are protected through the granting of tenure. What this means is that I spend a good deal of
time contending with peer review. Simply put, my career prospects are frequently predicated on peers’ assessments of my work. This makes peer review deeply
important and personal to me.
Unfortunately, my experience with peer review has been decidedly mixed and getting worse. No, it hasn’t been negative because I'm bitter that reviewers don’t like my work. My mentor
taught me early on that the best scholars take critique gracefully, so when
reviewers sincerely raise questions about my work, I tend to give the comments
significant thought. No, my experience with peer review has not been negative
because reviewers have been rude or even nasty (though I have had a few
borderline instances of this). My problem with peer review is that people don’t
seem to take it seriously enough. Moreover, there are too few organizational checks in many
review processes to prevent shoddy reviews from being sent out.
I can only speak to my own experience, which is based in the field of higher education. In the past year, I have received:
- One review for a journal manuscript that was a single sentence. This was after a 6-month wait for a first-round review.
- One review for a conference proposal in which the reviewer indicated “not applicable” for most of the categories and offered as written feedback: “This is his total research”.
- One review for a journal manuscript where the reviewer inexplicably conflated the theoretical framework and methods. (We used a theory that was derived from a long-term study that collected data at over 100 liberal arts and comprehensive universities. The reviewer thought we should expand the number of institutions in our study to include community colleges!)
- One review for a conference proposal in which the reviewer felt they could not evaluate the conclusions and deemed the paper would not be complete because data had only been collected at 2/3 institutional sites. (We offered preliminary findings based on the 25 interviews that had already been conducted at the two sites.)
It would be easy to simply shrug these experiences off as
byproducts of an imperfect but generally functional system. You might also
simply label me a poor loser, and I’ll own up to that. However, I think there
are larger issues at play—issues at the individual and organizational level.
At the individual level, I suspect that there are a range of
conditions and characteristics leading to reviews of poor quality.
#1 The just-in-time reviewer – This is the
reviewer that leaves just enough time before the deadline to quickly scan your
paper and offer a few generic but largely unhelpful comments. I also put in
this category the “always too busy” reviewer, who probably should not have
agreed to take on this responsibility but did and then found themselves simply
too busy to give it due consideration.
#2 The unprepared reviewer – This is the
reviewer who has been asked to review a paper they don’t understand for one
reason or another. Perhaps it uses a research method with which they are not
particularly well versed. Perhaps it is a graduate student or new professional
who simply hasn’t learned how to conduct a review properly.
#3 The ne’er satisfied reviewer – This is
the reviewer who will never, under any circumstances, concede that someone
else’s work has value. Alternatively, this reviewer has been burned by enough
reviewers that they use peer review to settle scores.
#4 The neglectful editor – I’ve never been
an editor, so I’m speculating here. However, many of the terrible reviews I’ve
received have ostensibly been seen and okayed by an editor. If you are an
editor, and you see that a reviewer patently misunderstood the study or
provided nothing but a few words of feedback, you are part of the problem.
At the organizational level, I see a host of problems that
facilitate shoddy peer review. Again, my comments here are specific to the
field of higher education.
#1 Too few scholarly associations and journals
– In the United States there is basically one, maybe two, scholarly
associations dedicated to the study of higher education broadly. Yes, there are
a host of professional organizations for a variety of administrative functional
areas. However, these organizations don’t always care much for scholarly
research, and if you are interested in things outside of student affairs, you
have relatively few conferences from which to choose. Similarly, there are just
a few generalist peer-reviewed journals. The result is intense competition to have papers
accepted in these venues. I have heard (and am even guilty) of submitting
absurd numbers of proposals to conferences in the hopes something will land.
Specific to peer review, this means that reviewers are being bombarded with a
huge number of papers for a small number of slots. It’s a recipe for
low-quality peer review.
#2 Small boxes,
narrow perspectives – Many of the higher education conferences force you to place your work within a category: students, faculty, policy, and so forth.
There are many, many research topics, however, that simply do not fit neatly in
one or any of the boxes. As a result, reviewers are sometimes perplexed about
whether to accept the paper or reject it for lack of “fitness”. In the same
vein, the rubrics that are used to evaluate papers for conferences tend to
adhere to very traditional structures of scholarly presentation: purpose
statement, literature review, theoretical framework, methods, findings,
significance. If you happen to be working on something just a little outside of the norm, you inevitably receive lower scores. Lastly, the field of higher
education is besieged by an infatuation with large datasets, both quantitative
and qualitative. I can’t tell you how many times I’ve seen: “This is a great
study, but data is from just one institution, so we can’t do anything with it.”
Recycling tropes of such narrow epistemological orientations lead to a
devaluing of many types of research through the peer review process.
#3 Lack of education
– We frequently fail to teach graduate students how to conduct professional
peer review. This means they learn how to conduct peer review through their own experiences with peer review, thus perpetuating a cycle.
There are likely a few other reasons at both the individual
and organizational level that have led, in my opinion, to the degradation of
peer review. Thankfully, I think there are some concrete steps we, as a field,
can take to address the problem.
First, I think editors need to play a more active role in
vetting reviewers and reviews. The best journal I’ve had the pleasure of
working with seemingly utilized a team of talented editors who had clearly both
read my paper and the reviewers’ comments. The result was excellent feedback
and stronger manuscript. Although it sounds extreme, if you are part of an
editorial team sending out shoddy reviews, I think you should think long and
hard about the integrity of your enterprise.
Second, we need some visionary young faculty and
professionals to dream up new venues for presentation and publication. We
desperately need more generalist venues for higher education scholarship, but
also alternative venues that don’t so strictly adhere to convention.
Third, we need to orient our graduate students to the peer
review process: why it matters, how it works, and what a professional review
looks like. I think it is wonderful to include graduate students in conference
review processes, but we have to ensure that they are prepared to take on the
responsibility.
Lastly, and most importantly, we need to respect peer
review. This entails giving peers’ work the time it deserves and providing
thoughtful, constructive feedback. It means not taking on too many commitments,
diluting the attention you can give to someone’s paper or proposal. If you have
ever complained about a peer review, be a part of the solution by ensuring your
own reviews are high quality. In other words, you don’t get to complain about a
review one day, then conduct a shoddy review the next day.
Professionally, much of my work lives and dies on the basis
of peer review. And I’m not alone here. So, I’m calling on higher education
faculty and professionals to take peer review seriously.
