Friday, July 15, 2016

Attention Higher Education Faculty and Professionals: We Need to Take Peer Review More Seriously

A significant part of my job as a professor is to write research-based manuscripts, grant applications, and conference proposals. My success in publishing journal articles, presenting papers, and securing extramural funding will help to determine that my academic freedom and due process rights are protected through the granting of tenure. What this means is that I spend a good deal of time contending with peer review. Simply put, my career prospects are frequently predicated on peers’ assessments of my work. This makes peer review deeply important and personal to me.

Unfortunately, my experience with peer review has been decidedly mixed and getting worse. No, it hasn’t been negative because I'm bitter that reviewers don’t like my work. My mentor taught me early on that the best scholars take critique gracefully, so when reviewers sincerely raise questions about my work, I tend to give the comments significant thought. No, my experience with peer review has not been negative because reviewers have been rude or even nasty (though I have had a few borderline instances of this). My problem with peer review is that people don’t seem to take it seriously enough. Moreover, there are too few organizational checks in many review processes to prevent shoddy reviews from being sent out.

I can only speak to my own experience, which is based in the field of higher education. In the past year, I have received: 
  •        One review for a journal manuscript that was a single sentence. This was after a 6-month wait for a first-round review.
  •       One review for a conference proposal in which the reviewer indicated “not applicable” for most of the categories and offered as written feedback: “This is his total research”.
  •      One review for a journal manuscript where the reviewer inexplicably conflated the theoretical framework and methods. (We used a theory that was derived from a long-term study that collected data at over 100 liberal arts and comprehensive universities. The reviewer thought we should expand the number of institutions in our study to include community colleges!)
  •     One review for a conference proposal in which the reviewer felt they could not evaluate the conclusions and deemed the paper would not be complete because data had only been collected at 2/3 institutional sites. (We offered preliminary findings based on the 25 interviews that had already been conducted at the two sites.) 

It would be easy to simply shrug these experiences off as byproducts of an imperfect but generally functional system. You might also simply label me a poor loser, and I’ll own up to that. However, I think there are larger issues at play—issues at the individual and organizational level.

At the individual level, I suspect that there are a range of conditions and characteristics leading to reviews of poor quality.

#1 The just-in-time reviewer – This is the reviewer that leaves just enough time before the deadline to quickly scan your paper and offer a few generic but largely unhelpful comments. I also put in this category the “always too busy” reviewer, who probably should not have agreed to take on this responsibility but did and then found themselves simply too busy to give it due consideration.

#2 The unprepared reviewer – This is the reviewer who has been asked to review a paper they don’t understand for one reason or another. Perhaps it uses a research method with which they are not particularly well versed. Perhaps it is a graduate student or new professional who simply hasn’t learned how to conduct a review properly.

#3 The ne’er satisfied reviewer – This is the reviewer who will never, under any circumstances, concede that someone else’s work has value. Alternatively, this reviewer has been burned by enough reviewers that they use peer review to settle scores.

#4 The neglectful editor – I’ve never been an editor, so I’m speculating here. However, many of the terrible reviews I’ve received have ostensibly been seen and okayed by an editor. If you are an editor, and you see that a reviewer patently misunderstood the study or provided nothing but a few words of feedback, you are part of the problem.

At the organizational level, I see a host of problems that facilitate shoddy peer review. Again, my comments here are specific to the field of higher education.

#1 Too few scholarly associations and journals – In the United States there is basically one, maybe two, scholarly associations dedicated to the study of higher education broadly. Yes, there are a host of professional organizations for a variety of administrative functional areas. However, these organizations don’t always care much for scholarly research, and if you are interested in things outside of student affairs, you have relatively few conferences from which to choose. Similarly, there are just a few generalist peer-reviewed journals. The result is intense competition to have papers accepted in these venues. I have heard (and am even guilty) of submitting absurd numbers of proposals to conferences in the hopes something will land. Specific to peer review, this means that reviewers are being bombarded with a huge number of papers for a small number of slots. It’s a recipe for low-quality peer review.

#2 Small boxes, narrow perspectives – Many of the higher education conferences force you to place your work within a category: students, faculty, policy, and so forth. There are many, many research topics, however, that simply do not fit neatly in one or any of the boxes. As a result, reviewers are sometimes perplexed about whether to accept the paper or reject it for lack of “fitness”. In the same vein, the rubrics that are used to evaluate papers for conferences tend to adhere to very traditional structures of scholarly presentation: purpose statement, literature review, theoretical framework, methods, findings, significance. If you happen to be working on something just a little outside of the norm, you inevitably receive lower scores. Lastly, the field of higher education is besieged by an infatuation with large datasets, both quantitative and qualitative. I can’t tell you how many times I’ve seen: “This is a great study, but data is from just one institution, so we can’t do anything with it.” Recycling tropes of such narrow epistemological orientations lead to a devaluing of many types of research through the peer review process.

#3 Lack of education – We frequently fail to teach graduate students how to conduct professional peer review. This means they learn how to conduct peer review through their own experiences with peer review, thus perpetuating a cycle.

There are likely a few other reasons at both the individual and organizational level that have led, in my opinion, to the degradation of peer review. Thankfully, I think there are some concrete steps we, as a field, can take to address the problem.

First, I think editors need to play a more active role in vetting reviewers and reviews. The best journal I’ve had the pleasure of working with seemingly utilized a team of talented editors who had clearly both read my paper and the reviewers’ comments. The result was excellent feedback and stronger manuscript. Although it sounds extreme, if you are part of an editorial team sending out shoddy reviews, I think you should think long and hard about the integrity of your enterprise.

Second, we need some visionary young faculty and professionals to dream up new venues for presentation and publication. We desperately need more generalist venues for higher education scholarship, but also alternative venues that don’t so strictly adhere to convention.

Third, we need to orient our graduate students to the peer review process: why it matters, how it works, and what a professional review looks like. I think it is wonderful to include graduate students in conference review processes, but we have to ensure that they are prepared to take on the responsibility. 

Lastly, and most importantly, we need to respect peer review. This entails giving peers’ work the time it deserves and providing thoughtful, constructive feedback. It means not taking on too many commitments, diluting the attention you can give to someone’s paper or proposal. If you have ever complained about a peer review, be a part of the solution by ensuring your own reviews are high quality. In other words, you don’t get to complain about a review one day, then conduct a shoddy review the next day.

Professionally, much of my work lives and dies on the basis of peer review. And I’m not alone here. So, I’m calling on higher education faculty and professionals to take peer review seriously.

Friday, February 5, 2016

Upscaling the American College Community

Last year, on the corner of the busiest intersection in College Park, Maryland—home of the University of Maryland’s flagship campus—a small but significant change took place. After 30 years of business, a derelict pizza shop closed its doors and, in its place, a popular chain of Portuguese chicken restaurants opened. This seemingly innocuous change is significant, not because of the reduction of pizza slices per capita (there are some 13 pizza joints near the campus) or the loss of a neighborhood institution. It is significant because it represents a microcosm of shifts taking place in College Park and many other college communities: upscaling.

There was a time when college communities were known, among other things, for cheapness. Cheap pitchers of beer at grimy bars. Cheap apartments and houses for rent. And, yes, even cheap slices of pizza. Goods and services were cheap in response to the well-worn mantra of the broke college student. If you wanted to attract college students, you competed on price more than quality. To be sure, there are still vestiges within every college community of businesses seeking to attract poor college students with basement prices. However, in many college communities there is a perceptible movement away from cheapness as a guiding business strategy. Instead, entrepreneurs are actively pursuing a more affluent segment of the consumer market.

As a professor of higher education, I have had the privilege of visiting many college communities, and in almost every case, the trend of upscaling is evident. By now, we’ve repeatedly heard and/or read about manifestations of this trend, particularly related to student housing. College communities have witnessed the construction of luxury apartment buildings, many of which are mixed-use developments, combining residential and commercial spaces. What this means in the context of college communities is that restaurants and convenience stores line the first floor of large apartment buildings. Students only need to walk down the hall and take an elevator to buy a burrito.

There are signs of upscaling beyond food and housing. Coffee shops feature designer roasts and every imaginable combination of espresso, milk, and flavoring. Bars are becoming less grimy and more glamorous, with signature cocktails and craft beer. Well-known fashion and technology brands are vying to occupy locations that receive heavy student (and parent) foot traffic. Entertainment venues like movie theaters are rolling out more services that carry higher prices. Mirroring trends in college pricing, the net effect of upscaling is that virtually everything in college communities is becoming more expensive. No matter where they look, students see opportunities to spend.

In some college communities, upscaling has been part of a purposeful plan to attract wealthy students and encourage faculty, staff, and young professionals to live near campus. As The New York Times recently reported, college communities in Philadelphia, Nashville, and Raleigh are building luxury housing not just for students, but also for members of the “creative class” in an effort to cultivate a “trendy live-work enclave.” The most idealistic of city planners hope that marrying the intellectual vibrancy of colleges with high-end housing replete with amenities will pull in young professionals and spark innovation, business creation, and economic growth. Some college communities—such as those David Brooks labeled “latte towns” in his analysis of the new upper class—have long flourished thanks to synergies between colleges, capital, and creativity.

Yet one question that arises from the upscaling of college communities is what it truly means for students, faculty, and staff. On the one hand, few people complain when an ugly building is torn down and replaced by an exciting restaurant. Who doesn’t want another option for late-night delivery or weekly office lunch? Additionally, from a safety perspective, there is value in students living in newer apartment complexes that are completely up to code. In the interest of work-life balance and protecting the environment, creating a college community in which faculty, staff, and students live nearby campus and forego long commutes is certainly beneficial. In fact, some would call the changes afoot in college communities as “smart growth.”

On the other hand, there is reason to challenge the notion that students, faculty, and staff have become affluent enough to explain and justify upscaling. If anything, research and experience suggests that colleges are welcoming increasing numbers of low-income and financially precarious students. These students may be the first in their family to attend college, they may be working adults, or they may be returning to school after a tour of duty in Afghanistan. Upscaling may price out those students who can’t pay to play, causing them to stop out or drop out, either due to financial pressures or a sense of marginalization.

The large proportion of students who rely on financial aid means that taxpayer money may be used to meet higher prices for goods and services. In this way, it is possible that the federal government subsidizes the “trendy live-work enclaves” developing within college communities. There are, of course, students from wealthy families attending college, perhaps enough to create a strong incentive to scale up. Still, the incongruence between mounting student loan debt and the transformation of college communities suggests to me a faulty assumption of affluence among students.

In addition to low-income students, international and graduate students may acutely feel the squeeze occasioned by upscaling. International students are generally not eligible for financial aid, and they may be shocked upon arrival at how far (or, more appropriately, not far) their money goes. We would be naïve to think that all or even a majority of international students come from wealthy families who can pay for what has become the quintessential American college experience. Much of the conversation around luxury housing in college communities has focused upon undergraduate students and young professionals. Graduate students, who may be attempting to support themselves and dependents on a paltry stipend, are seemingly excluded from the calculus of upscaling. As more and more luxury apartments are constructed, it becomes difficult for me to imagine how graduate students make ends meet.

Faculty and staff are perhaps better positioned to enjoy the fruits of upscaling in college communities because they earn a salary. It is true that if you want to convince faculty and staff to live near campus, you need more than cheap beer. Most faculty and staff desire decent housing options, reliable public transportation, good schools, and, if at all possible, places to exercise, relax, and enjoy a meal with family and friends. But this does not mean that faculty and staff seek luxury. Affordable housing and transportation remain paramount concerns, especially given that higher education continues to suffer through budget cuts that make the possibility of merit and cost-of-living pay increases a recurrent department meeting joke. Amidst the steady rise in the proportion of part-time instructional faculty at many colleges, the logic of upscaling is particularly twisted. In other words, while there are certainly benefits to changes underway in college communities, I’m left wondering who, precisely, constitutes the affluent segment of the consumer market.

College communities hold an important position in the American cultural imagination. We easily retrieve images of tree-lined streets, quaint pubs, funky shops, and red brick academic buildings. Having not visited every college community, I would guess that there are many places that still embody this description. However, I have seen enough to contend that our imagination is stuck in a version of college communities from thirty or even fifty years ago. Many of today’s college communities have dramatically changed in ways that require deep pockets to live comfortably. It is not at all coincidental that the college affordability crisis has emerged in tandem with the creation of college communities that are playgrounds for the affluent.