There are numerous, perennial debates over peer review within academia, most of which are above my pay grade. Academic peer review, as currently managed and exercised, has a number of serious problems – vicious and destructive reviewers, cumbersome and debilitating processes of submission and re-submission to meet the demands of a shifting cast of editors and reviewers at a single journal, journals that function as essentially closed shops, escalating demands on authors, and so forth.
But I have been thinking about another problem, in some ways external to academia, and that is the cooptation of peer review by other institutions. The core problem, as I see it, is the use of peer review to attach legitimacy to commissioned works that stand no chance of being rejected during the peer review process. In other words, if an institution commissions a paper (and promises payment for it), and then turns it over to internal or external reviewers, but essentially commits to publish the paper no matter what the reviewers say, then we have something less than peer review.
Of course, the reviewers’ comments, criticisms, and suggestions might drastically improve the paper, but if there is no chance of rejection, then it is just “peer input.” I have gone through this process myself with a few papers, and on my CV I do not list them as “peer-reviewed,” even though the commissioning institutions describe them as such. I have also been a reviewer in several such processes.
Admittedly, this can happen within academia too. Special journal issues can function this way, with no real chance of rejection for the papers included in the issue. I would also say that such papers are not fully peer-reviewed and I try to capture this distinction on my CV as well, although perhaps after writing this post I should re-categorize a few things. An even bigger question concerns books – I have to admit that in my experience publishing around ten articles and two books, I’ve found the review process much more “blind” with the articles than with the books, and I’ve found the chances of rejection to be much higher with articles. I do hear stories of book manuscripts being rejected by publishers because of negative readers’ reports, but I think a lot more of the rejection in the book publishing happens at the acquisition editor’s desk than in the review process.
There is something that bothers me much more, though, about non-academic institutions that co-opt peer review without any real risk of rejection for authors. The cooptation can give the illusion that a publication has been seriously vetted when in fact it hasn’t ever gotten an up-or-down vote from peers – rather, it is a commissioned work that reflects the priorities of those who commissioned it and their assessment of the author’s credentials, credibility, etc. The risk is that agenda-driven research that would not make it through a truly blind, unconstrained peer review will end up getting perceived by the wider public as somehow scientific, vetted, credible, etc.
More broadly, I’ve been thinking a bit about the two-way process whereby (a) academic institutions open their doors to non-academics as senior administrators, faculty members, and fellows, and (b) non-academic institutions adopt the language of academia to burnish their own images as objective, scientific, rigorous, etc. There are some real dangers here for academia, I think, in surrendering whatever independence remains to universities and in having the distinctions between scholarship and other forms of analysis get erased. No immediate solutions spring to my mind for how to address these dangers, other than to say that consumers (academic and non-academic) of “peer-reviewed” research need to be critical and open-eyed about that label really means. It would also be good, of course, for think tanks and other institutions to clarify what “peer-reviewed” really means to them.