When I wrote earlier this week about peer review, I did so from the heart and from my observations over the years.  My timing was somewhat random.  Apparently my post was somewhat prescient.  I will present 2 links that make the point much better than I did.

Peer review trickery?

Two researchers — Robin Lovell-Badge, who spoke in a personal capacity, and Austin Smith, from the University of Cambridge — told the BBC that sometimes scientists might write negative reviews of the work or request additional and unnecessary experiments in an effort to get their own papers, and those of their friends, published sooner.

In an open letter to the editors of major scientific journals published last year, a group of 14 researchers, including Smith, argue that "papers that are scientifically flawed or comprise only modest technical increments often attract undue profile. At the same time publication of truly original findings may be delayed or rejected."

To prevent this sort of corruption, they say, reviews, response to reviews, and associated editorial correspondence should be published as supplementary materials with the paper.

Peer review: What is it good for?

There remains much reverence of the traditional process of peer review. I may be over interpreting the tenor of Andrew Morrison’s editorial in BioEssays but it seems to me that he is saying, as many others have over the years “if we could just have the rigour of traditional peer review with the ease of publication of the web then all our problems would be solved”. Scientists worship at the altar of peer review, and I use that metaphor deliberately because it is rarely if ever questioned. Somehow the process of peer review is supposed to sprinkle some sort of magical dust over a text which makes it “scientific” or “worthy”, yet while we quibble over details of managing the process, or complain that we don’t get paid for it, rarely is the fundamental basis on which we decide whether science is formally published examined in detail.

There is a good reason for this. THE EMPEROR HAS NO CLOTHES! [sorry, had to get that off my chest]. The evidence that peer review as traditionally practiced is of any value at all is equivocal at best (Science 214, 881; 1981, J Clinical Epidemiology 50, 1189; 1998, Brain 123, 1954; 2000, Learned Publishing 22, 117; 2009). It’s not even really negative. That would at least be useful. There are a few studies that suggest peer review is somewhat better than throwing a dice and a bunch that say it is much the same. It is at its best at dealing with narrow technical questions, and at its worst at determining “importance” is perhaps the best we might say. Which for anyone who has tried to get published in a top journal or written a grant proposal ought to be deeply troubling. Professional editorial decisions may in fact be more reliable, something that Philip Campbell hints at in his response to questions about the open letter [BBC article]:

The problem with peer review is complex.  Sometimes peer review is done with an agenda.  Some reviewers are doing similar work and truly have a conflict.  Not all researchers approach this task with laudable ethics.

Another problem with peer review comes from those who believe that nit-picking and complex critiques make the reviewer look intelligent and more useful.  I have read many reviews written for the aggrandizement.

Our current review process causes heartache and often limits the conversation.  It spends the time of the researcher, time that could be spent on real research rather than rewriting and rewriting.

We never ask what the cost of not publishing a paper is or what the cost of delaying publication could be. Eric Weinstein provides the most sophisticated view of this that I have come across and I recommend watching his talk at Science in the 21st Century from a few years back. There is a direct cost to rejecting papers, both in the time of referees and the time of editors, as well as the time required for authors to reformat and resubmit. But the bigger problem is the opportunity cost – how much that might have been useful, or even important, is never published? And how much is research held back by delays in publication? How many follow up studies not done, how many leads not followed up, and perhaps most importantly how many projects not refunded, or only funded once the carefully built up expertise in the form of research workers is lost?

Rejecting a paper is like gambling in a game where you can only win. There are no real downside risks for either editors or referees in rejecting papers. There are downsides, as described above, and those carry real costs, but those are never borne by the people who make or contribute to the decision. Its as though it were a futures market where you can only lose if you go long, never if you go short on a stock. In Eric’s terminology those costs need to be carried, we need to require that referees and editors who “go short” on a paper or grant are required to unwind their position if they get it wrong. This is the only way we can price in the downside risks into the process. If we want open peer review, indeed if we want peer review in its traditional form, along with the caveats, costs and problems, then the most important advance would be to have it for unpublished papers.

I believe strongly that my instincts were correct.  I like the second article because he clearly points out that the Emperor Has No Clothes.  Bravo!

Related posts:

1. Who watches the watchers? – more on peer review
2. Does peer review help medicine?
3. Does peer review work?
4. Criticizing the NEJM
5. Muddling through the literature

Related posts brought to you by Yet Another Related Posts Plugin.