Disclaimer: I have frequently done peer reviews of medical articles. I have disagreed vehemently with peer reviews that I have received.

Flaws are found in validating medical studies

They are two of the most widely publicized pieces of medical research in recent years: Reports in prestigious journals declared that women who underwent hormone replacement therapy, and people who ingested large amounts of Vitamin E, had relatively low rates of heart disease.

Each study was vetted by peer review, the basic process for checking medical research, in which other researchers judge whether papers meet scientific standards.

But after research contradicted those studies — frustrating anyone who had followed their recommendations — some specialists began looking at whether peer review had failed to identify serious flaws in the research.

But the specialists found that it was almost impossible to discover what had happened in the vetting process, since peer reviewers are unpaid, anonymous, and unaccountable. Moreover, their reviews are kept confidential, making it impossible to know the parameters of the reviews.

Now, after a study that sent reverberations through the medical profession by finding that almost one-third of top research articles have been either contradicted or seriously questioned, some specialists are calling for radical changes in the system.

In advance of a world congress on peer review next month in Chicago, these specialists are suggesting that reviewers drop their anonymity and allow comments to be published. Some are proposing that peer reviewers be paid to ensure a more even quality of review and analysis among all journals.

Dr. Drummond Rennie, who relies on review as deputy editor of JAMA, the Journal of the American Medical Association, said of the process, ”The more we look into it, the harder it is to prove whether it does good or bad.”

Peer review has many problems. We review articles without compensation. Because of that, often one decides not to review an article, just because it is inconvenient.

While we like to think that evaluating a scientific article is simply a logical matter, I can easily argue that I bring biases to my reviews. I have received reviews that did not reflect my science, but rather how the reviewer considers the subject matter.

Much science creates controversy. Whenever there is controversy, people (for after all reviewers are people) take sides. We all know that when we submit an article, we run the risk of having the wrong reviewer. I suspect that I have been the wrong reviewer for some articles.

What I dislike about peer review is the arbitrariness. Editors, based on reviews, make decisions on which articles they think are important. My most important article, frequently cited, was published in a brand new journal, because none of the established articles would accept it for publication.

So does peer review help or hinder scientific progress? I believe that we would be better served by easier publication, tied with easier room for critiques and discussion, i.e., much more like blogging.

If I write something outrageous, I will certainly receive commentary criticizing my position. Others will support me, and this healthy debate should help readers develop better understanding.

We need more discourse on scientific articles. We need more commentary to help us put new findings into perspective. Perhaps we need to end politesse and “get down and dirty” with the data.

Related posts:

1. Does peer review help medicine?
2. Who watches the watchers? – more on peer review
3. More criticism on peer review
4. Good teachers beget more good teachers
5. Can expert panels improve health care?

Related posts brought to you by Yet Another Related Posts Plugin.