Unintended consequences of our research funding


Category : Medical Rants

Many other bloggers and tweeters have linked to this important article – Lies, Damned Lies, and Medical Science .  The subtitle is revealing:

Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors—to a striking extent—still drawing upon misinformation in their everyday practice? Dr. John Ioannidis has spent his career challenging his peers by exposing their bad science.

I encourage you to read this article and consider it carefully.  I will provide a few quotes to whet your appetite.

… He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences. Given this exposure, and the fact that his work broadly targets everyone else’s work in medicine, as well as everything that physicians do and all the health advice we get, Ioannidis may be one of the most influential scientists alive. Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem.

The article goes on to give his explanation:

…In poring over medical journals, he was struck by how many findings of all types were refuted by later findings. Of course, medical-science “never minds” are hardly secret. And they sometimes make headlines, as when in recent years large studies or growing consensuses of researchers concluded that mammograms, colonoscopies, and PSA tests are far less useful cancer-detection tools than we had been told; or when widely prescribed antidepressants such as Prozac, Zoloft, and Paxil were revealed to be no more effective than a placebo for most cases of depression; or when we learned that staying out of the sun entirely can actually increase cancer risks; or when we were told that the advice to drink lots of water during intense exercise was potentially fatal; or when, last April, we were informed that taking fish oil, exercising, and doing puzzles doesn’t really help fend off Alzheimer’s disease, as long claimed. Peer-reviewed studies have come to opposite conclusions on whether using cell phones can cause brain cancer, whether sleeping more than eight hours a night is healthful or dangerous, whether taking aspirin every day is more likely to save your life or cut it short, and whether routine angioplasty works better than pills to unclog heart arteries.


… His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials. The article spelled out his belief that researchers were frequently manipulating data analyses, chasing career-advancing findings rather than good science, and even using the peer-review process—in which journals ask researchers to help decide which studies to publish—to suppress opposing views. “You can question some of the details of John’s calculations, but it’s hard to argue that the essential ideas aren’t absolutely correct,” says Doug Altman, an Oxford University researcher who directs the Centre for Statistics in Medicine.

Please read the article. And remember that research funding is a major culprit!

This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

Wise words, well worth heading

Comments (7)

Still more evidence that being a medical skeptic is the right strategy.

In line with Dr. Kirsch, the problem is not the medical literature, but the unthinking acceptance of anything and everything that is written in it. All one must do is go back and read a journal from 30, 50 or 70 years ago and see how much is wrong according to what we know today. What makes anyone think that today's literature is different. Much of it is wrong, the question is what is wrong- and therein lies the crucial question. Computers and statistics make it appear much more "scientific" but not necessarily more correct. Arrogant journal editors foster the canonization of their publications, instead of encouraging critical appraisal
Thus it ever was, and thus it will ever be.    

Very interesting, but just try telling your doctor that the studies are wrong and/or misleading when s/he is writing "orders"  based on those very studies. 
As far as the bias, what ever happened to the basic foundation of scientific studies: Trying to prove that your hypothesis is wrong?  Oy….

Kind of like democracy, peer review is the worst system… except for all other that have been tried. I'm not sure what a better alternative it.
I get more nervous that we aren't more critical of research findings, reading them with a skilled and critical eye. It was particularly disturbing that Duke's IRB refused to end their lung cancer genomic studies even after many eminent biostats experts from outside its walls repeatedly called Potti's flawed results into question. If an IRB or SRC, charged with the imminent safety of subjects (and not unimportantly, the reputations of their institutions) can't be sufficiently critical, who can be?

I'll never forget sitting in the audience as a 3rd year med student in 1978 and listening to a well-known cardiologist say the MOST important thing to do for a patient with chest pain is to start a lidocaine drip. He had statistics to prove it.
And speaking of research driven science, alarmist global warming is a current offender. It isn't surprising to read almost every week about some natural phenomenom "caused" by global warming. I have a feeling a lot of post doc fellowships get funded that way.

Statistics are like a bra – what they show is important but what they hide is vital.

[…] by a blog that I read, here is my contribution. A clinical laboratory standards group, CLSI writes laboratory […]

Post a comment