Limitations of Scientific Studies

There was a great New Yorker article: The Truth Wears Off, about the fact that much scientific research is not able to be replicated and bias is inherent, and the general limitations of scientific studies.

Now we depend on these studies in our situation, but we also have seen pendulums swing, and previously held scientifc truths both be debunked and re-adopted (last week's NE Journal showed that antibiotics actually help ear infections, while the knowledge of the last few years was that they should not be used.)

So, here's a link to the article

http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer#ixzz1AIkDhn6D

A couple of quotes stood out for me:

Before the effectiveness of a drug can be confirmed, it must be tested and tested again. Different scientists in different labs need to repeat the protocols and publish their results. The test of replicability, as it's known, is the foundation of modern research. Replicability is how the community enforces itself. It's a safeguard for the creep of subjectivity. Most of the time, scientists know what results they want, and that can influence the results they get. The premise of replicability is that the scientific community can correct for these flaws.

But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It's as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn't yet have an official name, but it's occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread

...........

According to Ioannidis, the main problem is that too many researchers engage in what he calls "significance chasing," or finding ways to interpret the data so that it passes the statistical test of significance-the ninety-five-per-cent boundary invented by Ronald Fisher. "The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy," Ioannidis says. In recent years, Ioannidis has become increasingly blunt about the pervasiveness of the problem. One of his most cited papers has a deliberately provocative title: "Why Most Published Research Findings Are False."

The problem of selective reporting is rooted in a fundamental cognitive flaw, which is that we like proving ourselves right and hate being wrong. "It feels good to validate a hypothesis," Ioannidis said. "It feels even better when you've got a financial interest in the idea or your career depends upon it. And that's why, even after a claim has been systematically disproven"-he cites, for instance, the early work on hormone replacement therapy, or claims involving various vitamins-"you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it's true."

That's why Schooler argues that scientists need to become more rigorous about data collection before they publish. "We're wasting too much time chasing after bad studies and underpowered experiments," he says.

...........Such anomalies demonstrate the slipperiness of empiricism. Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can't bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren't surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that's often not the case. Just because an idea is true doesn't mean it can be proved. And just because an idea can be proved doesn't mean it's true. When the experiments are done, we still have to choose what to believe. ♦

Just putting this out there as another example of why scientific studies are so important to us, and why the data keeps shifting and the inherent problems with research.

Kira