For those that have not read it yet, The Atlantic recently featured an article profiling Dr. John Ioannidis who has made a career out of falsifying many of the findings of medical research that guides clinical practice. Ioannidis’ research should cause us all to appreciate the various bias we may bring to our own work:
[C]an any medical-research studies be trusted?
That question has been central to Ioannidis’s career. He’s what’s known as a meta-researcher, and he’s become one of the world’s foremost experts on the credibility of medical research. He and his team have shown, again and again, and in many different ways, that much of what biomedical researchers conclude in published studies—conclusions that doctors keep in mind when they prescribe antibiotics or blood-pressure medication, or when they advise us to consume more fiber or less meat, or when they recommend surgery for heart disease or back pain—is misleading, exaggerated, and often flat-out wrong. He charges that as much as 90 percent of the published medical information that doctors rely on is flawed. His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences. Given this exposure, and the fact that his work broadly targets everyone else’s work in medicine, as well as everything that physicians do and all the health advice we get, Ioannidis may be one of the most influential scientists alive. Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem. [my emphasis]
Unlike most famous researchers, Ioannidis is not famous for a positive discovery or finding (unless you count his mathematical proof that predicts error rates for different methodologically-framed studies). Instead, his status has been obtained because of his ability to falsify the work of others–to take their hypotheses and empirical research and show that they are wrong.
This is highly unusual, not only in the area of medical research, but in most academic disciplines. The article notes that researchers are incentivized to publish positive findings–preferably paradigm altering ones–and this leads to a breakdown in the scientific method. As Karl Popper so famously argued, knowledge accumulates based on the testing of theories that are then subjected to replication by other researchers. If the original findings are falsified–meaning that the evidence does not support the theory–the theory is scrapped and replaced with a new theory that has greater explanatory power. Knowledge is built through the cumulative falsification of theories. One can think about falsification as the successive chipping away at a block of stone–the more we chip away the closer we get to an actual form. If researchers are not incentivized to pursue falsification we all lose as a result, since incorrect findings are not vigorously retested and challenged. According to Ioannidis, if they are challenged it is often years–if not decades-after they have been generally accepted by research communities.
It would appear that Theodore Roosevelt was not entirely correct. The critic should, in fact, count a great deal.
[Cross-posted at Signal/Noise]
I remember one of my undergraduate professors saying of a particular IR scholar, with great admiration, that he was only one of the few people in the field willing to publish statistical findings that simply refuted widespread assumptions. Indeed, our unwillingness to publish such papers without findings in which p>.05 says a lot about the “scientific” character of IR…..