The vast majority of findings published in high-profile biomedical research publications can’t be reproduced by independent laboratories. Multiple groups have come to this same shocking conclusion in recent years, and it has deservedly generated considerable attention.
It matters because the ability to reproduce a result is fundamental to establishing the legitimacy of a new research finding. These early-stage results provide the foundation for studies that will ultimately be performed in people. Furthermore, the time invested in attempting to confirm an irreproducible result has a real opportunity cost; it distracts scientists and physicians from other more fruitful areas of endeavor.
It is important to realize that this debate, highlighting the deficiencies our scientific process, is taking place openly within the scientific community. This is a sign of the strength of our scientific system.
What constitutes ‘reproducibility’?
Each scientific publication has several component parts that fit together to drive an overall conclusion. This conclusion is typically summarized in one ‘big idea’ that is captured in the title of the scientific paper. While many people think peer review is designed to screen out flawed experiments, that’s not really what it does. Peer reviewers don’t have the time or inclination to repeat the experiments themselves to check accuracy—they mainly just read the manuscripts to make sure the scientists are drawing reasonable conclusions based on the data they’ve gathered.
At one level, a system for checking on scientific reproducibility could demand that each component be reproduced in specific detail. However, this demand seems unrealistic, particularly in biological systems where natural variation is expected. On the other hand, it seems completely reasonable that the ‘big idea’ or major conclusion should withstand close scrutiny. So while there may be considerable variability between individual experiments, the general conclusion should be able to be substantiated.
What has shocked so many in biomedicine the last couple years isn’t that many investigators are unable to reproduce the specific details of their experiments. The shocking part is that many cannot even confirm the ‘big idea’ when their experiments were performed blinded. Even the title of their paper was not confirmed.
Having discovered that investigators themselves could not confirm their own work, a review of the details of individual experiments provided the explanation. Experiments were not performed using what most would regard as standard scientific methodology (blinding researchers to their data during experiments; repeating experiments; reporting all results; use of appropriate controls; avoiding inappropriate data-selection or “cherry picking” data after an experiment is done; appropriate use of statistics).
What if anything, has changed?
The inability to substantiate key research findings is not just a recent phenomenon. It also seems unlikely that this is peculiar to biomedical research.
In fact, over the years, a key aspect of scientific meetings were conversations that took place outside the conference room, where
