The reproducibility paradox
Scientists are pushed to make exciting claims of novelty to get funding, says Harvard Medical School's Jeffrey Flier
First Published: Sat, Feb 04 2017. 11 17 PM IST
Jeffrey Flier, former dean of Harvard Medical School (2007-16), who currently teaches medicine at the institute, believes that many scientific papers published in biomedical science may be irreproducible.
Flier, who was in Ahmedabad to attend the Ramanbhai Foundation 8th International Symposium on current trends in healthcare organized by Cadila Healthcare, spoke to Mint on Sunday at length about why many researchers pitch their work as more innovative and exciting than it really is.
Biomedical science continues to produce rich research. At least going by the number of scientific papers that are published every year. How much of this research is reproducible, or can hold up to follow-up experiments?
There should be close to 1 million research papers published every year in thousands of journals. About 30-50% of the research done may fall in the “who cares” category. Having said that, it is hard to give a simple answer to this question. The meaning of reproducibility is not simple. Let me try and explain this.
When you publish a paper there are usually 6-8 figures that show data. So does reproducible mean, that every figure is reproducible in the exact way that you have published it? If one means that then, probably, there is a high fraction of research that is not reproducible.
But if reproducibility pertains to the most important and influential findings in the paper, to data that extends to key understandings, then the outcomes would look much better.
While a paper may be only partially reproducible, it may still advance science to a great degree. And make observations that influence many other scientists. What you are interested in is the progress of science and not merely a reproducibility score.
Expecting a perfect reproducibility index would reduce the rate of progress, slow down publishing and lower risk-taking. If you want to take risks then there is a chance of failure. You don’t want too much risk and you don’t want much failure. You want to strike a balance.
So is there a solution to the reproducibility problem that appears to plague research in this field?
The factors that push towards bad reproducibility problems need to be addressed. So, ideally, you would like the scientist to be judged on how their results are reproducible, and on the long-term impact as opposed to the short-term impact of just publishing it somewhere.
You would like to change the way some of the journals work where they work very hard to get the papers that will be picked up eventually by newspapers because that gives them readers, advertisers and all the attention.
But this is hard. I know it is not going to change by turning a switch; there are too many variables. Take the case of the National Institute of Health (NIH) in America that funds a large part of the research in bio-science. Everyone has learnt how hard it is to get an NIH grant. So people are often pushed to make exciting claims of novelty to get funded.
If you write a story that sounds more exciting than it is, then you get the money and you get the grant, only to later realize that your work isn’t as exciting. Then you start to worry about your grant renewal. And you are pushed once again to come up with a great story for your research.
So while it rare for someone to put out a complete fabrication, it is at the margins that people are making these decisions. Decisions to claim greater novelty and relevance than may exist.
Many things need to change here. From the scientists themselves to the institutions they work for, the journals and the way in which media publish research findings.
The part that I am most worried about is when people publish papers in journals that are hard to get into, they get covered by the press, but when their outcomes are proven faulty the news stories aren’t retracted.
You are suggesting that the negatives of a paper should be highlighted too?
Yes, because it is a big problem (the negatives). The phrase used is “publication bias”. It means that it is much easier to publish a paper that has a positive finding with claims to be important and transformative and all such words that people use... but it is much harder to publish a paper that is negative or says that something does not work or a paper that simply extends the previous finding.
Famous journals don’t want negative or incremental papers. This puts further pressures on scientists to come up with new, positive ideas that may be unrealistic.
What has been the impact of the internet and the digital era in the last couple of decades on reproducibility?
The frequency of finding research that is not reproducible is going up. People can use the internet and various tools to examine papers that, in the past, they were not able to.
Today, one of the most common ways to find out if papers have problems is through what is called image manipulation. There are certain common types of scientific results that are in the form of an image and now those images are all digital and it is actually pretty easy to move the image around, to change the contrast, to make it look better than it is or change the results.
So, it turns out now that there are image analysis programs that people can download and some journals are using them to review papers and check for manipulation.
How much money is spent on research every year in the US and how many of these projects see the light of day?
NIH spends roughly $35 billion and the industry spend is about $60-70 billion depending on what you include as research and development. The industry side is more applied science and NIH is more basic. Some people have estimated that half of the money spent is on research that turns out not to be true.
You can never get that to be zero because, as I said earlier, the idea that only unvarnished truth be published is impossible. You would stall progress if you try and do so.
What do you think of the newly formed Donald Trump administration? Also what is your take on India where new drug discovery and bio-medical research still remains largely undiscovered?
President Trump has said some positive things about research. He may, however, have people around him advising him to cut budget on research. So I don’t really know. I am worried because he is not predictable right now.
We have a lot of people from India doing excellent work in the field of medical science and research, so we know India has human capital to conduct outstanding research. The question is how much of it can go on in India?
From what I can see, there are major efforts to strengthen pharmaceutical development. I don’t know much about what is happening in basic science. In US, most biological scientists have to be funded by NIH for their research.
It is expensive to conduct some kinds of biological research. One needs infrastructure and proper equipment and it is difficult for an individual scientist to do it on his/her own.
In the US, NIH was growing steadily and its budget was nearly doubled from 1999 to 2004. Post-2004, its spending on research has declined, and today, many scientists are finding it difficult to support their research and some are even forced to look for other professions.
There are some rich people in the US who have made money on the internet and I have been in talks with them (for funding research at Harvard Medical School). But they want to fund some fancy new technology and not core research. I am looking for some philanthropist who would take on the personal goal of helping lay the foundation of biological science.