‘It’s Science, but Not Necessarily Right’, The New York Times Sunday Review, 26 June 2011

From The New Year Times Sunday Review, 26 June 2011 (story by Carl Zimmer)

ONE of the great strengths of science is that it can fix its own mistakes. “There are many hypotheses in science which are wrong,” the astrophysicist Carl Sagan once said. “That’s perfectly all right: it’s the aperture to finding out what’s right. Science is a self-correcting process.”

If only it were that simple. Scientists can certainly point with pride to many self-corrections, but science is not like an iPhone; it does not instantly auto-correct. As a series of controversies over the past few months have demonstrated, science fixes its mistakes more slowly, more fitfully and with more difficulty than Sagan’s words would suggest. Science runs forward better than it does backward.

Why? One simple answer is that it takes a lot of time to look back over other scientists’ work and replicate their experiments. Scientists are busy people, scrambling to get grants and tenure. As a result, papers that attract harsh criticism may nonetheless escape the careful scrutiny required if they are to be refuted.

In May, for instance, the journal Science published eight critiques of a controversial paper that it had run in December. In the paper, a team of scientists described a species of bacteria that seemed to defy the known rules of biology by using arsenic instead of phosphorus to build its DNA. Chemists and microbiologists roundly condemned the paper; in the eight critiques, researchers attacked the study for using sloppy techniques and failing to rule out more plausible alternatives.

But none of those critics had actually tried to replicate the initial results. That would take months of research: getting the bacteria from the original team of scientists, rearing them, setting up the experiment, gathering results and interpreting them. Many scientists are leery of spending so much time on what they consider a foregone conclusion, and graduate students are reluctant because they want their first experiments to make a big splash, not confirm what everyone already suspects.

“I’ve got my own science to do,” John Helmann, a microbiologist at Cornell and a critic of the Science paper, told Nature. The most persistent critic, Rosie Redfield, a microbiologist at the University of British Columbia, announced this month on her blog that she would try to replicate the original results — but only the most basic ones, and only for the sake of science’s public reputation. “Scientifically I think trying to replicate the claimed results is a waste of time,” she wrote in an e-mail.

For now, the original paper has not been retracted; the results still stand.

Even when scientists rerun an experiment, and even when they find that the original result is flawed, they still may have trouble getting their paper published. The reason is surprisingly mundane: journal editors typically prefer to publish groundbreaking new research, not dutiful replications.

In March, for instance, Daryl Bem, a psychologist at Cornell University, shocked his colleagues by publishing a paper in a leading scientific journal, The Journal of Personality and Social Psychology, in which he presented the results of experiments showing, he claimed, that people’s minds could be influenced by events in the future, as if they were clairvoyant.

Three teams of scientists promptly tried to replicate his results. All three teams failed. All three teams wrote up their results and submitted them to The Journal of Personality and Social Psychology. And all three teams were rejected — but not because their results were flawed. As the journal’s editor, Eliot Smith, explained to The Psychologist, a British publication, the journal has a longstanding policy of not publishing replication studies. “This policy is not new and is not unique to this journal,” he said.

As a result, the original study stands.

Even when follow-up studies manage to see the light of day, they still don’t necessarily bring matters to a close. Sometimes the original authors will declare the follow-up studies to be flawed and refuse to retract their paper. Such a standoff is now taking place over a controversial claim that chronic fatigue syndrome is caused by a virus. In October 2009, the virologist Judy Mikovits and colleagues reported in Science that people with chronic fatigue syndrome had high levels of a virus called XMRV. They suggested that XMRV might be the cause of the disorder.

Several other teams have since tried — and failed — to find XMRV in people with chronic fatigue syndrome. As they’ve published their studies over the past year, skepticism has grown. The editors of Science asked the authors of the XMRV study to retract their paper. But the scientists refused; Ms. Mikovits declared that a retraction would be “premature.” The editors have since published an “editorial expression of concern.”

Once again, the result still stands.

But perhaps not forever. Ian Lipkin, a virologist at Columbia University who is renowned in scientific circles for discovering new viruses behind mysterious outbreaks, is also known for doing what he calls “de-discovery”: intensely scrutinizing controversial claims about diseases.

Last September, Mr. Lipkin laid out several tips for effective de-discovery in the journal Microbiology and Molecular Biology Reviews. He recommended engaging other scientists — including those who published the original findings — as well as any relevant advocacy groups (like those for people suffering from the disease in question). Together, everyone must agree on a rigorous series of steps for the experiment. Each laboratory then carries out the same test, and then all the results are gathered together.

At the request of the National Institutes of Health, Mr. Lipkin is running just such a project with Ms. Mikovits and other researchers to test the link between viruses and chronic fatigue, based on a large-scale study of 300 subjects. He expects results by the end of this year.

This sort of study, however, is the exception rather than the rule. If the scientific community put more value on replication — by setting aside time, money and journal space — science would do a better job of living up to Carl Sagan’s words.

Carl Zimmer writes frequently for The New York Times about science and is the author, most recently, of “A Planet of Viruses.”