Replication and Reproducibility in Health Research

Why one study is not the end

A single exciting result is a starting point. Science builds confidence when other teams, with new data, get consistent answers. This page explains two terms that are often confused, summarizes what replication projects have found, and shows how to use the idea when you read about an emerging health technology. It is education, not medical advice.

Two different words

The US National Academies of Sciences, Engineering, and Medicine (NASEM) 2019 report defines reproducibility as computational reproducibility: obtaining consistent computational results using the same input data, steps, methods, code and conditions of analysis. It defines replicability as obtaining consistent results across studies aimed at the same question, each of which has obtained its own data. [1]

In short, reproducibility uses the original data and code, while replication needs new data collection. [1] The report adds a third idea, generalizability: how far a result applies in other settings. [1] Some people use the words the other way around, so check how a source defines them.

What large replication projects found

• Psychology. The Open Science Collaboration repeated 100 studies from three psychology journals. Replication effects were about half the size of the originals, and 36% of replications had statistically significant results compared with 97% of the originals. [2] This is psychology, not medicine, but it shows how a field-wide pattern can look.
• Laboratory cancer biology. The Reproducibility Project: Cancer Biology repeated 50 experiments from 23 papers (158 effects). For positive effects, the median replication effect was 85% smaller than the original, and 92% of replication effect sizes were smaller. Counting positive and null effects together, 46% (51 of 112) replicated by the project's measures. [3] These were experiments in cells and animals, so they say nothing direct about treatments in people.
• Clinical research. Among 49 highly cited clinical studies, 16% were later contradicted and another 16% had found stronger effects than later studies. [4]

The Cancer Biology authors caution that a successful replication does not definitively confirm an original finding, and a failure does not disconfirm it; it signals that more investigation is needed. [3]

Why results fail to repeat

No single cause explains it. One widely discussed analysis argued that a finding is less likely to be true when studies in a field are smaller, effect sizes are smaller, many relationships are tested, designs and analyses are flexible, and financial interests or competition to reach significance are greater. [5] That is an argument based on modeling, not a measured rate of error for any particular field.

Smaller studies also tend to report bigger effects. Across 93 meta-analyses, treatment effects averaged 48% larger in trials with fewer than 50 patients than in trials with 1,000 or more. [6] See why promising findings sometimes fail.

Using this when you read about emerging science

Several publications from one trial are not independent replications. Neither are repeated analyses of the same dataset.

Nothing here means science is broken or that every result is wrong. It means confidence should rise with repeat evidence. For personal health decisions, talk with a clinician.