Discipline by Discipline: A Data-Driven Audit of American Science's Reproducibility Record

Discipline by Discipline: A Data-Driven Audit of American Science's Reproducibility Record

Science's most uncomfortable audit is ongoing, and the numbers are not flattering. Over the past fifteen years, coordinated replication efforts have produced a growing body of empirical evidence about which fields of American research can withstand independent scrutiny — and which cannot. The results, available through publicly accessible project datasets, reveal a landscape that is uneven, politically complicated, and in some corners, quietly improving. What the data does not support is complacency.

This article draws on published findings from the Open Science Collaboration's Reproducibility Project: Psychology, the Social Sciences Replication Project, the Reproducibility Project: Cancer Biology, and several economic replication initiatives to construct a field-by-field account of where American science stands.

Psychology: The Field That Started the Conversation

The 2015 report from the Open Science Collaboration remains the most cited document in the replication literature. Researchers attempted to reproduce 100 psychological studies published in top-tier journals between 2008 and 2010. Only 36 replicated successfully using the original significance threshold. When the criterion shifted to effect size — a more forgiving but arguably more meaningful standard — roughly 47 percent showed effects in the same direction with comparable magnitude.

The raw failure rate generated significant controversy, but the dataset itself is more nuanced than the headlines suggested. Studies with larger original sample sizes replicated at higher rates. Effect sizes in original papers were systematically inflated relative to replication results, a pattern consistent with publication bias rather than fraud. Cognitive psychology replicated more reliably than social psychology, a distinction that has since become a recurring theme in reform discussions.

Since 2015, the field has made measurable structural changes. Pre-registration rates in psychology journals have risen substantially. The American Psychological Association and several major journals now require or strongly encourage registered reports, a format that commits editors to publish results regardless of outcome. The data on whether these reforms are working is still accumulating, but early registered-report studies show effect sizes roughly half the magnitude of historically published work — which researchers interpret not as failure but as correction.

Nutrition: A Field Resistant to Its Own Evidence

Nutritional epidemiology presents a different and more stubborn problem. Unlike psychology, the field has not organized a systematic replication initiative, which is itself a data point worth noting. The closest approximation comes from umbrella reviews — meta-analyses of meta-analyses — that have repeatedly found that nutritional associations in observational studies rarely hold up under prospective, controlled conditions.

John Ioannidis and colleagues have published extensively on this pattern, finding that a substantial proportion of nutritional claims that achieve media traction are based on observational data with inadequate confounder control, small effect sizes, and no replication requirement prior to publication. A 2013 analysis in the American Journal of Clinical Nutrition found that of 53 ingredients assessed from a random cookbook, 40 had been studied in relation to cancer risk — and the associations were frequently contradictory across studies.

The structural incentive here is funding-driven. Randomized controlled trials in nutrition are expensive, logistically difficult, and commercially unattractive to industry sponsors when the target nutrient is not a patentable compound. The result is a field that generates enormous public-facing claims from a methodological foundation that rarely supports them. There is no replication project for nutrition science because the field has not yet accepted that it needs one.

Economics: High Stakes, Slow Reform

Economics entered the replication conversation later than psychology but has moved with notable institutional seriousness. The Journal of Money, Credit and Banking conducted one of the earliest economics replication projects in the 1980s, finding that a majority of empirical results could not be reproduced even with access to original data. More recent initiatives have produced similarly sobering findings.

The Economic Research Replication Project, coordinated through the Institute for Replication, has systematically re-examined studies published in top economics journals. Across multiple rounds, roughly 50 to 60 percent of findings replicate in the strict sense — meaning the original result is recovered using original code and data. That figure drops when researchers introduce reasonable alternative specifications, a procedure sometimes called robustness testing.

Economics has two features that complicate replication differently from other fields. First, data access has historically been inconsistent; many flagship results were produced on proprietary or restricted datasets that independent researchers could not obtain. Second, the field relies heavily on natural experiments — policy changes, geographic discontinuities, historical events — that by definition cannot be re-run. Replication in this context means specification testing and out-of-sample validation rather than literal repetition.

Several top journals, including the American Economic Review, now mandate data and code deposits as a condition of publication. Compliance has improved, though the quality of deposited materials varies substantially. The infrastructure for replication exists; the culture of expecting it is still developing.

Biomedical Research: Large Stakes, Fragmented Record

The Reproducibility Project: Cancer Biology set out to replicate 50 high-impact studies published between 2010 and 2012. After confronting access barriers, material unavailability, and unresponsive original authors, the project completed replications on 23 studies. Of those, fewer than half reproduced the original effect at comparable magnitude.

The biomedical record is complicated by the distinction between conceptual and direct replication. Many researchers in the life sciences argue that biological systems are inherently variable and that exact replication is less meaningful than independent confirmation of a mechanism. This is a legitimate methodological position, but it has also functioned as a shield against accountability. When "conceptual replication" becomes the standard, almost any positive result in a related area can be counted as confirmation.

Preclinical research — the animal and cell-based studies that precede human trials — shows particularly high failure rates when subjected to independent testing. Estimates from industry sources suggest that a significant proportion of published preclinical findings cannot be reproduced internally before a compound enters trials. Bayer and Amgen have both published internal analyses suggesting replication rates in the range of 20 to 25 percent for externally published preclinical oncology results.

What the Data Collectively Reveals

Across these disciplines, several structural patterns emerge consistently. Fields with higher average sample sizes replicate more reliably. Studies that report larger effect sizes in original publications tend to show greater shrinkage upon replication — a signature of selective reporting. Fields that have adopted pre-registration and mandatory data sharing show early signs of improvement, though the time horizon for assessing cultural change is long.

The incentive architecture remains the central problem. Tenure, funding, and journal prestige are still disproportionately tied to novel positive findings. Replication studies, when they are published at all, appear in lower-impact venues and generate less professional recognition for their authors. Until that calculus changes, reform will remain incremental.

The datasets that document these failure rates are public, accessible, and underutilized by the broader research community. Researchers designing new studies, funding agencies evaluating proposals, and institutions building promotion criteria all have access to this evidence. The question is whether they choose to act on it.

Science holds itself to a standard of evidence that it has not consistently met when turned inward. The replication data is the mirror. What happens next depends on whether the disciplines in question are willing to look at it honestly.