IN BRIEF
The workflow was applied to 4,452 replication packages from five economics journals.
The authors report 3,460 papers or appendices with flagged discrepancies, more-than-tenfold speedups in 496 and proposed extensions in 923.
A discrepancy can reflect rounding or reproducibility problems; scientific significance requires case-by-case analysis.
WHY IT MATTERS
Reproducing research code requires data, software and interpretation. Screening thousands of packages could help researchers focus their attention on cases that need investigation.
Automated screening is not a verdict on the reliability of economics research. Finding a mismatch is different from establishing a false conclusion.
WHAT IS NOW POSSIBLE
Before: most checks were small-scale and labour-intensive; now: an automated workflow has been reported at a 4,452-package scale.
WHAT MADE IT POSSIBLE
The selected journals archive replication packages with code and data, subject to access restrictions.
An LLM workflow coordinates reruns, comparisons with published values, computational optimisation and suggestions for further analysis.
THE STORY
Three researchers set a language-model workflow a demanding task: attempt to rerun calculations from thousands of published economics papers. Their September 2026 NBER working paper describes tests on 4,452 replication packages from five journals. The system flagged discrepancies in 3,460 papers or appendices. The meaningful development is the scale of automated checking, not a verdict that thousands of studies are invalid.
A different recalculated number may reflect rounding, software versions, incomplete instructions or a mistake that changes the finding. Those possibilities need to be separated before concluding that a scientific claim is wrong. The tool, according to its authors, is a way to identify cases that deserve closer examination.
Matthew Schwartz, Isaiah Andrews and Jesse M. Shapiro describe three functions. First the workflow attempts to reproduce calculations and compare them with the published text. Second it searches for faster equivalent implementations. In 496 articles, the authors report that at least one calculation became more than ten times faster at similar or greater accuracy under their evaluation.
The third function proposes analyses compatible with the original paper's assumptions and research goals. The team reports 923 proposed extensions absent from the initial papers. An extension proposal is not automatically an independently verified discovery. Restricted data, software dependencies and methodological judgment also limit automated replication.
The release remains a working paper, not an independent certification of all flags. NBER discloses that Schwartz worked as an Anthropic contractor during the project and that LLMs were used in analysis and writing. The author-reported demonstrated consequence is a large-scale screening process; the reliability and scientific significance of individual alerts require further validation.
WHAT WE KNOW
NBER Working Paper 35782 reports a workflow used with 4,452 replication packages. [1]
The authors report 3,460 flagged discrepancies, 496 computation speedups and 923 proposed extensions.
An author disclosed a contractor relationship with Anthropic.
WHAT WE DON'T KNOW
The abstract does not independently verify each flagged instance.
Not every numerical difference materially changes a scientific conclusion.
Results on other disciplines and less structured data are not established.
WHAT HAPPENS NEXT
Classify flags by size, reproducibility and effect on published conclusions.
Enable outside groups to rerun the workflow and validate representative samples.
Evaluate the workflow as an aid to scientific review, not a replacement for judgment.
SOURCE TRACE
The authors report tests on 4,452 replication packages.
The workflow flagged discrepancies in 3,460 papers or appendices.
More-than-tenfold speedups are reported in 496 papers.
Proposed analytical extensions are reported for 923 papers.
SOURCES
- An LLM Workflow That Reproduces, Improves, and Extends Published Economics ResearchOpen source
- An LLM Workflow That Reproduces, Improves, and Extends Published Economics ResearchOpen source
CURRENT EVIDENCE
EARLY EVIDENCE
COMPUTATIONAL RESEARCH / PRELIMINARY
WHAT WOULD CHANGE THIS?
- An independent audit could validate or narrow the material discrepancy count.
- Public reproducibility would strengthen the claimed speedups.
- Testing different scientific corpora would establish wider applicability.