Reproducibility audit

A reproducibility audit is a check of computational reproducibility, not a new experiment: an independent analyst attempts to regenerate a study’s published numbers and figures from its deposited data and code, using the same data and the same analytic pipeline the original authors used. This is a narrower question than whether the finding itself is real — an audit can succeed (the code faithfully reproduces the reported numbers) even if the underlying study design is weak, and it can fail for purely mechanical reasons, such as a missing dependency version or an undocumented manual step, with no implication that the finding is wrong.

Reproducibility audits differ in scale and purpose from several adjacent practices this site covers separately. A replication study collects new data to test whether the finding itself holds up, rather than re-running existing code on existing data. Inferential reproducibility concerns whether an independent analyst reaches the same substantive conclusion from the same data using a different, defensible analytic approach, which a mechanical audit does not test. Crowdsourced replication and many-analysts studies distribute the same dataset to multiple independent teams to see how much conclusions vary by analyst, which is a different design again from a single audit pass against the original authors’ own pipeline.

Programme-level audits operate at two different scales. Journal- or funder-run pre-publication verification, such as the American Economic Association’s Data Editor checking every accepted manuscript’s code and data before publication, catches problems before they enter the record. Post-publication, field-wide audits, such as Hardwicke and colleagues’ reproducibility checks of published psychology papers, sample already-published work to estimate how much of a field’s literature is actually computationally reproducible as deposited. Outcomes are typically reported categorically — fully reproducible, reproducible with minor deviations, partially reproducible, or not reproducible — often broken down by the specific point of failure: missing data, missing code, code that runs but produces different numbers, or environment and dependency drift.

From a research-administration standpoint, a reproducibility audit is what most funder and journal code-and-data-availability policies are actually designed to make possible, and what a reproducible research practices programme is meant to prevent from failing in the first place: depositing code and data satisfies the letter of an availability policy, but only an audit actually tests whether what was deposited works.