Reproducible AI experiment

A reproducible AI experiment is the machine-learning-specific application of computational reproducibility: an independent party re-runs the released training and evaluation code, on the released data, with the released configuration, and obtains results that match the reported ones, either exactly or within the variability the original authors themselves reported across seeds. It differs from a general reproducibility audit mainly in what “matching” is allowed to mean, because ML experiments carry sources of legitimate non-determinism that most other computational research does not.

The obstacles are structural, not just researcher diligence. Stochastic initialisation and data-shuffling mean two runs of identical code on identical data can legitimately differ. Floating-point arithmetic is hardware- and library-version-dependent, so a GPU-trained model may not reproduce bit-for-bit on different hardware, and distributed training adds its own non-determinism from operation ordering across devices. A growing share of published work also depends on closed, versioned commercial model APIs whose weights can change or be withdrawn, making the original experiment permanently un-repeatable regardless of disclosure.

Because exact numerical equivalence is often not achievable even in good faith, ML venues have converged on a weaker but checkable bar: results falling within the reported variability across a stated number of random seeds, rather than an exact match. Pineau and colleagues’ widely cited 2021 paper in the Journal of Machine Learning Research on improving reproducibility in ML research, and the NeurIPS reproducibility checklist that grew out of it, formalised this into concrete disclosure requirements: report the number of seeds run and the variance across them, specify the hardware and software environment (a container image or equivalent), and release the full training and evaluation pipeline, not only the final model weights.

This is a narrower, engineering-oriented bar than replicability in the NASEM sense, which asks whether an independent team collecting genuinely new data reaches the same substantive conclusion. A reproducible AI experiment can be perfectly reproducible in the seed-variance sense while the underlying claim — that the architecture or method generalises beyond the benchmark it was tested on — remains untested. Reviewer checklists and reproducibility requirements at venues such as NeurIPS, ICML, and ICLR are aimed squarely at the narrower, checkable claim, precisely because the broader generalisation claim is much harder to adjudicate at review time.