Examples
Worked examples
- Is an instance
Botvinik-Nezer et al. (2020) gave 70 independent teams the identical neuroimaging dataset and set of hypotheses; the teams reached materially different conclusions on several hypotheses despite analysing exactly the same data
Counter-examples
Looks similar, but isn't
- Not an instance
A conventional replication study, where a new team collects new data to test whether a prior finding holds up, is not a many-analysts study — a many-analysts design holds the data fixed and varies only the analysts and their analytic choices
Editorial commentary
A many-analysts study is a research-methods design in which multiple independent teams are given the identical dataset and research question and asked to analyse it separately, without coordinating with one another. Because the data and the question are held fixed, any variation in the teams’ conclusions can be attributed specifically to differences in analytic choices — model specification, covariate selection, exclusion criteria, and similar researcher degrees of freedom — rather than to differences in what was measured.
What these studies have found
Several published many-analysts studies have documented substantial heterogeneity in conclusions even under these controlled conditions: Silberzahn et al. (2018) gave 29 teams the same dataset and question about football referees’ red-card decisions and skin colour, and teams reached a wide range of effect-size estimates and conclusions; Botvinik-Nezer et al. (2020) found comparable variability across 70 teams analysing the same neuroimaging dataset; Breznau et al. (2022) extended this to a large-scale crowdsourced comparative political-economy analysis with a similarly wide spread of results.
Why this matters for research integrity, not just methods
These findings are used as direct evidence of the scale of researcher degrees of freedom — the range of defensible analytic choices available in most real datasets — and are cited on both sides of a debate about single-analysis publication: some argue the results show any single published analysis should be treated with real caution about its dependence on unstated choices, while others argue the variation reflects legitimate differences in analytic judgment rather than evidence that any one analysis is wrong. Unlike p-hacking, disagreement between analysts in these studies is not attributed to misconduct — the teams were working in good faith and disclosed their methods.
Practical responses
Structural responses informed by this body of evidence include reporting results across a multiverse analysis or specification curve rather than a single model, pre-specifying the primary analysis before seeing the data, and treating single-team, single-pipeline results with appropriate caution when a finding has not been checked against alternative reasonable specifications.
References
- Silberzahn et al. (Advances in Methods and Practices in Psychological Science, 2018); Botvinik-Nezer et al., ‘Variability in the analysis of a single neuroimaging dataset by many teams’ (Nature, 2020); Breznau et al. (PNAS, 2022).
Also known as
multi-analyst study · crowd analytics
Machine-readable encodings
Use in your systems
<role vocab="credit"
vocab-identifier="https://casrai.org/dictionary/"
vocab-term="Many-analysts study"
vocab-term-identifier="https://casrai.org/dictionary/term/many-analysts-study" />{
"@context": "https://schema.org",
"@type": "DefinedTerm",
"@id": "https://casrai.org/dictionary/term/many-analysts-study",
"name": "Many-analysts study",
"identifier": "https://casrai.org/dictionary/term/many-analysts-study",
"description": "A many-analysts study is a research-methods design in which multiple independent teams are given the identical dataset and research question and asked to analyse it separately, used to directly measure how much a study’s conclusions vary purely as a function of analytic choices, holding the data and question constant.",
"inDefinedTermSet": "https://casrai.org/dictionary/domain/reproducibility#set",
"url": "https://casrai.org/dictionary/term/many-analysts-study",
"sameAs": [
"multi-analyst study",
"crowd analytics"
],
"license": "https://creativecommons.org/licenses/by/4.0/",
"publisher": {
"@id": "https://casrai.org/#organization"
},
"author": {
"@id": "https://casrai.org/#editorial-team"
},
"datePublished": "2026-05-21T02:22:50",
"dateModified": "2026-08-22T15:39:35",
"inLanguage": "en-GB",
"isAccessibleForFree": true
}
