Publication Bias

Publication bias is the systematic tendency for studies with statistically significant, positive, or novel results to be submitted for publication, accepted by journals, and published more quickly than studies with null, negative, or inconclusive results. Because the published literature is the primary evidence base researchers, systematic reviewers, and policymakers draw on, this selective visibility inflates the apparent size — and can even manufacture the apparent existence — of an effect wherever it goes undetected and uncorrected.

Why it matters for evidence synthesis

A systematic review or meta-analysis is only as reliable as the body of studies it can find and pool. If studies are missing from the published record specifically because of what they found, rather than at random, the pooled effect estimate is biased in a predictable direction — usually toward overstating a treatment’s benefit or an association’s strength. This is why publication bias sits inside the risk-of-bias assessment for every well-conducted synthesis, and why it is one of the domains formally downgraded under the GRADE approach to rating the certainty of evidence. A synthesis is not just summarizing what was found; it is implicitly summarizing what was allowed to be seen.

How it happens

Publication bias is rarely a single actor’s decision. It accumulates across several points in the pipeline from finished study to indexed literature:

• Researcher non-submission — investigators are demonstrably less motivated to write up and submit a null or messy result than a clean, significant one. Robert Rosenthal named this the “file drawer problem” in 1979: completed studies with unremarkable findings are set aside rather than submitted at all.
• Editorial and reviewer preference — positive, novel, or “interesting” findings are, all else equal, judged more publishable, which shapes what authors bother submitting in the first place as much as what gets accepted.
• Sponsor and funder suppression — a trial sponsor with a commercial or reputational stake in a particular outcome may simply decline to submit an unfavorable trial for publication, even though the study was completed and reported to a regulator.
• Publication delay (time-lag bias) — negative results, when they are eventually published, tend to appear later than positive ones, so any synthesis conducted before that lag closes will over-represent early, positive findings.

The Cochrane Handbook treats publication bias as one specific mechanism within a broader family it calls reporting bias — which also includes outcome reporting bias (a published study selectively reports only some of the outcomes it measured), language bias, citation bias, and time-lag bias. Publication bias specifically concerns whether an entire study enters the literature at all, based on its results; the others concern how a study that was published gets reported, cited, or found.

Detecting it: funnel plots and Egger’s test

This section is the short definitional treatment. For the full procedural version — reading a funnel plot on a worked example, running and interpreting Egger’s test, applying trim-and-fill, the minimum-studies threshold, and a step-by-step assessment workflow — see CASRAI’s How to Detect and Assess Publication Bias guide, and the funnel plot entry for the diagnostic itself.

Because publication bias operates on which studies a reviewer can see, it cannot be measured directly — only inferred from patterns in the studies that were found. The standard visual tool is the funnel plot: a scatter plot with each included study’s effect estimate on one axis and a measure of its precision (typically standard error or sample size) on the other. In a well-behaved set of studies with no bias, small, imprecise studies scatter widely around the pooled effect while large, precise studies cluster tightly near it, producing a symmetric, inverted-funnel shape. A gap or asymmetry in the plot — commonly a missing cluster of small studies on the unfavorable side — is consistent with publication bias, but the Cochrane Handbook is explicit that asymmetry is a generic signal of “small-study effects” with several possible causes, including genuine clinical heterogeneity between small and large trials, not proof of bias on its own.

Egger’s test (Egger, Davey Smith, Schneider & Minder, BMJ 1997) formalizes the same idea as a regression: it fits a weighted linear regression of each study’s standardized effect against its precision, and tests whether the regression intercept differs significantly from zero. A significant intercept is read as evidence of funnel-plot asymmetry. The test has well-documented limits — it has very low statistical power with fewer than roughly 10 studies and is not recommended below that threshold — and, like the visual funnel plot itself, it should be interpreted alongside domain knowledge about the studies involved rather than treated as a standalone verdict.

Documented real-world scale

The clearest documented illustration of publication bias comes from comparing a published literature against a complete, results-blind record of what was actually studied. Turner and colleagues compared FDA regulatory reviews of 74 antidepressant trials (which the FDA receives regardless of outcome) against the matching published literature. The FDA classified 38 of the 74 trials as positive and 36 as negative or questionable. Of the 38 positive trials, 37 were published. Of the 36 negative or questionable trials, only 3 were published as negative — 22 were never published at all, and 11 were published in a way that conveyed a positive outcome. The published literature alone suggested about 94% of trials were positive, against the FDA’s actual 51%, and pooling only the published trials inflated the apparent effect size by roughly 32% overall relative to pooling the complete FDA dataset.

What Current Cochrane Guidance Says — and Two Things Commonly Got Wrong

The vocabulary and the cautions in this area have both moved, and a good deal of secondary writing about publication bias still reflects an earlier version of the guidance. Two points from the current Cochrane Handbook chapter on bias due to missing evidence are worth stating precisely, because getting either wrong changes what a reviewer concludes.

The ten-study rule attaches to the test, not to the plot

The threshold is widely quoted as “you need ten studies before you can look for publication bias.” What the Handbook actually says is narrower: “tests for funnel plot asymmetry should be used only when there are at least 10 studies included in the meta-analysis,” because statistical power is low below that. The constraint is on the formal test. It does not forbid drawing and inspecting a funnel plot with fewer studies — it means a non-significant test result on a small evidence base is uninformative rather than reassuring, and should not be reported as evidence that bias is absent.

The practical consequence for anyone appraising a review: a review that reports “Egger’s test was non-significant” across seven studies has not demonstrated the absence of publication bias. It has run an underpowered test.

Egger’s test is not appropriate for every effect measure

This is the caution most often omitted. The Handbook notes that Egger’s test was the first and best known of the funnel-plot asymmetry tests, but that such tests “are not recommended for application to odds ratios and SMDs because of artefactual correlations” between the effect estimate and its standard error. For odds ratios and standardised mean differences, the correlation the test relies on is partly an artefact of how those statistics are constructed, so asymmetry can appear where none exists in the underlying evidence. The Handbook points instead to the alternative tests developed by Harbord and by Peters.

Since odds ratios and standardised mean differences are two of the most common effect measures in health and social-science meta-analysis, this is not an edge case. If a review reports Egger’s test on an odds-ratio outcome, the result needs reading with that caveat attached rather than at face value.

The terminology has shifted, and the distinctions carry weight

Current Cochrane guidance frames the field around non-reporting bias as the umbrella — bias arising when decisions about reporting results are influenced by the P‑value, magnitude, or direction of those results. Within that:

• Publication bias concerns whether an entire study enters the literature.
• Selective non-reporting concerns particular results within a study being withheld on the basis of their significance or favourability — the study is published, but not all of it is.
• Small-study effects is a description of a pattern, not a cause: the tendency for intervention effects estimated in smaller studies to differ systematically from those in larger ones.

The last of these is the one that most often gets over-read. Funnel plot asymmetry indicates small-study effects — and the Handbook is explicit that small-study effects may arise from non-reporting bias, but can equally stem from methodological flaws in smaller studies, genuine clinical heterogeneity between small and large trials, or statistical artefact. Asymmetry is a prompt to investigate, not a finding of bias.

The distinction between publication bias and selective non-reporting matters operationally as well as terminologically: they have different remedies. Prospective registration of a study addresses the first, by making the study’s existence discoverable regardless of outcome. Only registration of the pre-specified outcomes, and comparison of those against what was eventually reported, addresses the second.

Mitigations built into the modern publishing pipeline

None of these mitigations eliminates publication bias on its own, but together they are the standard toolkit a research-administration office or systematic reviewer relies on:

• Prospective trial registration. ICMJE has required prospective registration in a public registry (ClinicalTrials.gov or another WHO ICTRP primary registry) as a condition of considering a trial for publication since 2005. A registered-but-unpublished trial is discoverable, which lets a systematic reviewer identify what is missing from the published record rather than being limited to what was published. CASRAI’s guide on preregistering a study protocol and its pre-registration entry cover the registry-based mechanics.
• Registered Reports address the problem upstream of registration by having a journal peer-review and conditionally accept a study’s protocol before the results are known, so the editorial decision to publish is locked in independent of whether the findings turn out to be significant.
• Venues and policies that welcome null results. A negative-result paper reports a study whose outcome did not support the tested hypothesis; some journals and publisher policies (evaluating manuscripts on methodological soundness rather than the direction of the finding) explicitly counter the incentive that produces publication bias in the first place.
• Comprehensive search strategy in systematic reviews. The PRISMA 2020 reporting guideline requires reviewers to search trial registries and grey literature, not just indexed journal databases, specifically to reduce the impact of unpublished studies on a synthesis.
• Certainty downgrading. Under GRADE, a body of evidence with credible signs of publication bias (funnel-plot asymmetry, an unusually small evidence base dominated by industry-funded positive trials, etc.) has its overall certainty rating downgraded, separate from and in addition to any point-estimate correction.

Publication bias vs. a legitimate risk-of-bias exclusion

Publication bias is easy to conflate with ordinary study selection in a review, but they are not the same failure mode. Publication bias happens upstream, before a reviewer ever sees a study, because the study’s own results affected whether it was submitted or accepted anywhere. A reviewer excluding a study from a meta-analysis because it fails an explicit, pre-specified methodological quality criterion — for example, an invalid randomization method — is a normal, transparent part of systematic review methodology, applied identically regardless of whether that study’s result was positive or null. The first silently shrinks the evidence base by result; the second visibly and symmetrically screens it by method.

Frequently asked

Do I need at least ten studies before I can assess publication bias? Not to look, only to test. The Cochrane Handbook’s guidance is that “tests for funnel plot asymmetry should be used only when there are at least 10 studies included in the meta-analysis,” because power is low below that. You may still draw and inspect a funnel plot with fewer studies — what you must not do is report a non-significant test on a small evidence base as evidence that publication bias is absent.

Can I use Egger’s test on any outcome? No. The Cochrane Handbook states that funnel plot asymmetry tests “are not recommended for application to odds ratios and SMDs because of artefactual correlations” between the effect estimate and its standard error, and points to the Harbord and Peters tests as alternatives. Since odds ratios and standardised mean differences are extremely common effect measures, this caveat applies to a large share of published meta-analyses.

What is the difference between publication bias and selective non-reporting? Publication bias determines whether a whole study enters the literature; selective non-reporting determines whether all of a published study’s results are reported. They need different fixes: prospective registration makes an unpublished study discoverable, but only registering the pre-specified outcomes — and checking them against what was eventually reported — catches outcomes that were measured and then quietly dropped.

Does a symmetric funnel plot rule out publication bias? No. Symmetry is reassuring but not conclusive, particularly with a small number of studies, where a funnel plot and Egger’s test both have limited power to detect real asymmetry.

Is publication bias the same thing as p-hacking? No. P-hacking (manipulating analysis choices within a single study until a result crosses a significance threshold) happens during analysis of one study; publication bias happens afterward, at the level of which completed studies enter the literature at all. The two can compound each other but are distinct problems with different fixes.

Can publication bias affect a whole field even if most individual studies are done well? Yes. Publication bias is a property of the literature as a body, not of any single study’s internal validity — a set of individually rigorous, honestly reported studies can still produce a badly biased summary picture if enough negative results among them were simply never published.

References

• Higgins JPT, Thomas J, et al. (eds). Cochrane Handbook for Systematic Reviews of Interventions, Chapter 13: Assessing risk of bias due to missing evidence in a meta-analysis — source for the minimum-ten-studies guidance on funnel plot asymmetry tests, the caution against applying Egger’s test to odds ratios and standardised mean differences, and the non-reporting bias / selective non-reporting / small-study effects distinctions.
• Egger M, Davey Smith G, Schneider M, Minder C. “Bias in meta-analysis detected by a simple, graphical test.” BMJ. 1997;315(7109):629–634.
• Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy.” New England Journal of Medicine. 2008;358(3):252–260.
• Page MJ, McKenzie JE, Bossuyt PM, et al. “The PRISMA 2020 statement: an updated guideline for reporting systematic reviews.” BMJ. 2021;372:n71.
• Rosenthal R. “The file drawer problem and tolerance for null results.” Psychological Bulletin. 1979;86(3):638–641.