P-Curve Analysis: Testing for Evidential Value vs. P-Hacking

P-curve analysis tests whether a set of statistically significant findings in a literature has genuine evidential value or is better explained by selective reporting and p-hacking. Rather than asking “what is the average effect?”, it asks a narrower, more diagnostic question: given only the significant results a body of studies produced, is the shape of that distribution consistent with a real underlying effect, or with researchers fishing until something crossed the .05 threshold?

The method was developed by Uri Simonsohn, Leif Nelson, and Joseph Simmons and introduced in “P-curve: A key to the file-drawer,” Journal of Experimental Psychology: General, 143(2), 534–547 (2014). Two companion papers extended it: “P-curve and effect size: Correcting for publication bias using only significant results,” Perspectives on Psychological Science (2014), which adapts p-curve to estimate a bias-corrected average effect size, and “Better p-curves: Making p-curve analysis more robust to errors, fraud, and ambitious p-hacking,” Journal of Experimental Psychology: General (2015), which tightened the test against more sophisticated forms of researcher degrees of freedom. Together they define the method most people mean when they say “p-curve.”

What problem p-curve is built to solve

A pile of significant p-values in a literature is ambiguous on its own. It could mean the underlying effect is real and studies were adequately powered to detect it. It could also mean researchers ran many analyses, many outcomes, or many subgroups, and reported only the comparisons that happened to clear statistical significance — a pattern the field now generally calls questionable research practices or p-hacking. Counting “how many studies found a significant effect” cannot distinguish these two worlds, because both worlds can produce the same count. What differs is the internal shape of the p-values themselves, and that shape is what p-curve examines.

The core logic: shape, not count

P-curve restricts attention to the p-values that are themselves statistically significant (conventionally p < .05) for the test of interest across a set of studies, and asks how those p-values are distributed within that range.

• Right-skewed distribution — more p-values cluster near .01 than near .04. This is what a true, non-null effect produces: a real effect pushes p-values toward zero as a matter of statistical theory, so a right skew among only-the-significant results is consistent with genuine evidential value.
• Flat distribution — p-values are roughly uniform across .00–.05. Under the null hypothesis of no true effect, a p-value is uniformly distributed on [0,1] by construction, so a flat curve among significant results is exactly what you would expect if the null were true and only the results that happened to land below .05 were reported.
• Left-skewed distribution — more p-values cluster near .04–.05 than near .01. This is the more diagnostic red flag: it is difficult to produce by chance under a true effect, and is the signature p-curve’s authors associate most directly with p-hacking — researchers stopping data collection, adding covariates, or trying alternative outcome measures until a result barely crosses the threshold, then stopping.

P-curve formalizes this visual intuition with two statistical tests run against the null of “no evidential value”: one testing whether the observed curve is right-skewed (evidence for a real effect) and one testing whether it is flatter than a uniform distribution or left-skewed (evidence the results are p-hacked or that there is no true effect at all). A literature can fail both, pass both, or land ambiguously in between — p-curve reports the tests, it does not hand back a single up-or-down verdict.

Building a p-curve, step by step

1. Define the effect of interest precisely. P-curve is only valid for a set of test results that all bear on the same hypothesis. Mixing unrelated effects into one curve produces an uninterpretable average.
2. Select every study that tested that hypothesis, on criteria independent of the result obtained — this is the method’s single most important, and most frequently violated, precondition (see Limitations below).
3. Extract the exact test statistic behind each significant result (t, F, r, chi-square, or z) rather than reading off a reported “p < .05,” since p-curve needs the precise p-value, not a thresholded one.
4. Retain only the p-values below .05 for the test of the focal hypothesis in each study — non-significant results and tests of other hypotheses are excluded by design.
5. Submit the set to the p-curve app (the original tool at p-curve.com, built by the method’s authors) or to an equivalent implementation in R, which plots the distribution and runs the right-skew and flatness/left-skew tests.

How it differs from funnel plots and Egger’s test

Funnel plots and Egger’s regression test are the older, more established tools for this territory, and they ask a related but distinct question: whether small studies with less precise (larger) standard errors report systematically different effect sizes than large, precise studies — the classic signature of the file-drawer problem, where small null or contrary studies never get submitted or published. Egger’s test regresses the standardized effect against precision and looks for a significant intercept as evidence of asymmetry.

P-curve asks a different question with a different vulnerability profile:

• Funnel-plot asymmetry can arise from genuine heterogeneity between small and large studies, not just from publication bias — a documented limitation of that family of methods, and it also has very low statistical power below roughly ten included studies.
• P-curve does not rely on comparing large studies to small ones at all — it only needs the significant p-values themselves, so it can be applied even when the literature consists mostly of similarly-sized studies where a funnel plot has little to work with.
• P-curve is explicitly diagnostic of p-hacking within individual studies (selective analysis choices inside a single reported test), whereas funnel plots and Egger’s test are diagnostic of selective publication across studies (whole studies never seeing print). These are related but non-identical failure modes, and a literature can show one without the other.

Because the two families answer overlapping but non-redundant questions, methodologists generally treat them as complements rather than substitutes: a systematic review with a large enough set of studies can run a funnel plot and Egger’s test for cross-study selective publication, and a p-curve on the subset of significant, hypothesis-relevant tests, for within-study selective reporting. Neither one alone rules out the other failure mode. See CASRAI’s overview of detecting and assessing publication bias for the funnel-plot and Egger’s-test mechanics in full, and meta-analysis in R with metafor for the funnel-plot/regression-test implementation.

Limitations and the ongoing debate

P-curve’s authors and independent methodologists have both identified real constraints on the method:

• Sensitivity to study selection. The method’s validity rests entirely on selecting studies by whether they tested the hypothesis of interest, not by whether they produced a result the analyst finds convenient. Including a study because its result “fits the story,” or excluding one because it doesn’t, reintroduces exactly the selective-reporting problem p-curve exists to detect — the tool cannot tell the difference between a genuinely pre-specified set of tests and a cherry-picked one; that discipline has to come from the analyst’s selection protocol, ideally pre-registered or documented independently of the outcome.
• One test statistic per study. When a single study contributes multiple significant tests of the same underlying hypothesis, the original guidance is to select one focal test per study (the authors’ own instructions specify this) rather than treat every reported p-value as an independent data point — get this wrong and effectively over-weight studies that ran more tests.
• Robustness to more sophisticated p-hacking. The original 2014 test was shown to be vulnerable to more “ambitious” forms of p-hacking that manufacture a right-skewed rather than left-skewed curve; the 2015 “Better p-curves” paper exists specifically to address this and describes the conditions under which the original test can be fooled.
• Effect-size estimation is a separate, more contested extension. The 2014 Perspectives on Psychological Science companion paper extends p-curve to estimate a publication-bias-corrected average effect size from the curve’s shape, and this application in particular has drawn methodological debate distinct from the diagnostic (evidential-value) use case — critiques in the broader meta-analysis literature have raised concerns about how p-curve-based and related p-value-only effect-size estimators (including the closely related p-uniform method) perform when the true effect size varies meaningfully across the included studies, since the estimator assumes a single common effect. Readers using p-curve for effect-size correction, rather than the simpler evidential-value verdict, should treat that application as more actively debated than the core right-skew/flat test.
• It says nothing about studies never run or never submitted. Because p-curve only ever sees significant results that made it into the analyzed set, it cannot detect a hypothesis that was tested, came out null, and was never written up at all — that failure mode is squarely funnel-plot/Egger’s-test territory, not p-curve’s.

When to reach for p-curve

P-curve is most useful when you have a defined set of studies (or analyses within studies) that plausibly all tested the same effect, and the question is specifically “does this literature show signs of p-hacking or selective analysis,” rather than “is the pooled effect size biased.” It is a natural companion to a systematic review or meta-analysis, run alongside — not instead of — funnel-plot/Egger’s-test checks for cross-study publication bias, and alongside a look at outcome switching and selective outcome reporting for trial-registry-detectable selection. It pairs conceptually with registered reports, which prevent the problem p-curve is built to detect by fixing the analysis plan before results are known, and with a solid grounding in what a p-value does and doesn’t mean, since misreading individual p-values is exactly the raw material p-curve’s diagnostic works on.

Frequently asked questions

Does a right-skewed p-curve prove an effect is real?

It provides evidence consistent with evidential value, not proof. A right skew rules out “these results are entirely an artifact of selective reporting with no true effect at all” as the sole explanation; it does not establish the size of the true effect, its generalizability, or that every included study individually replicated it.

Can p-curve be run on a single study?

Formally p-curve requires a set of tests of the same hypothesis, but that set can come from within one paper that reports several tests of the same effect (for example, several experiments in one article), not only from a set of separate published papers. What matters is that all the p-values feed into the same underlying hypothesis and are selected on that basis, not on their outcome.

Is p-curve a replacement for a funnel plot?

No. They detect different failure modes — p-curve targets selective analysis within reported significant results, funnel-plot/Egger’s-test methods target whole studies missing from the literature entirely. Use both where the study count and data allow it; neither substitutes for the other.

What software runs a p-curve analysis?

The method’s authors maintain a browser-based app at p-curve.com that accepts pasted test statistics and outputs the disclosure table, the plotted curve, and both statistical tests. R implementations also exist that reproduce the same underlying tests for users who need it integrated into a scripted analysis pipeline.

Does p-curve replace judgment about which studies to include?

No, and this is its central limitation. P-curve is only as valid as the study-selection protocol feeding it; the analysis itself cannot detect or correct for a biased selection of which significant results were handed to it in the first place.