Evidence Synthesis Methods: The Review Families

The mistake that costs review teams the most time is choosing a label first and working out the methods afterwards. Someone decides the project is “a scoping review” because a full systematic review sounds unaffordable, and only at write-up discovers that the question was an effectiveness question that a scoping review is structurally unable to answer. The names are not reliable enough to reason from. Grant and Booth’s 2009 typology of 14 review types concluded that “few review types possess prescribed and explicit methodologies and many fall short of being mutually exclusive”—and a decade later Sutton and colleagues, cataloguing 48 review types across seven families, found the same problem persisting, with several pairs of names used interchangeably despite no notable difference in method.

So work the other way round. Decide what your question requires at each of four stages—search, appraisal, synthesis, analysis—then find the family whose conventions match, and only then pick the label you will put in the title. This page walks that sequence, family by family, with the commitments each one silently imposes.

Start with what the review must deliver

Three properties of the question determine almost everything downstream:

Is there a comparison? A question of the form “does X work better than Y for outcome Z” needs comprehensive retrieval, risk-of-bias assessment, and a defensible synthesis. That is the systematic review family, no matter how little time you have.
Is the unit of analysis a primary study or an existing review? If enough systematic reviews already exist, synthesising them is a different design with different search sources and a distinctive failure mode (see the overlap problem below).
Is the output a map or a verdict? Mapping what exists, clarifying a concept, or sizing a literature before committing to a full review are legitimate ends in themselves—and they license a different set of shortcuts than a verdict does.

The last of these is where most disputes with reviewers and supervisors originate. Our comparison of literature review vs. systematic review vs. scoping review covers that three-way fork at the level of individual deliverables.

The seven review families

Sutton, Clowes, Preston and Booth (2019) grouped 48 identified review types into seven families based on shared features. Their grouping is the most useful working map available, because families—unlike individual type names—are stable enough to plan against.

Traditional reviews. Narrative, critical, integrative, state-of-the-art and similar reviews. Generally purposive rather than exhaustive sampling; all use bibliographic database searching, but methods are frequently not stated explicitly. Sutton et al. note a clear drift in this family towards more systematic conduct and transparent reporting.
Systematic reviews. Comprehensive searching is the defining feature. Multiple bibliographic databases are essential—guidance in this family always requires more than one. Study-type focus (observational studies, economic evaluations, diagnostic accuracy) is often carried in the review’s name.
Review of reviews. Unique in prioritising one study type: existing systematic reviews and evidence syntheses rather than primary studies. Search sources shift to databases that index reviews and to systematic-review search filters, plus PROSPERO for reviews still in progress.
Rapid reviews. Grouped by how they abbreviate conventional systematic review methods. Sutton et al. identify the defining feature not as speed but as the negotiation between the review team and the commissioner about which methods will be curtailed.
Qualitative systematic reviews. Meta-ethnography, thematic synthesis, framework synthesis, critical interpretive synthesis and the rest. Separated from the systematic review family because search logic changes: interpretive reviews may sample theoretically rather than comprehensively.
Mixed-methods reviews. Reviews that include mixed-methods primary studies, or—more commonly—that integrate quantitative and qualitative data from separate studies.
Purpose-specific reviews. The hardest family to characterise: health technology assessments, concept analyses, scoping reviews, mapping reviews. Tailored so tightly to a single purpose that the design does not transfer.

SALSA: the four commitments that actually define a review

Grant and Booth analysed their 14 types against a simple analytical framework—Search, AppraisaL, Synthesis and Analysis. It remains the most practical way to specify a review before naming it, because it forces four separate decisions that a single label blurs together:

Search. Comprehensive, purposive, or theoretical? How many databases; grey literature or not; supplementary methods (reference checking, citation searching, hand searching, contact with authors) mandatory or discretionary?
Appraisal. Formal risk-of-bias assessment on every included study, a quality threshold for inclusion, or none at all?
Synthesis. Tabulation and narrative description, thematic or interpretive synthesis, or statistical pooling?
Analysis. What is actually produced—a map of what exists, a summary effect estimate, a programme theory, an explanation of why an intervention worked in some contexts and not others?

Write those four down before you write the protocol. If your four answers do not match any family’s conventions, you have either designed something novel that you must justify explicitly, or you have designed something incoherent—usually the latter.

Scoping or systematic? The fork that goes wrong most often

Munn and colleagues set out six indications where a scoping review is the right choice: to identify the types of available evidence in a field; to clarify key concepts or definitions in the literature; to examine how research is conducted on a topic; to identify key characteristics or factors related to a concept; as a precursor to a systematic review; and to identify and analyse knowledge gaps.

Notice what is not on that list: estimating whether something works. If the question is about effectiveness, feasibility, appropriateness or meaningfulness of an intervention, a scoping review will not answer it, and no amount of rigour in conduct will make it answer it.

Two operational differences follow directly and catch teams out:

Critical appraisal is not mandatory in a scoping review. Munn et al. state that reviewers may assess and report risk of bias, but because the output is a map rather than a critically appraised synthesised result, appraisal is generally not performed. That is a defensible design choice—but it also means the review cannot support a recommendation.
Registration routes differ. Munn et al.’s comparison table records PROSPERO registration of the protocol as “No” for scoping reviews, with an explicit footnote that this reflects the situation at the time and may change. Check the register’s current inclusion criteria before you assume either way; scoping review protocols are commonly deposited on general-purpose repositories instead.

Reporting also diverges. Systematic reviews follow the PRISMA 2020 statement and its 27-item checklist; scoping reviews follow PRISMA-ScR, developed by a 24-member expert panel and containing 20 essential reporting items plus 2 optional ones.

Rapid reviews: decide where the time comes from

“Rapid” is not a method. It is a family of negotiated compromises, and the useful question is always which stage absorbs the compression. Sutton et al. observe that while some rapid reviews abbreviate the search—a simple strategy with limited iterations and minimal validation, fewer databases, or restriction to secondary sources—in others the saving comes entirely from removing or simplifying appraisal, synthesis or analysis.

The Cochrane Rapid Reviews Methods Group produced interim guidance built from a scoping review of the underlying evidence, primary methods studies, and a survey sent to 119 representatives across 20 Cochrane entities (63 responded, a 53% response rate). It issues 26 methods recommendations covering the stages of review conduct, flagging where each aligns with standard Cochrane systematic review guidance. Crucially, that guidance is explicit that best practice is limited by the absence of evidence for some of the shortcuts taken—so the shortcuts you choose must be declared, and their consequences for the conclusions stated. Sutton et al. make the same point more bluntly: modifications and their limitations must be declared explicitly, perhaps at greater length than in a conventional systematic review.

Reviews of reviews and umbrella reviews: the overlap problem

An umbrella review, in the JBI methodology developed by Aromataris and colleagues, is characterised by only considering the highest level of evidence for inclusion—other systematic reviews and meta-analyses. Their methodology paper also supplied a 10-question critical appraisal tool for assessing risk of bias in the included reviews.

The distinctive failure mode here is primary-study overlap. If five included reviews all contain the same three trials, the umbrella review is not summarising five independent bodies of evidence; it is counting the same trials repeatedly. Pieper and colleagues found that only 32 of 60 published overviews mentioned overlap at all, and introduced the corrected covered area (CCA) to quantify it, reporting a median CCA of 4.0 across their sample with higher overlap in health technology assessment reports and in overviews containing more publications. Build the review-by-primary-study matrix, compute the CCA, and report it—this is now expected in a well-conducted overview.

Note also that Sutton et al. found “review of reviews”, “overview” and “umbrella review” used interchangeably in practice with no consistent methodological distinction. Define your terms in the protocol rather than assuming a reader shares yours.

Qualitative and mixed-methods synthesis: aggregative or interpretive

Within the qualitative family, the search design turns on three questions that Sutton et al. set out for information specialists:

Aggregative or interpretive? Aggregative reviews search like their quantitative counterparts—many databases, extensive supplementary sources. Interpretive reviews may sample theoretically instead, stopping when conceptual saturation is reached rather than when the search is exhausted.
Does theory matter? If the review explores or tests existing theory, a separate search for theory must run alongside the search for studies, because papers that contribute conceptually to a topic are typically distinct from those that evaluate it.
Does context matter? Where context drives the phenomenon, the team assembles a “thicker” picture of each study from multiple reports—reference checking, citation follow-up, author and study-identifier tracking.

Mixed-methods reviews carry their own retrieval problem. A filter for retrieving mixed-methods studies exists but, as Sutton et al. note, has not been validated. The alternatives are to run separate quantitative, qualitative and mixed-methods filters and accept the gap risk; to supplement filters with generic research terms and de-duplicate; or to run an unfiltered topic search and sift everything. Reviews in this family place a premium on sibling papers—a quantitative and a qualitative paper from the same study—which is why cluster-tracking procedures repay the time they cost.

For choosing among the qualitative synthesis methods specifically and reporting the result, see our guide to qualitative evidence synthesis and the ENTREQ statement. Theory-driven syntheses have their own reporting standards from the RAMESES project—separate statements for realist syntheses and for meta-narrative reviews, the latter reaching consensus on 20 publication standards through a three-round Delphi with a 33-member panel.

Living systematic reviews

A living systematic review is continually updated, incorporating relevant new evidence as it becomes available, rather than being refreshed in discrete editions. Elliott and colleagues argue it matters most where evidence is emerging rapidly, current evidence is uncertain, and new research could change a policy or practice decision—the conditions under which a conventional review decays fastest.

Treat it as a mode rather than a separate design. Sutton et al. explicitly recommended categorising the living systematic review as a sub-type of systematic review rather than a distinct type. The commitment you are making is operational: a standing search schedule, a screening pipeline that runs on that schedule, and a publication route that can accept incremental updates.

Match the reporting guideline before you write the protocol

Reporting guidelines are not a write-up formality; they specify what you must have recorded while working. Choose yours at protocol stage:

PRISMA 2020 — systematic reviews. 27-item checklist plus an abstract checklist and revised flow diagrams for original and updated reviews. Replaces the 2009 statement.
PRISMA-ScR — scoping reviews. 20 essential items and 2 optional items.
ENTREQ — qualitative evidence synthesis.
RAMESES — realist syntheses and meta-narrative reviews, as two separate standards.

Guideline coverage is uneven across the families. Sutton et al. found formal search-methods guidance well established for the systematic review family and largely absent for others—they could identify no formal searching guidance at all for reviews of reviews. Where no guidance exists for your type, say so in the protocol and state which adjacent standard you are borrowing from. The Cochrane Handbook chapter guide is the fastest route to the specific methodological chapter for a given decision.

A meta-analysis is a synthesis step, not a review type

Meta-analysis is something a systematic review may or may not do. The Cochrane Handbook is unambiguous: “A systematic review need not contain any meta-analyses,” and “meta-analyses of very diverse studies can be misleading.” Where variation in results is substantial—particularly if the direction of effect is inconsistent—the Handbook warns that it may be misleading to quote an average intervention effect at all. It also cautions against jumping prematurely into statistical analysis, because results can mislead if insufficient attention has gone into formulating the question, specifying eligibility criteria, identifying studies, collecting data, considering risk of bias and planning comparisons.

Deciding whether to pool is therefore a judgement made after extraction, not a design commitment made in the title. Our comparison of systematic review vs. meta-analysis works through where the boundary falls.

What these reviews actually cost

Analysing 195 completed reviews registered in PROSPERO, Borah and colleagues found a mean of 67.3 weeks from registration to publication (IQR 42), a mean of 5 authors per review, and a mean yield rate—included studies as a proportion of records retrieved—of 2.94%. Searches returned between 27 and 92,020 citations. Reviews reporting funding took longer and involved more people than those that did not.

That is the baseline. It is not a floor. Clark and colleagues completed a full systematic review in 61 person-hours across 9 workdays (12 calendar days) using systematic review automation tools, with a small experienced team—two researcher clinicians, an information specialist and an epidemiologist—working on nothing else. Their task-level timings are the useful part for planning: data extraction was the most expensive single task at 461 person-minutes, followed by write-up, abstract screening, full-text screening and risk of bias; de-duplication was the cheapest at 16 person-minutes.

The lesson for method selection is that the expensive stages are appraisal and extraction, not searching. A “rapid” design that only trims the search is optimising the wrong end of the pipeline.

Duplication is a real risk, so check first

Ioannidis’s analysis of PubMed found 266,782 items tagged as systematic reviews and 58,611 as meta-analyses between January 1986 and December 2015, with annual publication of systematic reviews rising 2,728% between 1991 and 2014 against 153% growth for all PubMed-indexed items. Same-topic meta-analyses frequently overlap and can exceed twenty on a single topic; 185 meta-analyses of antidepressants for depression were published between 2007 and 2014 alone.

Before committing to any design, search PROSPERO for registered reviews in progress and the review-indexing databases for completed ones. Finding an existing high-quality review on your exact question is not a wasted week—it converts your project into an update, an overview, or a review with a genuinely different angle.

A working decision sequence

State the question and the decision it must inform. If no decision depends on it, a map may be the honest deliverable.
Search PROSPERO and the review databases for existing or in-progress reviews on the same question.
Decide the unit of analysis: primary studies, or existing reviews.
Write your four SALSA commitments: search, appraisal, synthesis, analysis.
Identify which family those commitments land in, and adopt that family’s reporting guideline.
Cost the appraisal and extraction stages honestly against the team and calendar you have.
If the honest cost exceeds the budget, negotiate which methods are curtailed—explicitly, in writing, with whoever commissioned it—rather than quietly renaming the review.
Register or deposit the protocol, then name the review type.

Frequently asked questions

How many types of evidence synthesis are there?

There is no settled count, because the naming is not standardised. Grant and Booth analysed 14 review types in 2009; Sutton and colleagues identified 48 distinct types in 2019 and grouped them into seven families. New types continue to appear, and several existing names are used interchangeably. Plan against the seven families, not against a count.

Is a scoping review easier than a systematic review?

Not necessarily, and it is not a lower-effort substitute. Scoping reviews often have broader inclusion criteria and therefore larger result sets to screen. What changes is the purpose and the appraisal burden: critical appraisal is not mandatory, and the output is a map rather than a graded answer. Choosing one because you are short of time, when the question is an effectiveness question, produces a review that cannot answer its own question.

What is the difference between a scoping review and a mapping review?

In practice the two terms are used interchangeably. Sutton and colleagues list scoping versus mapping as one of the pairs with no notable methodological difference in current usage, and propose clearer distinctions as a step towards a more secure typology. Because usage is inconsistent, define what you mean in the protocol rather than relying on the label to communicate it.

Do I need to register my review?

Register or deposit the protocol before you start screening, whatever the type—it is the main defence against outcome switching and post hoc criteria changes. Which register applies depends on the type: PROSPERO covers systematic reviews, and Munn and colleagues recorded that it did not accept scoping reviews at the time of their comparison, with a footnote that this could change. Check the register’s current inclusion criteria for your specific review type rather than assuming.

Can a review include both quantitative and qualitative studies?

Yes—that is the mixed-methods review family. The practical difficulty is retrieval: a filter for mixed-methods studies exists but has not been validated, so teams must either combine separate quantitative, qualitative and mixed-methods filters, supplement filters with generic research terms and de-duplicate, or run an unfiltered topic search and sift the whole set. Decide which of the three you are using, and say so.

Is a meta-analysis a type of review?

No—it is a statistical synthesis method used within a review. A systematic review may include one, several, or none. The Cochrane Handbook states plainly that a systematic review need not contain any meta-analyses, and warns that pooling very diverse studies can mislead.

What is a living systematic review?

A systematic review that is continually updated as new evidence appears, rather than revised in discrete editions. It suits fields where evidence emerges rapidly, current evidence is uncertain, and new research could change decisions. Methodologically it is best treated as a sub-type of systematic review rather than a distinct design; the commitment is operational—a standing search and screening schedule and a publication route that accepts incremental updates.