Written and maintained by CASRAI Editorial Board
Last updated
Most critical appraisal tools are built for one study design at a time — a checklist for randomized controlled trials, a separate one for qualitative research, another for cohort studies. That works fine until a review needs to appraise all three inside the same synthesis, which is exactly the situation a systematic mixed studies review creates: one review question, evidence drawn from qualitative, quantitative, and mixed methods studies at once, and no single existing appraisal family covering all of them without switching tools mid-review.
The Mixed Methods Appraisal Tool (MMAT) was built to close that gap. Developed by Pierre Pluye and colleagues at McGill University, with the current 2018 version led by Quan Nha Hong, the MMAT lets a reviewer appraise qualitative studies, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies using one consistent instrument rather than assembling a different checklist per design.
What the MMAT actually checks
Every study first passes through two universal screening questions, asked regardless of design: is there a clear research question (or objective), and does the collected data allow that question to be addressed? A study that fails either screening question is not appraised further with the design-specific criteria — the screening step exists to catch studies that shouldn’t be included in the appraisal at all before spending time on the detailed checklist.
Once a study passes screening, the reviewer identifies which of five categories it belongs to and applies that category’s five criteria:
| Category | Applies to |
|---|---|
| Qualitative | Interview, focus group, ethnographic, and other qualitative designs |
| Randomized controlled trials | Studies with random allocation to comparison groups |
| Non-randomized studies | Comparative studies without randomization — cohort, case-control, and similar designs |
| Quantitative descriptive | Surveys and other studies describing a population or phenomenon without a comparison group |
| Mixed methods | Studies that integrate qualitative and quantitative components within one design |
Each criterion is answered Yes, No, or Can’t tell — the same judgment-based format used by JBI’s checklists and CASP, rather than a points-based scale.
Why there’s no overall score
A recurring question about the MMAT is why it doesn’t produce one number a reviewer can rank studies by. The tool’s developers deliberately avoided a summed or averaged score, on the reasoning that collapsing five distinct methodological judgments into one figure obscures exactly which aspect of a study is weak — a study that fails on sampling but succeeds everywhere else looks identical, under a single score, to one that fails on data analysis but succeeds on sampling. MMAT guidance instead recommends reporting the rating for each individual criterion and, if a summary view is genuinely needed, describing it narratively (e.g., “met 4 of 5 criteria, with the recruitment strategy under-described”) rather than converting it into an arithmetic total.
Using the MMAT in a review workflow
In practice, MMAT appraisal happens after study selection and data extraction, alongside (not instead of) the reporting requirements a systematic review already carries under PRISMA 2020. Two independent reviewers typically apply the MMAT to each included study, resolve disagreements by discussion, and report the resulting per-criterion judgments — not a composite score — in the review’s methods and results sections. For a review that specifically integrates qualitative and quantitative findings rather than just including both study types, appraisal usually pairs with a stated integration approach; see CASRAI’s guide to convergent vs. segregated mixed-methods synthesis and to joint displays for how the appraised studies typically get combined once appraisal is done.
The MMAT appraises individual primary studies feeding into a review — it is not a tool for appraising the review itself. If what you’re evaluating is the systematic review as a finished piece of methodology (its search strategy, its synthesis process), that’s a different instrument; see AMSTAR-2 below.
How the MMAT differs from other appraisal tools on this site
- JBI’s 13 checklists cover a wider range of individual study designs (case reports, diagnostic accuracy studies, economic evaluations, and more) but require picking a separate checklist per design rather than one instrument spanning several designs at once.
- CASP checklists take the same one-checklist-per-design approach as JBI, maintained by a different organization (the Critical Appraisal Skills Programme, Oxford).
- AMSTAR-2 appraises a completed systematic review itself, not the primary studies included in it — a categorically different appraisal target than the MMAT.
- RoB 2 / ROBINS-I are domain-based risk-of-bias tools specific to randomized and non-randomized intervention studies, producing a structured low/some-concerns/high judgment per bias domain rather than the MMAT’s five-criteria Yes/No/Can’t-tell format.
For a broader map of which appraisal tool applies to which situation across all of these, see CASRAI’s critical appraisal tool selection guide. For background on mixed methods research generally, see CASRAI’s Dictionary entry.
Limitations worth knowing
The MMAT’s category-selection step assumes the reviewer can correctly classify a study’s design before appraising it — a study that blends elements of two categories (e.g., a mixed methods study with a quasi-experimental quantitative strand) requires judgment about which set of criteria is the better fit, and reviewers using the tool independently can reasonably classify the same borderline study differently. Because the tool deliberately produces no overall score, it also isn’t a substitute for a formal risk-of-bias tool where a review’s downstream synthesis (e.g., a meta-analysis) requires a structured bias judgment per domain rather than a general quality checklist.
Frequently asked questions
Is the MMAT the same as a risk-of-bias assessment?
No. The MMAT produces a general methodological-quality judgment per criterion, not a structured per-domain bias rating. A tool such as RoB 2 or ROBINS-I is built specifically to feed a formal bias judgment into further synthesis; the MMAT is built to appraise and describe study quality across a mixed-studies evidence base.
Do I need special training to use the MMAT?
The tool’s own documentation recommends that reviewers have basic familiarity with qualitative, quantitative, and mixed methods research designs before applying it, since correctly classifying a study’s category is a prerequisite step. It does not require a separate certification, unlike some appraisal or reporting instruments.
Can the MMAT be used outside a systematic review?
It’s designed specifically for appraising studies included in a systematic mixed studies review, but some research teams also use it more informally to structure quality discussion of a smaller set of studies (e.g., for a scoping review’s optional quality assessment, or a narrative synthesis) — check your target journal’s or protocol’s required methodology before assuming that’s an accepted substitute for a formally reported MMAT appraisal.
