Meta-analysis (MA)

Meta-analysis (MA) employs statistical methods to aggregate effect sizes from multiple studies, intending to estimate the overall effect magnitude, determine the probable effect direction, or assess statistical significance. Meta-analyses are typically built upon a systematic review framework, which defines the criteria for study inclusion and ensures methodological transparency. (See: How to Perform a Meta-analysis below.) In this sense, meta-analysis is best understood as a statistical tool that may contribute additional insight to a well-conducted systematic review. However, it is not a review in itself.

Purpose and Function: The primary utility of a meta-analysis lies in resolving differences between high-quality randomized controlled trials (RCTs) (with similar design) that report conflicting findings. When applied correctly, a meta-analysis may enhance statistical power, refine estimates of effect size, or clarify the likely direction of effect. Its value is most evident when the included studies are sufficiently similar in design and outcome measures, and when a definitive conclusion cannot be drawn from individual trials alone.

What a Meta-analysis Is Not: Meta-analysis is not original research, nor is it a “superior” form of evidence. It is a form of secondary analysis (an average of averages) and should not be interpreted as more valid than the underlying primary studies. Although this form of synthesis can be powerful, considering a meta-analysis superior to the original data it summarizes is analogous to treating a reviewer’s movie rating as more authoritative than the film itself. As noted above, meta-analyses have a distinct purpose and function, but they also introduce new risks: additional layers of bias and error, statistical distortion, and misinterpretation.

“Suggesting that meta-analysis is the highest level of evidence is like suggesting Rotten Tomatoes is better than Netflix.”
— Dr. Brent Brookbush, DPT, MS, CPT, HMS, IMT

Related Articles

• Meta-analysis Problems: Why do so many imply that nothing works?
• Is There a Single Best Approach to Physical Rehabilitation?
• NCCA-accredited Certified Personal Trainer (CPT) Exams: Meta-analysis of Topics (Domains)

Related Terms

• Systematic Review
• Evidence-based practice
• Levels of evidence
• Meta-analysis
• P-value
• Null Hypothesis
• Vote-Counting
• Regression to the Mean
• Randomized Controlled Trial
• Experimental Research
• Prospective Study
• Longitudinal Study
• Case Control Study
• Cross-sectional Study
• Retrospective Study

Advantages and Disadvantages of MAs

Advantages of Meta-analysis

• Clarifies the direction of effect among multiple studies with conflicting results.
• May increases the statistical power of results by pooling participants' data; however, using meta-analysis for this purpose requires careful and conservative interpretations and conclusions.
• May estimate the magnitude of effect across multiple studies; however, this is only true if conclusions are published with a transparent description of the groups pooled in the study (i.e., what averages are being averaged).

Disadvantages and Risks

• Highly sensitive to methodological flaws included studies
• Vulnerable to multiple forms of bias (sampling, publication, accumulation, etc.)
• Misuse as a “shortcut to publication” contributes to misleading or nihilistic interpretations
• Overreliance on statistical significance may obscure clinical relevance
• Misinterpretation of "regression to the mean" and "failure to refute the null" as an implication that interventions are not effective, despite clear effect directions in the included RCTs.

Applied Example

• A meta-analysis might be appropriate when 6–10 large, high-quality RCTs report conflicting outcomes regarding the effectiveness of a drug intervention. However, if those RCTs report similar findings (e.g., all support the intervention), an MA adds little value and increases the risk of introducing bias, particularly if it fails to reject the null hypothesis.

Frequently Asked Questions (FAQs)

Is meta-analysis the highest level of evidence?

• No. Meta-analysis is a statistical tool, not a tier of evidence. Its value depends entirely on the quality and compatibility of the studies included.

When should meta-analysis be used?

• When multiple high-quality RCTs, using similar methods and populations, yield contradictory results on the same outcome.

Why might a meta-analysis fail to reject the null when the individual studies show a clear effect?

• This may result from regression to the mean, the inclusion of underpowered or heterogeneous studies, poor hypothesis formulation, or the statistical dilution of clear trends.

Should I trust a meta-analysis over the trends shown in individual studies?

• Not necessarily. If the individual RCTs show consistent trends and the meta-analysis fails to reject the null, the MA may be misleading. Trend consistency often provides more meaningful information.

Brookbush Institute Perspective on Meta-analysis

Meta-analysis is not a higher level of evidence.

It is a tool used within systematic reviews. Inappropriate elevation of MAs above well-designed comparative studies can obscure trends, especially when the underlying studies are methodologically weak or heterogeneous.

Meta-analysis should be reserved for specific contexts.

MAs are most appropriate when there are many high-quality, conflicting RCTs on the same topic, using similar designs and outcome measures. This is rare in movement science and rehabilitation, where participant pools are small, study designs vary widely, and contradictions are uncommon.

Interpretation errors are common

A “failure to reject the null” in a meta-analysis is not equivalent to evidence that an intervention is ineffective. It may instead indicate:

• Insufficient sample size or data
• Inadequate representation of the research question
• High heterogeneity or poor study design comparability
• Regression to the mean
• Accumulation, author, or publication bias

Vote-counting may be superior in certain contexts

In homogeneous literature with consistent trends, structured vote-counting (as employed by the Brookbush Institute) may yield more valid conclusions than pooling dissimilar studies into a single estimate. (See: Systematic Review)

How to Perform a Meta-analysis

A meta-analysis follows several structured steps that parallel the broader systematic review process, culminating in the statistical aggregation of effect sizes. Although advanced statistical software (e.g., RevMan, R, STATA) is typically required, the foundational process is as follows:

1. Define the Research Question and Inclusion Criteria

• Identify a focused scientific question or topic. Determine inclusion/exclusion criteria (e.g., study design, quality, publication date, outcome measures). Ensure that included studies investigate the same outcome using sufficiently similar methods, populations, and interventions.

2. Extract Effect Sizes and Variance Measures

• For continuous outcomes: extract means, standard deviations (SD), and sample sizes
• For dichotomous outcomes: extract odds ratios (OR), risk ratios (RR), or risk differences
• Compute or extract the standard error (SE) or confidence intervals (CI) if not provided

3. Choose and Apply a Statistical Model

• The fixed-effect model assumes a single true effect across all studies (less appropriate when heterogeneity exists)
• Random-effects model assumes effect sizes vary across studies due to underlying differences (preferred in most movement science applications)

4. Calculate the Weighted Mean Effect Size

Caption: Meta-analysis (MA) Equations

5. Assess Heterogeneity

• Use Cochran’s Q test and I² statistic to determine how much of the variation in effect sizes is due to between-study heterogeneity. I²>50% suggests substantial heterogeneity, calling into question the appropriateness of pooling.

6. Check for Bias and Sensitivity

• Evaluate publication bias using funnel plots, and perform sensitivity analyses by removing outliers or low-quality studies. Consider meta-regression or subgroup analysis to explore potential moderators.

7. Report Findings Transparently

• Report effect size (e.g., standardized mean difference or odds ratio), confidence intervals, p-values, and heterogeneity statistics. Clearly state the model used, study weights, and limitations

Caution: When Not to Perform a Meta-analysis

• The studies are too heterogeneous in design or population
• The effect sizes cannot be meaningfully aggregated
• Contradictory results are not present (a trend may be clearer through vote-counting)
• The number of studies or sample sizes is too small to yield reliable results

References

• Humaidan, P., & Polyzos, N. P. (2012). (Meta) analyze this: systematic reviews might lose credibility. Nature Medicine, 18(9), 1321–1321.
• Simon, C., & Bellver, J. (2014). Scratching beneath ‘The Scratching Case’. Human Reproduction, 29(8), 1618–1621.
• Ioannidis, J. P. (2016). The mass production of redundant, misleading, and conflicted systematic reviews and meta‐analyses. The Milbank Quarterly, 94(3), 485–514.
• Murad, M. H., et al. (2016). New evidence pyramid. BMJ Evidence-Based Medicine, 21(4), 125–127.
• Ter Schure, J., & Grünwald, P. (2019). Accumulation bias in meta-analysis. F1000Research, 8.
• Lin, L. (2018). Bias caused by sampling error in meta-analysis with small sample sizes. PLOS ONE, 13(9), e0204056.
• van Wely, M. (2014). The good, the bad, and the ugly: meta-analyses. Human Reproduction, 29(8), 1622–1626.