Article
The impact on statistical significance when switching to random-effects and to the Hartung-Knapp method in systematic reviews of advanced cancer patients – sensitivity analyses from a meta-epidemiological study (SCOPE)
Search Medline for
Authors
-
Waldemar Siemens
- Medical Center, University of Freiburg, Faculty of Medicine, University of Freiburg, Clinic for Palliative Care, Freiburg, Deutschland
-
Guido Schwarzer
- Faculty of Medicine and Medical Center, University of Freiburg, Institute of Medical Biometry and Statistics, Freiburg, Deutschland
-
Miriam S. Rohe
- Medical Center, University of Freiburg, Faculty of Medicine, University of Freiburg, Clinic for Palliative Care, Freiburg, Deutschland
-
Joerg J. Meerpohl
- Medical Center-University of Freiburg, Institute for Evidence in Medicine (for Cochrane Germany Foundation), Freiburg, Deutschland
-
Gerhild Becker
- Medical Center, University of Freiburg, Faculty of Medicine, University of Freiburg, Clinic for Palliative Care, Freiburg, Deutschland
| Published: | February 12, 2020 |
|---|
Outline
Text
Background/research question: In biomedical research, the assumptions of the random-effects model are often more appropriate than of the fixed-effect model [1]. Further, it has been suggested to replace the standard method in random-effects meta-analyses by the Hartung-Knapp method [2].
This project aims to examine the impact on statistical significance when using the random-effects model and the Hartung-Knapp-method, respectively.
Methods: A meta-epidemiological study (CRD42019134904) was conducted. Three databases were searched from 2009-2019: Medline, the Cochrane Database of Systematic Reviews and Web of Science. We included systematic reviews with meta-analysis of at least four randomized controlled trials with pharmacological, surgical or radiotherapeutic interventions in advanced cancer patients. The first statistically significant meta-analysis of a health-related outcome presented in the respective review was chosen for the analyses.
Analyses were performed with the statistical software R.
Results: A total of 5608 hits were screened. Finally, 261 systematic reviews were included. A fixed-effect model was used in 138 (52.9%) and a random-effects model in 123 (47.1%) meta-analyses.
Six of 138 fixed-effect meta-analyses (4.4%) lost statistical significance when recalculating them. When switching from a fixed-effect to a random-effects model, 19/132 (14.4%) of the meta-analyses were no longer statistically significant.
Using the Hartung-Knapp method instead of the standard method for random-effects meta-analyses resulted in 43/123 (35.0%) meta-analyses that were no longer statistically significant.
Conclusion: A relevant number of meta-analyses lost statistical significance when applying up-to-date meta-analysis methods. This implicates that many meta-analyses cited in medical literature and treatment guidelines, e.g. to recommend drugs, would probably come to a different conclusion when applying the random-effects model or the Hartung-Knapp method. We strongly advocate the use of up-to-date methods and sensitivity analysis when presenting results from meta-analyses.
Competing interests: The authors have no conflict of interests to declare.
References
- 1.
- Higgins JP, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. J R Stat Soc Ser A Stat Soc. 2009 Jan;172(1):137-159
- 2.
- Jackson D, Law M, Rücker G, Schwarzer G. The Hartung-Knapp modification for random-effects meta-analysis: A useful refinement but are there any residual concerns? Stat Med. 2017 Nov 10;36(25):3923-3934. DOI: 10.1002/sim.7411
