eckert2023

Prévia do material em texto

Letter to the Editor
Letter to the Editor: Double-counting due to inadequate
statistics leads to false-positive findings in “Effects of
creatine supplementation on memory in healthy individuals:
a systematic review and meta-analysis of randomized
controlled trials”
Igor Eckert and Eric Pascher
Prokopidis et al have conducted a meta-analysis of randomized, placebo-controlled
clinical trials to assess the effects of oral creatine supplementation on memory per-
formance of healthy individuals. However, concerns were raised regarding the val-
idity of their statistical analyses, which may have led to misleading conclusions. In
this letter, we describe the statistical issue at hand and its potential implications.
Dear Editor,
We have read with great interest the work by
Prokopidis et al1, who conducted a meta-analysis of
randomized, placebo-controlled clinical trials to assess
the effects of oral creatine supplementation on memory
performance of healthy individuals and found beneficial
effects in favor of the intervention. In this letter, how-
ever, we argue that such conclusions regrettably were
based on flawed statistical analyses that warrant
correction.
The statistical issue at hand is known as unit-of-
analysis error and occurs when the number of observa-
tions in an analysis does not match the number of units
randomized, violating the assumption of independence
in traditional meta-analysis models.2,3 Typically, in
randomized, controlled trials, participants are random-
ized to 1 of 2 intervention groups, and a single measure-
ment for each outcome from each participant is
collected and analyzed. However, studies may report
data for the same outcome, using several measurement
scales. This leads to a form of dependence whereby
multiple estimates are obtained from the same individu-
als. When this is the case, reviewers are faced with
difficult analytical decisions, such as about whether a
meta-analysis should be performed, and what data and
models ought to be used.
Prokopidis et al1 decided to simply combine all the
available data of every study, using effect sizes from
multiple outcome measures in a single analysis.
Although mathematically possible, this practice perva-
sively violates the assumption of independent observa-
tions. The number of unique participants originally
included across 8 studies (n¼ 225) is nearly quadrupled
in the main meta-analysis (n¼ 847 participants;
Figure 2 in the Prokopidis et al1 article). Including cor-
related, overlapping, or duplicate samples as if they
were independent data is not only logically fallacious,
doing so also artificially increases statistical power and
inflates precision, leading to increased risk of false-
positive findings and invalid estimates of effect.4 That
most studies were judged as at high risk of bias further
aggravates this analytical issue, overstating evidence of
questionable validity.
Double counting of evidence in meta-analyses is a
serious but relatively common problem,5 despite plenty
of published guidance on how to avoid it. When studies
Affiliation: I. Eckert is with the Universidade Federal de Ciências da Sa�ude de Porto Alegre, Rio Grande do Sul, Brazil. E. Pascher is with the
Faculdade de Medicina de S~ao Jos�e do Rio Preto, S~ao Paulo, Brazil.
Correspondence: I. Eckert, Universidade Federal de Ciências da Sa�ude de Porto Alegre, Sarmento Leite Street, 245, Centro Hist�orico, Porto
Alegre, Rio Grande do Sul 90050-170, Brazil. E-mail:igoreckert2@gmail.com.
Key words: creatine, dietary supplements, memory, meta-analysis, statistical analysis.
VC The Author(s) 2023. Published by Oxford University Press on behalf of the International Life Sciences Institute.
All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
https://doi.org/10.1093/nutrit/nuac108
Nutrition ReviewsVR Vol. 00(0):1–2 1
D
ow
nloaded from
 https://academ
ic.oup.com
/nutritionreview
s/advance-article/doi/10.1093/nutrit/nuac108/6987899 by jcam
eron on 17 January 2023
https://orcid.org/0000-0002-5228-9059
report multiple effect sizes, authors may (1) combine
the average effect sizes of each study, although at a cost
of decreased statistical power; (2) use a full multivariate
meta-analysis, if within-study correlations are available;
or (3) use the robust variance estimation method for
nonindependent effect sizes, which does not require
knowledge of the within-study covariance.6
Alternatively, it is possible to select only the most suit-
able outcome measure that answers the specific research
question of interest (ie, a single analysis using data of
the most valid measurement of memory performance
from each study) or to perform a separate meta-analysis
for each conceptually similar memory test. Regardless
of the chosen method, it is essential that the assump-
tions and limitations are clearly stated and inferences
drawn cautiously, particularly those involving post hoc
decisions that were not prespecified in a statistical anal-
ysis plan.7
CONCLUSION
Evidence-based practitioners often rely on meta-
analyses for clinical decision-making, expecting that
such studies are conducted according to rigorous meth-
ods that safeguard against bias. When this type of study
fails to provide accurate results, misguided decisions in
clinical practice and marketing messages with unwar-
ranted health claims by the supplement industry are
likely to ensue. We believe the meta-analysis presented
by Prokopidis et al1 are ultimately flawed and have led
to false-positive results. The analysis should be cor-
rected to provide truthful estimates.
Acknowledgments
Author contributions. I.E. conceived this letter. I.E. and
E.P. wrote and critically reviewed the final text.
Funding. No external funding was received to support
this work.
Declaration of interest. The authors have no relevant
interests to declare.
REFERENCES
1. Prokopidis K, Giannos P, Triantafyllidis KK, Kechagias KS, Forbes SC, Candow DG.
Effects of creatine supplementation on memory in healthy individuals: a systematic
review and meta-analysis of randomized controlled trials. Nutr Rev. 2022. doi:
10.1093/nutrit/nuac064.
2. Whiting-O’Keefe QE, Henke C, Simborg DW. Choosing the correct unit of analysis in
Medical Care experiments. Med Care. 1984;22:1101–1114.
3. Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic
Reviews of Interventions. Version 6.3 (Updated February 2022). Cochrane, 2022.
Available at: https://www.training.cochrane.org/handbook.
4. Matt GE, Cook TD. Threats to the validity of generalized inferences. In: Cooper HM,
Hedges LV, Valentine JC, eds. The Handbook of Research Synthesis and Meta-
Analysis. 2nd ed. New York: Russell Sage Foundation; 2009:531–557.
5. Senn SJ. Overstating the evidence – double counting in meta-analysis and related
problems. BMC Med Res Methodol. 2009;9:10.
6. Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression
with dependent effect size estimates. Res Synth Methods. 2010;1:39–65.
7. Shamseer L, Moher D, Clarke M, et al.; PRISMA-P Group. Preferred reporting items
for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration
and explanation. BMJ. 2015;350:g7647.
2 Nutrition ReviewsVR Vol. 00(0):1–2
D
ow
nloaded from
 https://academ
ic.oup.com
/nutritionreview
s/advance-article/doi/10.1093/nutrit/nuac108/6987899 by jcam
eron on 17 January 2023
https://doi.org/10.1093/nutrit/nuac064
https://www.training.cochrane.org/handbook