TY - JOUR
T1 - Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions
AU - Wang, Shirley V.
AU - Sreedhara, Sushama Kattinakere
AU - Schneeweiss, Sebastian
AU - Franklin, Jessica M.
AU - Gagne, Joshua J.
AU - Huybrechts, Krista F.
AU - Patorno, Elisabetta
AU - Jin, Yinzhu
AU - Lee, Moa
AU - Mahesri, Mufaddal
AU - Pawar, Ajinkya
AU - Barberio, Julie
AU - Bessette, Lily G.
AU - Chin, Kristyn
AU - Gautam, Nileesa
AU - Ortiz, Adrian Santiago
AU - Sears, Ellen
AU - Stefanini, Kristina
AU - Zakarian, Mimi
AU - Dejene, Sara
AU - Rogers, James R.
AU - Brill, Gregory
AU - Landon, Joan
AU - Lii, Joyce
AU - Tsacogianis, Theodore
AU - Vine, Seanna
AU - Garry, Elizabeth M.
AU - Gibbs, Liza R.
AU - Gierada, Monica
AU - Isaman, Danielle L.
AU - Payne, Emma
AU - Alwardt, Sarah
AU - Arlett, Peter
AU - Bartels, Dorothee B.
AU - Bate, Andrew
AU - Berlin, Jesse
AU - Bourke, Alison
AU - Bradbury, Brian
AU - Brown, Jeffrey
AU - Burnett, Karen
AU - Brennan, Troyen
AU - Chan, K. Arnold
AU - Choi, Nam Kyong
AU - de Vries, Frank
AU - Eichler, Hans Georg
AU - Filion, Kristian B.
AU - Freeman, Lisa
AU - Hallas, Jesper
AU - Happe, Laura
AU - Hennessy, Sean
AU - REPEAT Initiative
N1 - Funding Information:
S.V.W. received salary support from grants to Brigham and Women’s Hospital from Boehringer Ingelheim, Johnson & Johnson and Novartis Pharmaceuticals for unrelated work. She was supported by NHLBI RO1HL141505 and NIA R01AG053302 during the conduct of this work. E.M.G., L.R.G., and M.G. are employees of and have stock options in Aetion, Inc. D.L.I. and E.P. were former employees of Aetion, Inc. during the implementation of this study. Dr. Patorno was supported by the National Institute on Aging (K08AG055670). She is investigator of an investigator-initiated grant to the Brigham and Women’s Hospital from Boehringer Ingelheim, not related to the topic of the submitted work. J.J.G. received salary support from grants from Eli Lilly and Company and Novartis Pharmaceuticals Corporation to the Brigham and Women’s Hospital and was a consultant to Optum, Inc., all for unrelated work during the conduct of this study. He has since become an employee of Johnson & Johnson. Brian Nosek is Executive Director of the Center for Open Science, a nonprofit technology and culture change organization with a mission to increase openness, integrity, and reproducibility of research. A.B. in an employee of GSK and was at study initiation was an employee of Pfizer. He is a shareholder and hold stock options in GSK and previously held stock and stock options at Pfizer. D.B. is an employee of UCB Pharma. J.M.F. was supported by NHLBI RO1HL141505 during the conduct of this study. She has since become an employee of Optum Epidemiology. M.P.L. was supported by NHLBI F32 HL149256. J.R.R. was a paid consultant to Aetion during the first year of this study for unrelated work. K.B.F. is supported by a senior salary support award from the Fonds de recherche du Québec – santé (Quebec Foundation for Research–Health) and a William Dawson Scholar award from McGill University. David Martin has since become an employee of Moderna. All other authors have no conflict of interest to declare.
Funding Information:
This project was funded by the non-profit Arnold Ventures, with support from Brigham and Women’s Hospital (SVW, SS). Drs. Wang and Schneeweiss were supported by NHLBI RO1HL141505 and NIA R01AG053302 during the conduct of this work.
Publisher Copyright:
© 2022, The Author(s).
PY - 2022/8/31
Y1 - 2022/8/31
N2 - Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson’s correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratiooriginal/hazard ratioreproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.
AB - Studies that generate real-world evidence on the effects of medical products through analysis of digital data collected in clinical practice provide key insights for regulators, payers, and other healthcare decision-makers. Ensuring reproducibility of such findings is fundamental to effective evidence-based decision-making. We reproduce results for 150 studies published in peer-reviewed journals using the same healthcare databases as original investigators and evaluate the completeness of reporting for 250. Original and reproduction effect sizes were positively correlated (Pearson’s correlation = 0.85), a strong relationship with some room for improvement. The median and interquartile range for the relative magnitude of effect (e.g., hazard ratiooriginal/hazard ratioreproduction) is 1.0 [0.9, 1.1], range [0.3, 2.1]. While the majority of results are closely reproduced, a subset are not. The latter can be explained by incomplete reporting and updated data. Greater methodological transparency aligned with new guidance may further improve reproducibility and validity assessment, thus facilitating evidence-based decision-making. Study registration number: EUPAS19636.
U2 - 10.1038/s41467-022-32310-3
DO - 10.1038/s41467-022-32310-3
M3 - Article
C2 - 36045130
SN - 2041-1723
VL - 13
JO - Nature Communications
JF - Nature Communications
IS - 1
M1 - 5126
ER -