Systematic review search strategies are poorly reported and not reproducible: a cross-sectional meta-research study

Melissa L Rethlefsen; Tara J Brigham; Carrie Price; David Moher; Lex M Bouter; Jamie J Kirkham; Sara Schroter; Maurice P Zeegers

doi:10.1016/j.jclinepi.2023.111229

Systematic review search strategies are poorly reported and not reproducible: a cross-sectional meta-research study

Melissa L Rethlefsen^*, Tara J Brigham, Carrie Price, David Moher, Lex M Bouter, Jamie J Kirkham, Sara Schroter, Maurice P Zeegers

^*Corresponding author for this work

Research output: Contribution to journal › (Systematic) Review article › peer-review

Abstract

OBJECTIVE: To determine the reproducibility of biomedical systematic review search strategies. STUDY DESIGN AND SETTING: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling 6 key PRISMA-S reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records. RESULTS: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; 6 searches differed by more than 1000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible. CONCLUSION: Systematic review search reporting is poor. To correct this will require a multi-faceted response from authors, peer reviewers, journal editors, and database providers.

Original language	English
Article number	111229
Number of pages	12
Journal	Journal of Clinical Epidemiology
Volume	166
Early online date	Dec 2023
DOIs	https://doi.org/10.1016/j.jclinepi.2023.111229
Publication status	Published - Feb 2024

Keywords

database searches
reporting guidelines
reproducibility
search strategies
systematic reviews
transparency

Access to Document

10.1016/j.jclinepi.2023.111229Licence: CC BY

Cite this

@article{753d685fc5a54523930dc2dcd9612e2c,

title = "Systematic review search strategies are poorly reported and not reproducible: a cross-sectional meta-research study",

abstract = "OBJECTIVE: To determine the reproducibility of biomedical systematic review search strategies. STUDY DESIGN AND SETTING: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling 6 key PRISMA-S reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records. RESULTS: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; 6 searches differed by more than 1000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible. CONCLUSION: Systematic review search reporting is poor. To correct this will require a multi-faceted response from authors, peer reviewers, journal editors, and database providers.",

keywords = "database searches, reporting guidelines, reproducibility, search strategies, systematic reviews, transparency",

author = "Rethlefsen, {Melissa L} and Brigham, {Tara J} and Carrie Price and David Moher and Bouter, {Lex M} and Kirkham, {Jamie J} and Sara Schroter and Zeegers, {Maurice P}",

year = "2024",

month = feb,

doi = "10.1016/j.jclinepi.2023.111229",

language = "English",

volume = "166",

journal = "Journal of Clinical Epidemiology",

issn = "0895-4356",

publisher = "Elsevier Science",

}

TY - JOUR

T1 - Systematic review search strategies are poorly reported and not reproducible

T2 - a cross-sectional meta-research study

AU - Rethlefsen, Melissa L

AU - Brigham, Tara J

AU - Price, Carrie

AU - Moher, David

AU - Bouter, Lex M

AU - Kirkham, Jamie J

AU - Schroter, Sara

AU - Zeegers, Maurice P

PY - 2024/2

Y1 - 2024/2

N2 - OBJECTIVE: To determine the reproducibility of biomedical systematic review search strategies. STUDY DESIGN AND SETTING: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling 6 key PRISMA-S reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records. RESULTS: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; 6 searches differed by more than 1000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible. CONCLUSION: Systematic review search reporting is poor. To correct this will require a multi-faceted response from authors, peer reviewers, journal editors, and database providers.

AB - OBJECTIVE: To determine the reproducibility of biomedical systematic review search strategies. STUDY DESIGN AND SETTING: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling 6 key PRISMA-S reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records. RESULTS: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; 6 searches differed by more than 1000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible. CONCLUSION: Systematic review search reporting is poor. To correct this will require a multi-faceted response from authors, peer reviewers, journal editors, and database providers.

KW - database searches

KW - reporting guidelines

KW - reproducibility

KW - search strategies

KW - systematic reviews

KW - transparency

U2 - 10.1016/j.jclinepi.2023.111229

DO - 10.1016/j.jclinepi.2023.111229

M3 - (Systematic) Review article

SN - 0895-4356

VL - 166

JO - Journal of Clinical Epidemiology

JF - Journal of Clinical Epidemiology

M1 - 111229

ER -