Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Brooke Levis; Parash Mani Bhandari; Dipika Neupane; Suiqiong Fan; Ying Sun; Chen He; Yin Wu; Ankur Krishnan; Zelalem Negeri; Mahrukh Imran; Danielle B Rice; Kira E Riehm; Marleine Azar; Alexander W Levis; Jill Boruff; Pim Cuijpers; Simon Gilbody; John P A Ioannidis; Lorie A Kloda; Scott B Patten; Roy C Ziegelstein; Daphna Harel; Yemisi Takwoingi; Sarah Markham; Sultan H Alamri; Dagmar Amtmann; Bruce Arroll; Liat Ayalon; Hamid R Baradaran; Anna Beraldi; Charles N Bernstein; Arvin Bhana; Charles H Bombardier; Ryna Imma Buji; Peter Butterworth; Gregory Carter; Marcos H Chagas; Juliana C N Chan; Lai Fong Chan; Dixon Chibanda; Kerrie Clover; Aaron Conway; Yeates Conwell; Federico M Daray; Janneke M de Man-van Ginkel; Jesse R Fann; Felix H Fischer; Sally Field; Jane R W Fisher; Daniel S S Fung; Depression Screening Data (DEPRESSD) PHQ Group; Miranda T. Schram; Andrea Benedetti

doi:10.1001/jamanetworkopen.2024.29630

Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Brooke Levis, Parash Mani Bhandari, Dipika Neupane, Suiqiong Fan, Ying Sun, Chen He, Yin Wu, Ankur Krishnan, Zelalem Negeri, Mahrukh Imran, Danielle B Rice, Kira E Riehm, Marleine Azar, Alexander W Levis, Jill Boruff, Pim Cuijpers, Simon Gilbody, John P A Ioannidis, Lorie A Kloda, Scott B PattenRoy C Ziegelstein, Daphna Harel, Yemisi Takwoingi, Sarah Markham, Sultan H Alamri, Dagmar Amtmann, Bruce Arroll, Liat Ayalon, Hamid R Baradaran, Anna Beraldi, Charles N Bernstein, Arvin Bhana, Charles H Bombardier, Ryna Imma Buji, Peter Butterworth, Gregory Carter, Marcos H Chagas, Juliana C N Chan, Lai Fong Chan, Dixon Chibanda, Kerrie Clover, Aaron Conway, Yeates Conwell, Federico M Daray, Janneke M de Man-van Ginkel, Jesse R Fann, Felix H Fischer, Sally Field, Jane R W Fisher, Daniel S S Fung, Depression Screening Data (DEPRESSD) PHQ Group, Miranda T. Schram, Andrea Benedetti^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

IMPORTANCE: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates. OBJECTIVE: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates. DESIGN, SETTING, AND PARTICIPANTS: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled. MAIN OUTCOMES AND MEASURES: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population. RESULTS: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes. CONCLUSIONS AND RELEVANCE: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

Original language	English
Article number	2429630
Number of pages	15
Journal	Jama network open
Volume	7
Issue number	11
DOIs	https://doi.org/10.1001/jamanetworkopen.2024.29630
Publication status	Published - 4 Nov 2024

Keywords

Humans
Patient Health Questionnaire
Cross-Sectional Studies
Depression/diagnosis
Mass Screening/methods
Sensitivity and Specificity
Depressive Disorder, Major/diagnosis
Female
Male

Access to Document

10.1001/jamanetworkopen.2024.29630Licence: CC BY-NC-ND

Cite this

Levis, B., Bhandari, P. M., Neupane, D., Fan, S., Sun, Y., He, C., Wu, Y., Krishnan, A., Negeri, Z., Imran, M., Rice, D. B., Riehm, K. E., Azar, M., Levis, A. W., Boruff, J., Cuijpers, P., Gilbody, S., Ioannidis, J. P. A., Kloda, L. A., ... Benedetti, A. (2024). Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool. Jama network open, 7(11), Article 2429630. https://doi.org/10.1001/jamanetworkopen.2024.29630

@article{0636306a43894f179604da11986ddc03,

title = "Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool",

abstract = "IMPORTANCE: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates. OBJECTIVE: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates. DESIGN, SETTING, AND PARTICIPANTS: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled. MAIN OUTCOMES AND MEASURES: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population. RESULTS: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes. CONCLUSIONS AND RELEVANCE: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.",

keywords = "Humans, Patient Health Questionnaire, Cross-Sectional Studies, Depression/diagnosis, Mass Screening/methods, Sensitivity and Specificity, Depressive Disorder, Major/diagnosis, Female, Male",

author = "Brooke Levis and Bhandari, {Parash Mani} and Dipika Neupane and Suiqiong Fan and Ying Sun and Chen He and Yin Wu and Ankur Krishnan and Zelalem Negeri and Mahrukh Imran and Rice, {Danielle B} and Riehm, {Kira E} and Marleine Azar and Levis, {Alexander W} and Jill Boruff and Pim Cuijpers and Simon Gilbody and Ioannidis, {John P A} and Kloda, {Lorie A} and Patten, {Scott B} and Ziegelstein, {Roy C} and Daphna Harel and Yemisi Takwoingi and Sarah Markham and Alamri, {Sultan H} and Dagmar Amtmann and Bruce Arroll and Liat Ayalon and Baradaran, {Hamid R} and Anna Beraldi and Bernstein, {Charles N} and Arvin Bhana and Bombardier, {Charles H} and Buji, {Ryna Imma} and Peter Butterworth and Gregory Carter and Chagas, {Marcos H} and Chan, {Juliana C N} and Chan, {Lai Fong} and Dixon Chibanda and Kerrie Clover and Aaron Conway and Yeates Conwell and Daray, {Federico M} and {de Man-van Ginkel}, {Janneke M} and Fann, {Jesse R} and Fischer, {Felix H} and Sally Field and Fisher, {Jane R W} and Fung, {Daniel S S} and {Depression Screening Data (DEPRESSD) PHQ Group} and Schram, {Miranda T.} and Andrea Benedetti",

year = "2024",

month = nov,

day = "4",

doi = "10.1001/jamanetworkopen.2024.29630",

language = "English",

volume = "7",

journal = "Jama network open",

issn = "2574-3805",

publisher = "American Medical Association",

number = "11",

}

Levis, B, Bhandari, PM, Neupane, D, Fan, S, Sun, Y, He, C, Wu, Y, Krishnan, A, Negeri, Z, Imran, M, Rice, DB, Riehm, KE, Azar, M, Levis, AW, Boruff, J, Cuijpers, P, Gilbody, S, Ioannidis, JPA, Kloda, LA, Patten, SB, Ziegelstein, RC, Harel, D, Takwoingi, Y, Markham, S, Alamri, SH, Amtmann, D, Arroll, B, Ayalon, L, Baradaran, HR, Beraldi, A, Bernstein, CN, Bhana, A, Bombardier, CH, Buji, RI, Butterworth, P, Carter, G, Chagas, MH, Chan, JCN, Chan, LF, Chibanda, D, Clover, K, Conway, A, Conwell, Y, Daray, FM, de Man-van Ginkel, JM, Fann, JR, Fischer, FH, Field, S, Fisher, JRW, Fung, DSS, Depression Screening Data (DEPRESSD) PHQ Group, Schram, MT & Benedetti, A 2024, 'Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool', Jama network open, vol. 7, no. 11, 2429630. https://doi.org/10.1001/jamanetworkopen.2024.29630

TY - JOUR

T1 - Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

AU - Levis, Brooke

AU - Bhandari, Parash Mani

AU - Neupane, Dipika

AU - Fan, Suiqiong

AU - Sun, Ying

AU - He, Chen

AU - Wu, Yin

AU - Krishnan, Ankur

AU - Negeri, Zelalem

AU - Imran, Mahrukh

AU - Rice, Danielle B

AU - Riehm, Kira E

AU - Azar, Marleine

AU - Levis, Alexander W

AU - Boruff, Jill

AU - Cuijpers, Pim

AU - Gilbody, Simon

AU - Ioannidis, John P A

AU - Kloda, Lorie A

AU - Patten, Scott B

AU - Ziegelstein, Roy C

AU - Harel, Daphna

AU - Takwoingi, Yemisi

AU - Markham, Sarah

AU - Alamri, Sultan H

AU - Amtmann, Dagmar

AU - Arroll, Bruce

AU - Ayalon, Liat

AU - Baradaran, Hamid R

AU - Beraldi, Anna

AU - Bernstein, Charles N

AU - Bhana, Arvin

AU - Bombardier, Charles H

AU - Buji, Ryna Imma

AU - Butterworth, Peter

AU - Carter, Gregory

AU - Chagas, Marcos H

AU - Chan, Juliana C N

AU - Chan, Lai Fong

AU - Chibanda, Dixon

AU - Clover, Kerrie

AU - Conway, Aaron

AU - Conwell, Yeates

AU - Daray, Federico M

AU - de Man-van Ginkel, Janneke M

AU - Fann, Jesse R

AU - Fischer, Felix H

AU - Field, Sally

AU - Fisher, Jane R W

AU - Fung, Daniel S S

AU - Depression Screening Data (DEPRESSD) PHQ Group

AU - Schram, Miranda T.

AU - Benedetti, Andrea

PY - 2024/11/4

Y1 - 2024/11/4

N2 - IMPORTANCE: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates. OBJECTIVE: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates. DESIGN, SETTING, AND PARTICIPANTS: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled. MAIN OUTCOMES AND MEASURES: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population. RESULTS: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes. CONCLUSIONS AND RELEVANCE: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

AB - IMPORTANCE: Test accuracy studies often use small datasets to simultaneously select an optimal cutoff score that maximizes test accuracy and generate accuracy estimates. OBJECTIVE: To evaluate the degree to which using data-driven methods to simultaneously select an optimal Patient Health Questionnaire-9 (PHQ-9) cutoff score and estimate accuracy yields (1) optimal cutoff scores that differ from the population-level optimal cutoff score and (2) biased accuracy estimates. DESIGN, SETTING, AND PARTICIPANTS: This study used cross-sectional data from an existing individual participant data meta-analysis (IPDMA) database on PHQ-9 screening accuracy to represent a hypothetical population. Studies in the IPDMA database compared participant PHQ-9 scores with a major depression classification. From the IPDMA population, 1000 studies of 100, 200, 500, and 1000 participants each were resampled. MAIN OUTCOMES AND MEASURES: For the full IPDMA population and each simulated study, an optimal cutoff score was selected by maximizing the Youden index. Accuracy estimates for optimal cutoff scores in simulated studies were compared with accuracy in the full population. RESULTS: The IPDMA database included 100 primary studies with 44 503 participants (4541 [10%] cases of major depression). The population-level optimal cutoff score was 8 or higher. Optimal cutoff scores in simulated studies ranged from 2 or higher to 21 or higher in samples of 100 participants and 5 or higher to 11 or higher in samples of 1000 participants. The percentage of simulated studies that identified the true optimal cutoff score of 8 or higher was 17% for samples of 100 participants and 33% for samples of 1000 participants. Compared with estimates for a cutoff score of 8 or higher in the population, sensitivity was overestimated by 6.4 (95% CI, 5.7-7.1) percentage points in samples of 100 participants, 4.9 (95% CI, 4.3-5.5) percentage points in samples of 200 participants, 2.2 (95% CI, 1.8-2.6) percentage points in samples of 500 participants, and 1.8 (95% CI, 1.5-2.1) percentage points in samples of 1000 participants. Specificity was within 1 percentage point across sample sizes. CONCLUSIONS AND RELEVANCE: This study of cross-sectional data found that optimal cutoff scores and accuracy estimates differed substantially from population values when data-driven methods were used to simultaneously identify an optimal cutoff score and estimate accuracy. Users of diagnostic accuracy evidence should evaluate studies of accuracy with caution and ensure that cutoff score recommendations are based on adequately powered research or well-conducted meta-analyses.

KW - Humans

KW - Patient Health Questionnaire

KW - Cross-Sectional Studies

KW - Depression/diagnosis

KW - Mass Screening/methods

KW - Sensitivity and Specificity

KW - Depressive Disorder, Major/diagnosis

KW - Female

KW - Male

U2 - 10.1001/jamanetworkopen.2024.29630

DO - 10.1001/jamanetworkopen.2024.29630

M3 - Article

SN - 2574-3805

VL - 7

JO - Jama network open

JF - Jama network open

IS - 11

M1 - 2429630

ER -

Data-Driven Cutoff Selection for the Patient Health Questionnaire-9 Depression Screening Tool

Abstract

Keywords

Access to Document

Fingerprint

Cite this