Cross-trial prediction in psychotherapy: External validation of the Personalized Advantage Index using machine learning in two Dutch randomized trials comparing CBT versus IPT for depression

AbstractObjective: Optimizing treatment selection may improve treatment outcomes in depression. A promising approach is the Personalized Advantage Index (PAI), which predicts the optimal treatment for a given individual. To determine the generalizability of the PAI, models needs to be externally validated, which has rarely been done. Method: PAI models were developed within each of two independent trials, with substantial between-study differences, that both compared CBT and IPT for depression (STEPd: n = 151 and FreqMech: n = 200). Subsequently, both PAI models were tested in the other dataset. Results: In the STEPd study, post-treatment depression was significantly different between individuals assigned to their PAI-indicated treatment versus those assigned to their non-indicated treatment (d = .57). In the FreqMech study, post-treatment depression was not significantly different between patients receiving their indicated treatment versus those receiving their non-indicated treatment (d = .20). Cross-trial predictions indicated that post-treatment depression was not significantly different between those receiving their indicated treatment and those receiving their non-indicated treatment (d = .16 and d = .27). Sensitivity analyses indicated that cross-trial prediction based on only overlapping variables didn't improve the results. Conclusion: External validation of the PAI has modest results and emphasizes between-study differences and many other challenges.