Using the Personalized Advantage Index to determine which veterans may benefit from more vs. less comprehensive intensive PTSD treatment programs

ABSTRACT

Background: Intensive PTSD treatment programs (ITPs) are highly effective but tend to differ greatly in length and the number of adjunctive services that are provided in conjunction with evidence-based PTSD treatments. Individuals’ treatment response to more or less comprehensive ITPs is poorly understood.

Objective: To apply a machine learning-based decision-making model (the Personalized Advantage Index (PAI)), using clinical and demographic factors to predict response to more or less comprehensive ITPs.

Methods: The PAI was developed and tested on a sample of 747 veterans with PTSD who completed a 3-week (more comprehensive; n = 360) or 2-week (less comprehensive; n = 387) ITP.

Results: Approximately 12.32% of the sample had a PAI value that suggests that individuals would have experienced greater PTSD symptom change (5 points) on the PTSD Checklist for DSM-5 in either a more- or less comprehensive ITP. For individuals with the highest 25% of PAI values, effect sizes for the amount of PTSD symptom change between those in their optimal vs. non-optimal programs was d = 0.35.

Conclusions: Although a minority was predicted to have benefited more from a program, there generally was not a substantial difference in predicted outcomes. Less comprehensive and thus more financially sustainable ITPs appear to work well for most individuals with PTSD.

HIGHLIGHTS

• A Personalized Advantage Index (PAI) was developed for a 3-week (more comprehensive) and a 2-week (less comprehensive) intensive PTSD treatment program to predict treatment responses.

• Using the PAI, approximately 12% of the sample was predicted to have experienced meaningfully greater in another program than the one in which they participated.

• Findings suggest a less comprehensive and more financially sustainable 2-week intensive PTSD treatment program would work well for most veterans in the present study.

Antecedentes: Los programas de tratamiento intensivo de TEPT (ITP en su sigla en inglés) son muy efectivos, pero tienden a diferir mucho en la duración y la cantidad de servicios complementarios que se brindan junto con los tratamientos de TEPT basados en evidencia. La comprensión de la respuesta de los individuos al tratamiento de ITPs más o menos integrales es deficiente.

Objetivo: Aplicar un modelo de toma de decisiones basado en aprendizaje automático (el Índice de Ventajas Personalizadas [PAI en su sigla en inglés]), utilizando factores clínicos y demográficos para predecir la respuesta a ITPs más o menos integrales.

Métodos: El PAI se desarrolló y probó en una muestra de 747 veteranos con TEPT que completaron un ITP de 3 semanas (n = 360) o 2 semanas (n = 387).

Resultados: Aproximadamente el 12.32% de la muestra tuvo un valor de PAI que indicó un cambio clínicamente significativo en los síntomas de TEPT (5 puntos) en la Lista de verificación de TEPT. Para los individuos con el 25% más alto de valores de PAI, el tamaño del efecto para el cambio en la cantidad de síntomas de TEPT entre aquellos en programas óptimos versus no óptimos fue d = 0.35.

Conclusiones: Aunque se predijo que una minoría se había beneficiado más de otro programa, en general no hubo una diferencia sustancial en los resultados previstos. Los ITPs menos completos y, por tanto, más sostenibles desde el punto de vista financiero parecen funcionar bien para la mayoría de las personas con trastorno de estrés postraumático.

背景：强化 PTSD 治疗计划 (ITP) 非常有效，但在长度和与循证的 PTSD 治疗结合提供的辅助服务数量方面往往存在很大差异。人们对或多或少全面 ITP 的治疗反应知之甚少。

目的：应用基于机器学习的决策模型（个性化优势指数（PAI）），利用临床和人口因素来预测对或多或少全面 ITP 的反应。

方法：PAI 在 747 名患有 PTSD、完成了 3 周（n = 360）或 2 周（n = 387）ITP的退伍军人样本中开发和测试，这些退伍军人。

结果：大约 12.32% 的样本的 PAI 值表明 PTSD 检查表上有临床意义的 PTSD 症状变化（5 分）。对于 PAI 值最高 25% 的个体，最佳方案与非最佳方案之间 PTSD 症状变化量的效应大小为 i = 0.35。

结论：尽管预计少数人会从另一个计划中受益更多，但预测结果通常没有显著差异。不太全面但在财务上更可持续的 ITP 似乎对大多数PTSD患者有效。

KEYWORDS:

• Cognitive processing therapy
• intensive treatment
• massed treatment
• treatment selection
• Personalized Advantage Index
• veterans
• PTSD

PALABRAS CLAVE:

• Terapia de procesamiento cognitivo
• tratamiento intensivo
• tratamiento masivo
• selección de tratamiento
• Índice de ventajas personalizado
• veteranos
• Trastorno de estrés postraumático

关键词:

• 认知加工疗法； 强化治疗； 集中治疗； 治疗选择； 个性化优势指数； 退伍军人； PTSD

1. Introduction

Machine learning-based decision-making models have been developed to personalise treatment processes. One such example, the Personalized Advantage Index (PAI), weighs multiple variables to identify predictive indicators associated with the best clinical outcomes. The PAI estimates the extent to which an individual’s predicted optimal treatment would outperform other treatments (DeRubeis et al., 2014; Huibers et al., 2015; van Bronswijk et al., 2021).

Personalised medicine has great potential for individuals seeking posttraumatic stress disorder (PTSD) treatment, as individuals respond differently to established interventions (Deisenhofer et al., 2018; Herzog & Kaiser, 2022; Stirman et al., 2021). PTSD treatments are increasingly delivered in intensive treatment programs (ITPs), which range from 1 to 3 weeks and use varying amounts of adjunctive services to complement daily evidence-based PTSD treatments (Held et al., 2022; Sciarrino et al., 2020). Given ITPs differ in their comprehensiveness, it is important to better understand whether the comprehensiveness of ITPs makes some individuals more or less likely to respond, even when more and less comprehensive ITPs have previously been shown to be generally equally as effective (Held et al., 2022).

The present study examined the PAI in the context of more vs. less comprehensive intensive PTSD treatment formats (i.e. the length of treatment and the amount of adjunctive services). Although the two ITPs examined in the present study, a 3-week (more-) and a 2-week (less comprehensive) CPT-based IPT, have previously been demonstrated to produce equivalent outcomes for most individuals, we expected that some individuals would be predicted to experience meaningfully greater PTSD symptom reduction in one program over another. We conducted this study with the goal that insights could further aid in personalising treatments for individuals with PTSD by helping us better understand who may benefit from more vs. less comprehensive ITPs.

2. Method

2.1. Participants

All study procedures were approved by the Rush University Medical Center Institutional Review Board. A waiver of consent was obtained because assessments collected were part of routine clinical care. A total of 360 veterans with PTSD who completed the 3-week ITP between April 2016 and March 2020 as well as 387 veterans with PTSD who completed a 2-week ITP between June 2020 and August 2022 were included in this study. Demographic characteristics are shown in Table 1. Only participants who completed final PCL-5 assessment and had relevant demographic and baseline severity measurements were included in analyses.

Table 1. Demographic characteristics for veterans who participated in 2- and 3-week intensive PTSD treatment programs.

	2-Week ITP (n = 387)		3-Week ITP (n = 360)
	N	%	N	%
Sex
Female	186	48.06	118	32.78
Male	201	51.94	242	67.23
Ethnicity
Not Latinx	314	81.14	288	80.00
Race
American Indian/Alaskan Native	5	1.29	7	1.95
Asian	10	2.58	6	1.67
Black or African American	81	20.93	75	20.84
Native Hawaiian/Pacific Islander	7	1.81	1	0.28
Other	29	7.49	28	7.78
Refusal	2	0.52	1	0.28
White	253	65.38	241	66.94
Marital status
Married	186	49.47	187	52.38
Single	89	23.67	76	21.29
Divorced	86	22.87	79	22.13
Widowed	1	0.27	1	0.28
Domestic partner	1	0.27	1	0.28
Legally separated	9	2.39	13	3.64
Military service branch^a
Air Force	48	12.40	28	7.78
Army	216	55.81	235	65.28
Coast Guard	5	1.29	3	0.83
Marines	63	16.28	59	16.39
Navy	54	13.95	34	9.45
Service era
Post 11 September 2001	321	67.70	326	90.56
Deployed
Yes	262	82.95	281	78.06
	M	SD	M	SD
Age	44.20	9.66	41.58	9.43

^a Percentages may not add up to 100 because participants could self-report serving in multiple branches; ITP = Intensive PTSD Treatment Program.

2.2. Intensive PTSD treatment programs

2.2.1. 3-week ITP

The more comprehensive 3-week ITP consisted of a total of 104 h of direct clinical care. In addition to 14 daily individual CPT sessions, individuals received 13 daily group CPT sessions, four art therapy sessions, 13 mindfulness sessions, 12 yoga sessions, as well as 19 educational classes on a variety of topics, such as emotion regulation skills, healthy cooking, communication, substance use education and sleep habits.

2.2.2. 2-week ITP

The less comprehensive 2-week ITP consisted of a total of 67 h of direct clinical care. Individuals received 16 twice daily individual CPT sessions, each followed by a homework session, 13 combined mindfulness/yoga sessions, 7 cognitive restructuring or emotion regulation skill-focused sessions, and 5 art therapy sessions. More comprehensive information about the two ITPs can be found elsewhere (Held et al., 2022).

2.3. Measures

At baseline, veterans reported demographic characteristics, including age, sex, race, ethnicity, cohort type, and post-11 September 2001 service status, and other demographic variables (see Table 1 and Supplemental Table 1). PTSD severity change was the primary outcome measure for this study; PTSD severity was assessed at baseline and post-treatment was assessed via the PTSD Checklist for DSM-5 (PCL-5; Weathers et al., 2013). A large number of baseline clinical predictors were explored, though only depression (Patient Health Questionnaire (PHQ-9; Kroenke et al., 2001)), negative posttrauma cognitions (Posttraumatic Cognitions Inventory (PTCI; Foa et al., 1999)), and physical and mental functioning (VA Rand 12 Survey (Selim et al., 2009)) were used in the final analyses. See Supplemental Materials for a full description of predictors of PTSD symptom change that led to the selection of the described variables.

2.4. Statistical analysis

To generate models assessing potential differential predictions for individuals based on the ITP comprehensiveness, we followed the basic framework for generation of a PAI (DeRubeis et al., 2014; van Bronswijk et al., 2021). We used a variable selection algorithm involving recursive partitioning via random forest with 5-folds cross validation (see Supplemental Materials) to explore potential predictors of change in PTSD severity during the ITP and potential moderators of change in PTSD differences by ITP comprehensiveness. Variables selected as important predictors of PCL-5 change were next examined as prognostic or prescriptive variables by determining whether they predicted change in PCL-5 or interacted with program comprehensiveness in predicting change in PCL-5.

We then employed leave-one-out cross validation (LOOCV) in model generation using linear regression models to predict PCL-5 change during the program. This involved creating separate models for each of the 747 individuals, in which all individuals except the individual of interest are included in generating the model. A factual prediction was generated for each individual using their actual treatment program, as well as a counterfactual prediction, which repeats the individual’s prediction under the counterfactual condition that they had been in the other ITP. The ITP with the highest amount of predicted PCL-5 change for an individual could be conceptualised as their optimal treatment program. From these values, an individual’s PAI is the absolute difference between the prediction for the individual’s amount of PCL-5 change during the program if receiving their optimal ITP versus receiving their suboptimal ITP. A sensitivity analysis also examined robustness of findings to balancing demographic variables across treatment programs using propensity scores (see Supplemental Materials). Analyses were conducted in R version 4.2.2 (R Core Team, 2022).

3. Results

Exploration of variable selection algorithms indicated that demographic variables were neither important predictors of PCL-5 changes nor interacting significantly with ITP comprehensiveness in predicting PCL-5 changes. Of the clinical variables, baseline PCL-5 was the predictor that accounted for most variability in PCL-5 change. Other clinical variables did not generally account for meaningful proportions of PCL-5 change after adjusting for this. However, interactions between ITP comprehensiveness and both PTCI and VR12 MCS emerged as significant predictors of PCL-5 change and were included in final prediction models along with their appropriate interaction terms in generating PAI values.

The PAI values for the sample ranged from 0.01 to 13.50 (M = 2.65, SD = 2.07), with approximately 12.32% of the sample indicating PAI values above 5, which is considered reliable change (National Center for PTSD, 2022). Those individuals who were determined to have participated in their optimal ITP saw an average PCL-5 reduction of 22.13 points (SD = 17.55). Individuals classified to have participated in their suboptimal ITP saw an average PCL-5 reduction of 19.84 points (SD = 16.90). Among those with the highest 25% of PAI values, effect sizes for the amount of PCL-5 change between those in their optimal vs non-optimal programs was d = 0.35 (see Supplemental Materials).

Although prediction differences based on ITP comprehensiveness were modest for most individuals, as expected given prior studies using similar data that demonstrated equivalent outcomes between programs, a small subset had PAI values that were potentially clinically significant (i.e. 10-point PCL-5 change; National Center for PTSD, 2022). For example, an individual with a baseline PCL-5 score of 29 and baseline PTCI and VR12 MCS scores of 67 and 50.87, respectively, would be predicted to improve by 11 PCL-5 points in the 2-week ITP, but by 21 PCL-5 points in the 3-week ITP. Conversely, an individual with a baseline PCL-5 score of 67 and with baseline PTCI and VR12 MCS scores of 230 and 14.80, respectively, would be predicted to improve by 23 PCL-5 points if in the 2-week ITP program, but 15 PCL-5 points if in the 3-week program. Thus, in their optimally predicted program, individuals’ PCL-5 changes would be closer to the overall program averages previously reported for the 3-week (average change: 18.65 points) and 2-week ITPs (average change: 21.64 points) (Held et al., 2022). Overall, among participants in the 3-week program, 52.66% were predicted to have performed better in the 2-week program, while 50.40% of the 2-week sample were predicted to perform better in the 3-week program.

4. Discussion

The goal of the present study was to develop a PAI to identify individuals who would be predicted to respond differently if they participated in a more (i.e. 3-week) or less (i.e. 2-week) comprehensive CPT-based ITP. Given that the two ITPs examined for this study have previously been shown to produce equivalent PTSD symptom reductions (3-week PCL-5 change: 18.65 points; 2-week PCL-5 change: 21.64 points) despite the differences in the program length and treatment components in addition to CPT (Held et al., 2022), we did not expect to find substantial differences in predicted outcomes for the majority of individuals. As expected, 87.68% were predicted to have PCL-5 change score differences of 4 points or less regardless of the ITP in which they would have participated. These lack of substantial differences for the majority of the sample aligns with prior PAI research (van Bronswijk et al., 2021) and support that a less comprehensive ITP will work well for the majority of individuals (Held et al., 2022).

A small number of individuals were predicted to experience meaningfully different PTSD symptom reductions depending on the comprehensiveness of the ITP in which they would have participated. ITP fit was driven primarily by differing severity of baseline PTSD severity, negative posttrauma cognitions, and mental functioning rather than demographics (see Supplemental Materials for additional information about the final PAI). For individuals who were predicted to achieve meaningfully greater PTSD symptom reductions in one ITP over the other, the PAI suggested that differences could be as large as 10 PCL-5 points. In the present study, approximately 6.3% of individuals were predicted to have benefited from additional adjunctive services and a slightly longer treatment program. It is possible that the 3-week ITP’s additional skill building sessions and group CPT, where individuals may receive input on and help with challenging their negative posttrauma cognitions, provided additional scaffolding and time to practice needed for some individuals to benefit from CPT. Conversely, for individuals predicted to respond less favourably to the longer and more comprehensive ITP, the additional services might ‘dilute’ the effects of CPT, which is hypothesised to have the biggest impact on their PTSD symptom severity. It may also be possible that the inclusion of the cognitive restructuring sessions, which were not part of the 3-week ITP, were critical in reducing symptoms in the less comprehensive program.

The study findings offer clinical and operational implications. Given that most individuals experienced comparable PTSD symptom reductions, and assuming that future research can replicate these findings, we suggest prioritising less comprehensive ITP treatment formats. Longer and more comprehensive ITPs that include a much greater focus on adjunctive services in addition to an evidence-based PTSD treatment could be offered on an as-needed basis for individuals where PAI scores indicate that they would experience meaningfully larger symptom reductions in such programs. This would be more parsimonious financially and, from a public health perspective, would also allow for more individuals to receive care. By developing and offering more flexible programming models, where more treatment is only provided to those who will clearly benefit from it, will enable individuals to receive the treatment that is predicted to be optimal for them.

4.1. Limitations

First, the clinical programs, although they occurred within the same organisation, did not occur at the same time, have the exact same treatment components, or have the same treatment providers. Thus, differences could have been due to clinician factors or historical factors. Second, the study relied primarily on self-report measures to develop the PAI, which may have limited its ability to predict treatment response. Moreover, some of the self-reported measures used timeframes that overlapped with treatment (e.g. past week), which may impact the interpretability of findings. Additionally, use of change scores ignores potential variation in severity measurements and can lead to reliability concerns, though they are routinely used in assessing treatment-based change, and have been supported under adequate modelling conditions (Mattes & Roheger, 2020). Finally, given the nature of this study, we were unable to determine the exact reasons for greater predicted success in one ITP over the other. Therefore, possible explanations above should be viewed as speculations that warrant further study.

5. Conclusion

Developing a prognostic index can help identify individuals who are likely to respond optimally to more or less comprehensive ITPs, which can help match individuals to treatments in which they are expected to be most successful. Clinically, it will be critical to study how insights about predicted treatment response can be effectively shared with individuals and integrated with other best practices to enhance treatment engagement, for example as part of shared decision making.

Acknowledgements

We would like to thank the participating veterans and their families, as well as acknowledge the administrators, research assistants, and clinicians at the Road Home Program.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets used in the current study are not publicly available. Datasets can be obtained from the corresponding author upon reasonable request.

Additional information

Funding

Philip Held receives grant support from Wounded Warrior Project®, the Department of Defense [grant number W81XWH-22-1-0739], and the Agency for Healthcare Research and Quality [grant number R21 HS028511]. Debra Kaysen receives grant support from the Department of Defense [CDMRP; Research Grant W81XWH-17-1-0002] and receives support from the Veterans Administration. The content is solely the responsibility of the authors and does not necessarily represent the official views of Wounded Warrior Project®, the Department of Defense, the Agency for Healthcare Research and Quality, Veterans Administration, or any other funding agency.