Interactive Q-Learning for Quantiles

ABSTRACT

A dynamic treatment regime is a sequence of decision rules, each of which recommends treatment based on features of patient medical history such as past treatments and outcomes. Existing methods for estimating optimal dynamic treatment regimes from data optimize the mean of a response variable. However, the mean may not always be the most appropriate summary of performance. We derive estimators of decision rules for optimizing probabilities and quantiles computed with respect to the response distribution for two-stage, binary treatment settings. This enables estimation of dynamic treatment regimes that optimize the cumulative distribution function of the response at a prespecified point or a prespecified quantile of the response distribution such as the median. The proposed methods perform favorably in simulation experiments. We illustrate our approach with data from a sequentially randomized trial where the primary outcome is remission of depression symptoms. Supplementary materials for this article are available online.

KEYWORDS:

• Dynamic treatment regime
• Personalized medicine
• Sequential decision making
• Sequential multiple assignment randomized trial

Supplementary Materials

Online supplementary materials include discussions of modeling adjustments for heteroscedastic second-stage errors and patient-specific thresholds, a proof of Lemma 3.1 and toy example illustrating where this lemma does not apply, additional simulation results, and proofs of the theorems in Section 3.3.

Funding

Eric Laber acknowledges support from NIH grant P01 CA142538 and DNR grant PR-W-F14AF00171. Leonard Stefanski acknowledges support from NIH grants R01 CA085848 and P01 CA142538 and NSF grant DMS-0906421 and DMS-1406456.