Bayesian and Deep Learning Extensions to Dynamic Treatment Regime

Abstract

This dissertation introduces new statistical and machine learning methods for estimating optimal dynamic treatment regimes (DTR) to aid the practice of personalized medicine. Traditional Q-learning often struggles in high-dimensional treatment spaces, especially with unbalanced treatment assignment, thus limiting its clinical utility. To address these challenges, I propose two extensions. First, Bayesian Weighted Q-learning, which incorporates prior information from previous clinical trials to stabilize inference in small-sample or imbalanced settings. Adaptive Bayesian priors reduce bias from uneven allocation while improving interpretability. Second, a Conditional Variational Autoencoder (CVAE) Q-learning approach, which uses deep generative models to compress complex, high-dimensional treatment combinations into a low-dimensional latent space. This enables more accurate estimation of treatment effects and supports the discovery of optimal multitreatment strategies. Both methods are integrated into multi-stage decision-making via backward induction and evaluated through extensive simulations and real-world data from the Systolic Blood Pressure Intervention Trial (SPRINT). Results show that the approaches can uncover clinically meaningful regimes, highlighting heterogeneity in treatment benefit across patient subgroups. To support practical use, I introduce the QlearningPlus R package, which implements standard Q-learning alongside the proposed extensions, providing a unified toolset for researchers and clinicians.