Bootstrap Cross-validation Improves Model Selection in Pharmacometrics

ABSTRACT

Cross-validation assesses the predictive ability of a model, allowing one to rank models accordingly. Although the nonparametric bootstrap is almost always used to assess the variability of a parameter, it can be used as the basis for cross-validation if one keeps track of which items were not selected in a given bootstrap iteration. The items which were selected constitute the training data and the omitted items constitute the testing data. This bootstrap cross-validation (BS-CV) allows model selection to be made on the basis of predictive ability by comparing the median values of ensembles of summary statistics of testing data. BS-CV is herein demonstrated using several summary statistics, including a new one termed the simple metric for prediction quality (SMPQ), and using the warfarin data included in the Monolix distribution with 13 pharmacokinetics (PK) models and 12 pharmacodynamics (PD) models. Of note the two best PK models by AIC had the worst predictive ability, underscoring the danger of using single realizations of a random variable (such as AIC) as the basis for model selection. Using these data BS-CV was able to discriminate between similar indirect response models (inhibition of input versus stimulation of output). This could be useful in situations in which the mechanism of action is unknown (unlike warfarin).

Keywords:

• 0.632 bootstrap
• penalized likelihood
• AIC
• warfarin
• Monolix
• predictive ability
• indirect response model

Acknowledgements

This work comes from my masters project, Using Resampling to Improve Model Selection in Pharmacometrics, defended on 2019-01-18 and chaired by Jill Fiedler-Kelly, President of Cognigen Corp., a SimulationsPlus Company, and Adjunct Associate Professor, Dept. of Pharmaceutical Sciences, The University at Buffalo, SUNY Buffalo, NY, USA. I thank Jill for serving in this capacity. Jill was especially helpful in an earlier model selection project, which, unfortunately, did not work out as a useful methodology. Had that project been the subject of this paper, she would most definitely be listed as a co-author. I also thank Robert R. Bies, Associate Professor in the Dept. of Pharmaceutical Sciences, The University at Buffalo, for serving as my committee member and for suggesting investigating ε-shrinkage. I thank them both for their editorial comments in the earlier write-up of this work as a thesis. I also thank Greg Warnes for helpful conversations and for bringing to my attention the 0.632 bootstrap.