Better to use cross-validation or training/holdout for predictive modeling?

Question

I'm working with a small behavioral health care dataset (22,090 records) and have been asked to develop a predictive model that identifies patients at higher risk for hospitalization & health costs in FY2013 based on information in FY2012. The final predictive model will eventually be used to flag high risk members in FY2015 based on FY2014 data.

In order to compare the performance of different methodologies (CART, SVM, logistic regression, etc.) and avoid overfitting, I'm considering two options:

• Use 5 or 10 fold cross validation on my existing data FY2012-FY2013.
• Train competing models on FY2011-FY2012 data and compare their performance on the FY2012-FY2013 dataset.

Which approach will help me find the best-fitting predictive model: cross-validation or training/holdout?

Answer 1

I do exactly what you describe as one of my modeling duties. Your "best" solution depends on several factors. For one, how are you defining "high risk" – top X% of members? For cost predictions your most straightforward solution will be to predict cost and then rank the members. Given your limited amount of data, this could be your best option, though given tons of data, it absolutely isn't if you're only truly interested in the most costly X%. In this case, coding the top X% as 1 and everyone else as 0 or -1 will tend to work better, though as you can imagine, this becomes increasingly untenable as your N decreases. I'm also in the process of implementing a case-weighted binary target, which I suspect will work better than the two options previously mentioned, but probably not as well as my monster ensemble (though that one takes a while to train even using 8 i7 cores and 32 GB RAM ;).

Part of this also depends on the number of predictors you're using; you need more observations as the number of predictors increases. Are you using EHR or just claims data?

Don't shut out the idea of training on FY2011–2012 and then testing on a bootstrap oversampled FY2012–2013. Or you could create 10 bootstrapped FY2011–2012 test sets and average the performance across models. I like this option for model selection. Since your final model will really be trained on both years (I'm assuming), you can, after model selection, do k-fold CV for a less biased estimate of future performance.

I'm not opposed to talking to you about this problem privately.