# Better to use cross-validation or training/holdout for predictive modeling?

I'm working with a small behavioral health care dataset (22,090 records) and have been asked to develop a predictive model that identifies patients at higher risk for hospitalization & health costs in FY2013 based on information in FY2012. The final predictive model will eventually be used to flag high risk members in FY2015 based on FY2014 data.

In order to compare the performance of different methodologies (CART, SVM, logistic regression, etc.) and avoid overfitting, I'm considering two options:

• Use 5 or 10 fold cross validation on my existing data FY2012-FY2013.
• Train competing models on FY2011-FY2012 data and compare their performance on the FY2012-FY2013 dataset.

Which approach will help me find the best-fitting predictive model: cross-validation or training/holdout?

• The sample size is just barely large enough to do split-sample validation. But I would use 10-fold cross-validation or use the optimism bootstrap. May 29 '14 at 20:56
• Technically the 2nd option isn't a split sample, is it? The FY2011-FY2012 is an independent sample that will also contain ~20,000 members (well, there will be some overlap of members with the FY2012-FY2013 dataset so it's not completely independent). May 29 '14 at 21:09

I do exactly what you describe as one of my modeling duties. Your "best" solution depends on several factors. For one, how are you defining "high risk" – top X% of members? For cost predictions your most straightforward solution will be to predict cost and then rank the members. Given your limited amount of data, this could be your best option, though given tons of data, it absolutely isn't if you're only truly interested in the most costly X%. In this case, coding the top X% as 1 and everyone else as 0 or -1 will tend to work better, though as you can imagine, this becomes increasingly untenable as your N decreases. I'm also in the process of implementing a case-weighted binary target, which I suspect will work better than the two options previously mentioned, but probably not as well as my monster ensemble (though that one takes a while to train even using 8 i7 cores and 32 GB RAM ;).