I'm working with a small behavioral health care dataset (22,090 records) and have been asked to develop a predictive model that identifies patients at higher risk for hospitalization & health costs in FY2013 based on information in FY2012. The final predictive model will eventually be used to flag high risk members in FY2015 based on FY2014 data.
In order to compare the performance of different methodologies (CART, SVM, logistic regression, etc.) and avoid overfitting, I'm considering two options:
- Use 5 or 10 fold cross validation on my existing data FY2012-FY2013.
- Train competing models on FY2011-FY2012 data and compare their performance on the FY2012-FY2013 dataset.
Which approach will help me find the best-fitting predictive model: cross-validation or training/holdout?