I'm trying to do predictive analytics on recruitment data for a non-profit organisation. The organisation has a (small) army of recruiters which phone members and try to get them to volunteer at this yearly event. The list of members is completely new each year.
My goal is to build a model which indicates whether a member is likely to participate or not based on their interaction with the recruiter (they code their interaction with the member as; refusal, 'will think about it', 'maybe', yes etc.). This can guide the recruiters' about which of the members to focus extra hard on.
I have two data sets available: One set from 2015 and another from 2016 - each dataset has around 2000 observations. Based on the 2015 data my aim is to estimate a model which, based on the phone data, can predict whether a member is likely to participate or not after 6 weeks of recruitment in the 2016 data.
Question I'm confused about how I should partition the data for training, validation and testing purposes. Much of the literature I've read (also in this forum) is about how to partition the data when you have just one data set. However, in my situation, I believe I have the test data given (the 2016 data).
Which of the following approaches would you recommend:
1: Train and validate/tune models with k-fold cross-validation on the 2015 data -> then test the model on the 2016 data.
2: Train and validate/tune models on entire 2015 data (without cross-validation) -> then test the model on the 2016 data.
3: Train models on 70% of the 2015 data, validate on 30% of the 2015 data -> then test the model on the 2016 data.
4: Merge the 2015 and 2016 -> use 70% for k-fold cross-validation for training and tuning/validation -> test the models on the remaining 30%
5: Merge the 2015 and 2016 -> use 100% for k-fold cross-validation for training, tuning/validation and testing of the models.
Maybe you have a sixth approach in mind.
I guess I'm confused about whether cross-validation is really necessary when in practice the entire 2015 data can be reserved for training purposes? Or whether I should rather merge the two data sets?
Thanks a lot!