I'm working on a model to predict churn. I understand the concept of training and testing, or at least I thought I did.
Let's say it's the first of the month and our database has 10,000 subscribers, in addition to 10,000 who have cancelled. I now have a dataset of 20k records. If I wanted to build a model to predict which of these 10K active subscribers are likely to cancel, with a view to sending them an email with e.g. a discount offer code, how would I split my data between training and testing, and then for prediction?
- Take the entire dataset of 20K, split 80% train and 20% test. But then, what do I predict on? Do I want to use all data to get best training since I'll have more records?
- Take a sample of say 10k records from the total dataset, test and train with 0.8 and 0.2 as above and then predict on the whole data set? But then that would mean I'm predicting on training data? That must be "wrong" surely?
This made sense in class. Now that I'm trying to apply it to real world I'm a little lost.
Of the 20k records, what would be a "traditional" way of splitting it up between test and training data, and then which data for prediction?