I need to determine a statistically sound methodology for creating training and validation datasets for a churn model. Testing sets and model selection aren't a problem.
The data spans 4 years of business trading. Customers have 12 month contracts. When contracts are up for renewal customers either renew for 12 months or churn. Customers start contracts at different times of the year. Renewal always lags contract inception by 12 months.
The aim is to use a classifier to produce a churn probability score for each customer requiring renewal in an upcoming 3 month period, regardless of their contract year (tenure). The scoring would occur for a new batch of customers every 3 months.
Here's a made up example of what a scoring batch could look like. Say between Jun and Aug 2018 the following customers are up for renewal: 3000 1st yr customers, 2000 2nd yr customers, 1500 3rd yr customers.
Below are the best options I can think of. However I don't have enough knowledge to know which makes the most sense.
- Create training and validation sets using all 4 years of data. Score each batch of customers requiring renewal in the upcoming 3 month period against this model. Add 3 months of data to the main training set every 3 months.
- Create separate models for all customers who churned historically at year 1, year 2, year 3, etc. Separate each batch of customers to score based on tenure. Score 1st year contract holders against year 1 model, 2nd year contract holders against year 2 model, etc.
- Use a rolling window. For example, train on the last 12 months of data. Score on customers requiring renewal in the upcoming 3 month period. Then shift both the training and scoring along 3 months every 3 months.
Which is the most suitable option? Or is there a better approach?