Are there any special considerations for creating a training and testing split when working with survival analysis? My specific problem is to use 5 years of customer data to predict retention for future customers. I am not sure how best to split off a testing set to validate predictions. For example:
- I could split by time. There are continually new entrants (new customers start at age 0) so if I split off year 5 I would still have observations where age is less than 5, but I wouldn't be able to evaluate predictions for a specific individual more than 1 year ahead.
- I could split by individual customer. With repeated measures data, I have generally seen that you should ensure each individual is either entirely in the training set or entirely in the testing set so you don't get a test set that is unusually similar to the training set.
- I could simply split randomly.
Regarding 2 and 3, would I want to stratify by age? Observations at the older ages are fairly rare.
I am leaning towards the time split being the best because there is concern that the environment may change over time and the variables I have may not be able to fully account for it. Could some sort of bootstrap validation on the training data be done to validate survival probabilities over multiple years (as well as over time)?