I understand out of sample validation very well. Can you explain what is out of time validation?
A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models sensitivity (% of actual churners correctly predicted by the model) on out of sample(test) dataset was 70% but when they rolled out the model and tracked the result after 6 months accuracy is just 47%.
Note: There is no campaign which is run till now, so we don't have any campaign start date or end date etc.
- What is out of time validation?
- How to do out of time validation?
- My thought is doing an out of time validation would have helped identify the issue with the model before it was rolled out. Is it correct?