What is out of time validation in logistic regression model? I understand out of sample validation very well. Can you explain what is out of time validation? 
Context
A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models sensitivity (% of actual churners correctly predicted by the model) on out of sample(test) dataset was 70% but when they rolled out the model and tracked the result after 6 months accuracy is just 47%. 
Note: There is no campaign which is run till now, so we don't have any campaign start date or end date etc. 
Questions


*

*What is out of time validation?

*How to do out of time validation?

*My thought is doing an out of time validation would have helped identify the issue with the model before it was rolled out. Is it correct?


Thanks
 A: Out-of-time validation is just out-of-sample validation on a later data-set than that on which you fitted your model; where application of a model to a population changing over time is the concern, rather than application to populations of different cities, species, materials, or whatever. So to do it you'd need samples from different times (& note that if you had those at the time of fitting the model it'd usually be more useful to use the whole data-set & include time in the model). It's anyone's guess whether it would have alerted you to a problem in this particular case.
The calibration of a model often degrades much faster than its discrimination (I'd bet that changing the probability cut-off used to predict a churner would have resulted in greater accuracy—are you monitoring discrimination & calibration?), so re-calibration once in a while can be helpful. See Steyerberg et al. (2004), "Validation and updating of predictive logistic regression: a study on sample size and shrinkage", Statistics in Medicine, 23, p.2567.
