4
$\begingroup$

I understand out of sample validation very well. Can you explain what is out of time validation?

Context
A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models sensitivity (% of actual churners correctly predicted by the model) on out of sample(test) dataset was 70% but when they rolled out the model and tracked the result after 6 months accuracy is just 47%.

Note: There is no campaign which is run till now, so we don't have any campaign start date or end date etc.

Questions

  1. What is out of time validation?
  2. How to do out of time validation?
  3. My thought is doing an out of time validation would have helped identify the issue with the model before it was rolled out. Is it correct?

Thanks

$\endgroup$
3
$\begingroup$

Out-of-time validation is just out-of-sample validation on a later data-set than that on which you fitted your model; where application of a model to a population changing over time is the concern, rather than application to populations of different cities, species, materials, or whatever. So to do it you'd need samples from different times (& note that if you had those at the time of fitting the model it'd usually be more useful to use the whole data-set & include time in the model). It's anyone's guess whether it would have alerted you to a problem in this particular case.

The calibration of a model often degrades much faster than its discrimination (I'd bet that changing the probability cut-off used to predict a churner would have resulted in greater accuracy—are you monitoring discrimination & calibration?), so re-calibration once in a while can be helpful. See Steyerberg et al. (2004), "Validation and updating of predictive logistic regression: a study on sample size and shrinkage", Statistics in Medicine, 23, p.2567.

$\endgroup$
  • $\begingroup$ Can you give an example? Suppose I have data from Jan 2012 to Dec 2014 and July to Dec 2014 is my performance window. From which all months should I take data from test sample and use as out of time sample? $\endgroup$ – GeorgeOfTheRF Feb 16 '15 at 13:11
  • $\begingroup$ Well, a simple way would be to fit a model on Jan 12 to Jun 14 data & validate it on July 14 to Dec 14 data. Of course how much data you have & any changes over time that you know about will influence the best way to split the sample. $\endgroup$ – Scortchi Feb 16 '15 at 13:40
  • $\begingroup$ Ok. In this case we not using most recent data for modelling. Won't that affect the accuracy of model etc? Is the idea of out of time validation not to use some time for modeling and test the model on that time? $\endgroup$ – GeorgeOfTheRF Feb 16 '15 at 14:00
  • $\begingroup$ Yes, setting aside the most recent data for validation affects the model's accuracy to some extent - depending on how much data you have & how the population changes over time. That's why accounting for time in the model is preferable - that could be as simple as including a trend &/or seasonal term, perhaps interactions of other predictors with trend/seasonality, or involve more complex methods such as GLMMs. $\endgroup$ – Scortchi Feb 16 '15 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.