1
$\begingroup$

If i have a classification or clustering model built for retail customers till last year.How do i check if my model is still valid to this year's data?

We can check accuracy of model to new data but what are the other methods to check?

$\endgroup$
1
$\begingroup$

I would use an iterative holdout cross-validation approach, in which you train the model on a set of historical data, and test it on the next pseudo-future validation set.

For example, lets say you are interested in using 2015 data to predict 2016 performance. To evaluate how it might perform, you could evaluate this same model of prediction by training on 2010 data to predict 2011, 2011 data to predict 2012, etc. This would give you an idea of how it might perform for the upcoming year, which you could assume will be representative of 2016. Unless there is a business shift that would cause a substantial change in the data generating process, this is a solid approach to estimating how a time series forecast will into a specified future period.

Finally, as new data become available for 2016, you can dynamically track your model's performance.

$\endgroup$
  • $\begingroup$ thanks for the explanation. What if new data has different bahaviour compared to old one,how to proceed in this case? Are there any graphs to look at for these kind of scenarios? $\endgroup$ – alily Apr 20 '16 at 18:26
  • $\begingroup$ If the new data has different behavior (i.e. the data generating process has changed), it will require model redevelopment to account for the process that is changing. Unfortunately this is very problem specific, so I can't really provide a general graph to look at. Basically just try to find what is causing the problem, and try to account for it in the model. $\endgroup$ – Underminer Apr 20 '16 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.