If i have a classification or clustering model built for retail customers till last year.How do i check if my model is still valid to this year's data?
We can check accuracy of model to new data but what are the other methods to check?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
I would use an iterative holdout cross-validation approach, in which you train the model on a set of historical data, and test it on the next pseudo-future validation set.
For example, lets say you are interested in using 2015 data to predict 2016 performance. To evaluate how it might perform, you could evaluate this same model of prediction by training on 2010 data to predict 2011, 2011 data to predict 2012, etc. This would give you an idea of how it might perform for the upcoming year, which you could assume will be representative of 2016. Unless there is a business shift that would cause a substantial change in the data generating process, this is a solid approach to estimating how a time series forecast will into a specified future period.
Finally, as new data become available for 2016, you can dynamically track your model's performance.