Logistic regression suddenly starts missing Posting the following issue just in case anyone else has had experience with it and successfully dealt with it. Would appreciate any ideas.
I have a logistic regression based model that gave excellent prediction when applied to historical data used to develop it  (several '000 data points). r-squared value of approx 50% resulting in a prediction accuracy of 70%+ which is more than adequate for the application. The prediction model was then implemented on live data and gave the same near perfect prediction performance for the next 26 data points and then suddenly started missing by increasing margins with occasional random hits. All the terms in the model are statistically significant with p-values<0.1. The VIF's are all well under 2. The following is a plot of error. - Regards Max 
 A: Such behaviour would be considerd quite common in my field, chemometrics: 
For example, imagine you have some sensors and their output is at any time related to some property of interest (think e.g. concentration of product, or in your case maybe "product meets specification"). A number of samples are taken and together with reference analyses (e.g. clearance) used to train a model. That model is then applied during production. 
At some point later, things start to change in a way that predictions become less and less useful, e.g. because:


*

*the sensors age/deteriorate, we see a drift.

*production process conditions change in a way that was not covered during training sample collection. This could be anything from permanent changes such as different supplier or seasonal (summer instead of winter) changes. 


The bottomline is that 


*

*Drift is something that will occur. The question is only: when is it going to be so bad that something needs to be done.  

*So after a time, model maintenance (update) is necessary.

*It may be good to include more checks on data quality (if sensor aging is suspected/known to be the bottleneck) in the model or as preprocessing check. 

*Also (depending on data and model), monitoring out-of-model space may be a good idea.

*Knowlege about how frequently the model performance needs to be checked (and QC techniques like control charts can be used for monitoring predictive performance) or the model needs to updated is often considered part of establishing an analytical method.
Instructions like "run/take QC samples every work day" are quite common. As these QC samples exist, they may form the basis for model updates (maintenance). 

*Ideally, the question of relevant time-frames will be tackled during method validation. 

*Particularly if we're talking modeling high dimensional data or rather indirect measurements, whoever is responsible for the model needs to know about changes in production (such as changing supplier): the model may not be robust/rugged wrt. conditions that were covered neither in training nor in validation.  
(Of course, all this depends on how crucial the predictions are - e.g. pharma production has to adapt quite different standards compared to a production line for a low grade "technical" product)
Hope this gives you some search terms to start tackling the underlying question. 
