How to do model maintenance in production environment? So let's suppose I have built a brand new logistic regression model for fraud detection and have moved that to production. However, with time, there have been changes in the behavior of customers and consequently, the model's performance deteriorates over time. How can I retrain the model in the production environment to enhance its performance?
 A: This seems like a problem with more than one right answer, but here's one.
A classic approach to model management is to take a champion/ challenger approach. That is, the current model is the 'champion'. Any new model is the 'challenger'. For the new model to be adopted, it must improve performance on the agreed metric(s) (or, given multiple metrics, improve on one without degradation on others). In contrast to the apparent assumption of your question, this is not done 'in production', rather the challenger model and the champion model are validated on data extracted from the production environment. The champion model runs undisturbed in production until all tests are completed, along with any necessary business processes (see below).
The business I work at is currently going through this process. While on one level, this is exactly as simple as the above description makes it sound, there are other considerations that add to the complexity. One aspect that we have given a lot of thought to is that by definition changing the model is change, and the new model won't behave as before. Hence, in addition to ensuring that the new model really is better than the old, there is a task of ensuring through some kind of communication process, so that all the model users understand there has been a change, and understand how to interpret the new model correctly. You don't want people secretly running the old model somehow from the Excel sheet on their C:\ drive, for example (or making an extract of the old model's results and continuing to apply them).
