I couldn't find the right phrase to use so let me explain what I mean with this question.

Lets say I build a logistic regression model for predicting the probability a customer will go rogue and stop paying his debt. Lets assume it is the best one you can build with the data I have and no further improvement with this exact data can be made.

As time passes more data is collected, of course, and some variable can (they can, right?) become less "useful/significant" to the model and the opposite can happen to other variables I am currently using. At the same time other variables can emerge and could prove to be useful to the prediction.

That being said, I was wondering how often is it "okay" to re-train and re-test your model? Meaning I extract data, split it into training and test set and do everything else as if I am building a new model.

It probably depends on the quantities of data, economic situation and other factors due to the fundamentals of the data? All things being equal (no major real world events have occurred) is there a recommended interval of time after which it is advised to re-evaluate or re-build a model (if needed of course)?

  • 1
    $\begingroup$ If you use this online via some computer system, then you will get predictions from your model. Stote them, and when the "truth" arives for the case, store it too.So can can compute some cumulative measure of misfit, and when that seems to jump, there might be time to reconsider your model. $\endgroup$ – kjetil b halvorsen Mar 10 '17 at 13:42
  • $\begingroup$ @kjetilbhalvorsen Well, let me tell you a bit more about the data. I have customers which are with status A and are fine and dandy. I have customers that have a status B, which means they are canceled because they stopped paying their bills (in other words - bad customers). I have a few, around 20, non-static measurements that will 99% surely change with time. This is why i take only the probability and if it is lets say more then 50% to go bad i tell other departments to contact these customers to work the situation out so they don't get status B. $\endgroup$ – Emil Filipov Mar 10 '17 at 13:47
  • $\begingroup$ @kjetilbhalvorsen Meaning. I don't know whether or not i will get the truth about my predictions. The client may go bad in a few months and that will show that i was kind of right about him, but he may stay a paying customer, although with some delay, for the rest of his existence. I guess another way to go about it is to just take really high risk customers and wait for the truth about them? $\endgroup$ – Emil Filipov Mar 10 '17 at 13:52

Split-sample validation is always controversial unless you have enormous training and test samples. You might consider improving the model at will and doing a rigorous bootstrap validation at any point. Also consider putting smooth time trends in the model. As shown in my RMS book and notes (see http://www.fharrell.com/p/blog-page.html) you are somewhat safe to be making the model more complex in an honest fashion, because adding parameters results in confidence limits for predictions that are self-penalizing (confidence bands get wider as you add more parameters, but narrower as you add more data). But most practitioners fail to show the confidence bands (not a good idea). Also consider the heuristic slope shrinkage estimator gamma-hat which estimates the slope of the calibration plot on the linear predictor scale. It is a function of the model likelihood ratio chi-square statistic and the number of candidate parameters entertained or fitted.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.