1
$\begingroup$

Okay,

I couldn't find the right phrase to use so let me explain what I mean with this question.

Lets say I build a logistic regression model for predicting the probability a customer will go rogue and stop paying his debt. Lets assume it is the best one you can build with the data I have and no further improvement with this exact data can be made.

As time passes more data is collected, of course, and some variable can (they can, right?) become less "useful/significant" to the model and the opposite can happen to other variables I am currently using. At the same time other variables can emerge and could prove to be useful to the prediction.

That being said, I was wondering how often is it "okay" to re-train and re-test your model? Meaning I extract data, split it into training and test set and do everything else as if I am building a new model.

It probably depends on the quantities of data, economic situation and other factors due to the fundamentals of the data? All things being equal (no major real world events have occurred) is there a recommended interval of time after which it is advised to re-evaluate or re-build a model (if needed of course)?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ If you use this online via some computer system, then you will get predictions from your model. Stote them, and when the "truth" arives for the case, store it too.So can can compute some cumulative measure of misfit, and when that seems to jump, there might be time to reconsider your model. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 10, 2017 at 13:42
  • $\begingroup$ @kjetilbhalvorsen Well, let me tell you a bit more about the data. I have customers which are with status A and are fine and dandy. I have customers that have a status B, which means they are canceled because they stopped paying their bills (in other words - bad customers). I have a few, around 20, non-static measurements that will 99% surely change with time. This is why i take only the probability and if it is lets say more then 50% to go bad i tell other departments to contact these customers to work the situation out so they don't get status B. $\endgroup$
    – Emil Filipov
    Commented Mar 10, 2017 at 13:47
  • $\begingroup$ @kjetilbhalvorsen Meaning. I don't know whether or not i will get the truth about my predictions. The client may go bad in a few months and that will show that i was kind of right about him, but he may stay a paying customer, although with some delay, for the rest of his existence. I guess another way to go about it is to just take really high risk customers and wait for the truth about them? $\endgroup$
    – Emil Filipov
    Commented Mar 10, 2017 at 13:52

1 Answer 1

Reset to default
2
$\begingroup$

Split-sample validation is always controversial unless you have enormous training and test samples. You might consider improving the model at will and doing a rigorous bootstrap validation at any point. Also consider putting smooth time trends in the model. As shown in my RMS book and notes (see https://hbiostat.org/rms) you are somewhat safe to be making the model more complex in an honest fashion, because adding parameters results in confidence limits for predictions that are self-penalizing (confidence bands get wider as you add more parameters, but narrower as you add more data). But most practitioners fail to show the confidence bands (not a good idea). Also consider the heuristic slope shrinkage estimator gamma-hat which estimates the slope of the calibration plot on the linear predictor scale. It is a function of the model likelihood ratio chi-square statistic and the number of candidate parameters entertained or fitted.

Improve this answer
$\endgroup$
5
  • $\begingroup$ If someone wants to continually update their model, I would think they should just switch to a Bayesian analysis in which their initial model is the prior, and they can update with new data as often as they like. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 20, 2022 at 16:18
  • $\begingroup$ @gung-ReinstateMonica Does this also work if (such) model is affecting the generation of the new data (that will be used for future updates)? Assuming OP, based on the predictions of the current/previous models, rejects certain client applications and thus is not collecting data of a certain part of the population, how will future model updates be affected? Likely, such continuous updates can erroneously discard some variables as insignificant (e.g., large debt), since every new data sample will be from clients without debt. Can one continue updating old models without generating such bias? $\endgroup$
    – runr
    Commented Apr 20, 2022 at 17:01
  • 1
    $\begingroup$ @runr, I'm not primarily a Bayesian (although, I'm not dogmatically opposed, either). There are several things that are involved in your question. 1) Bayesian updating needn't be involved in 'significance', just refining the posterior distribution. 2) Missing data & causal inferences can be addressed in a Bayesian setting not unlike in a Frequentist setting. 3) If you're using the model to make decisions that impact future data, that whole system should be modelable in a sufficiently complex setup. Others may be able to provide more detailed answers. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 20, 2022 at 17:11
  • $\begingroup$ @gung-ReinstateMonica Thanks for your response. In your opinion, Is the 3rd point of your comment worth asking a new question, or are such approaches well known and answered/defined in the literature/SE? This should be a common problem in practice, however, I've rarely seen any literature on addressing it except for "reject inference" in some very specific (accept/reject client) contexts. $\endgroup$
    – runr
    Commented Apr 21, 2022 at 13:53
  • $\begingroup$ @runr, you can always ask a new question, if you want. It strikes me as a much bigger / more complicated topic than would be able to be covered in an answer here. Maybe someone would be able to point you towards a book or some papers, IDK. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 21, 2022 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.