5
$\begingroup$

I´m currently learning about how to determine if your model keeps performing well or has degraded (particurlarly, for classification problems). My question is, what kind of info can I derivate if I know that the distribution of the scores has changed substantially?

TIA!

Note: I'm particularly using the POPULATION STABILITY INDEX (PSI) REPORT enter image description here

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
6
$\begingroup$

This is sometimes called "concept drift:" when you're developing the model and deploying it, the scores do a nice job of matching the true values, but over time, the scores and true values start to diverge.

Concept drift arises in lots of contexts that have a time component that is not explicitly modeled. For example, new malware will be invented to evade detection, so a malware classifier using machine learning will need to be refreshed. It would be nice to have a model that automatically adapts itself, but solving that problem is a much harder task than just building a new model on some schedule.

I'm not sure that you can learn anything more specific from the observation that you're experiencing concept drift.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi! Thanks for answering. I have one question: shouldn't you notice concept drift looking at the resulting confusion matrix, once you have the 'real labels' after predicting? I mean, if you look only at the scores distribution, and it changed, you figure something is happening, but yet you don't know if it's good or bad Maybe your classifier used to give high scores, and now, because of the input data, it gives very low scores, but it's performing right! I think maybe it can be used as an alert: if scores distribution changed a lot, go check the confusion matrix, as there may be a problem $\endgroup$
    – onofricamila
    Jul 7, 2020 at 20:44
  • 1
    $\begingroup$ I think what you're saying is consistent with my first paragraph: you care about the degree to which your predictions match the labels There are lots of ways to measure this (confusion matrix, log-loss, Brier score, etc.). Concept drift is what happens when, after some time, the predicted scores no longer match the labels as well as they did on day 0. So, you might compare Brier score at day 0 with Brier score at day 90; a large mismatch could be an indicator of concept drift. Or you could measure Brier score every day and look for a trend, etc. $\endgroup$
    – Sycorax ♦
    Jul 7, 2020 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.