I have built a classic binary classifier and constructed a ROC curve for it, like the following: ROC Curve

In this case, the positive class represents "bad" things that should be excluded. In the wild, the environment changes - often rapidly - between populations with low numbers of positives and high numbers of positives. In populations with low numbers of positives, I wish to exclude very little so I use a cutpoint on the left side of the ROC curve. In populations with high numbers of positives, I'm ok accidentally excluding many negatives (say up to FPR=10-15%) as a protective measure so I use a cutpoint further to the right. In all, I have three cutpoints that I use for safe, neutral, and unsafe environments. For changing between environments, I use a crude method whereby when the number of detected positives goes up, I start using more sensitive cutpoints.

However, all of this feels rather pragmatic. I'm quite certain that there is some method where I can use my observed thresholds (likely weighted with respect to sampling time) to treat this as a continuous problem, perhaps with some type of "inertia"/"momentum" for how quickly to be transitioning between cutpoints as the observations change. It seems the first step in doing this is possibly to use multiple recently observed thresholds to estimate the current environment's true positive rate or "location" on the ROC curve. One thought I've had is that I could aggregate the observed confusion matrices corresponding to the observed thresholds from a lookup table, but I'm a bit wary of this approach as e.g. summing them may not be the correct way to think about the problem.

So, what kind of tools are available for adaptively selecting the cutpoint in changing environments? Or estimating "where" on the ROC curve the current environment is most likely to be?

Thank you for any insight you might have!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.