2
$\begingroup$

Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods, and this is the case of $p >> n$, here $p$ is the number of possible variables.

I'm trying to develop a classification model, where the desired outcome is the probability estimate, rather than only the output class.

However, for various variable/model combinations that I've tried, the resulting models are able to distinguish the bad cases quite nicely, but not the good cases. In other words, the estimated distribution function of (smoothed) empirical outcomes vs the model estimated probabilities is not monotone, roughly having the shape of a tilted parabola (like letter $J$). I hope the problem is clear, if not - please comment, I'll gladly clarify.

My question: What are the common strategies in order to reshape any model to focus the estimation on the good cases? If it is possible at all?

Intuitively, it seems that a possible strategy would be to define a custom metric for the minimizer, that would have asymmetric weights for good vs bad cases. I.e., in the case of penalized linear regression, the variable selection could then be skewed towards distinguishing goods instead of maximizing total AUC. But so far I'm unable to find any realized similar solutions. I guess, implementing something like that is not a trivial task.

Alternatively, is it possible to achieve this by transforming the input variables in a certain way?

Any hints or suggestions would be really useful!

$\endgroup$

1 Answer 1

0
$\begingroup$

It depends.

I usually set some cut off level which maximize / minimize some objective function derived from the business goals. Not so much any separate metric based on statistics or information theory.

Small amount of observations in minority class could be a problem and then you might have to need to oversample that minority class.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.