0
$\begingroup$

if you have a confusion matrix :

             actual

           0       1

      0   2658    204

      1     24    110

y-axis = predicted x-axis = actual

You have 90% weight on specificity (i.e. 2658+24/(2658+24+204+110)

and 10% weight on sensitivity (i.e. 204+110 /(2658+24+204+110)

The primary reason why you use that calculation is to know if you have to oversample the minority class so that you can balance out the imbalance between the minority and majority class?

$\endgroup$
2

1 Answer 1

2
$\begingroup$

Kind of but not really.

First, you'd normally talk about weights on decisions; specifically, weights on different types of errors.

Second, you ideally should separate the prior probability or prevalence (the fraction of observations that are truly positive) from the weight or loss applied to each type of error. You can have balanced training data (prevalence=0.5) but decide that false positives are (say) much less important than false negatives and adjust the loss function to give more weight to false negatives. Or you could have imbalanced training data but decide that false positives and negatives are about equally important. These are separate concepts.

After getting past these we get to a fourth point. If you have unbalanced data but you want to have approximately equal numbers of false positives and false negatives, one way to do this is to oversample the minority class. Making this decision is one reason to compute the prevalence. Oversampling the minority class for this reason is in some ways just a hack -- you can also get approximately equal numbers of false positives and false negatives by adjusting the loss function. These two approaches are exactly equivalent for logistic regression. They are often pretty close for other classifiers; this paper by Leo Breiman and co-workers compares them for random forests.

And, finally, a fourth point. If you have unbalanced data and don't adjust the loss function or oversample, your classifier could end up predicting that everything most likely goes in the majority class. That's because everything mostly likely goes in the majority class, and it is often a feature, not a bug. Medical students have to be taught "when you hear hooves, think horses, not zebras", because rare diseases are over-represented in medical training relative to medical practice. Sometimes you want your classifier to think zebras, but often you want it to think horses.

$\endgroup$
6
  • $\begingroup$ I like that about horses/zebras: it predicts the majority class most of the time because it is quite likely to get it right by doing so! (And it looks like there’s another Scrubs fan on here, too!) $\endgroup$
    – Dave
    Commented Mar 27, 2023 at 1:56
  • $\begingroup$ I think 90% weight on specificity is I can have 90% confidence, which is strong, in specificity. But, for sensitivity, since there is 10% weight on sensitivity, it's not as reliable as specificity. Is this one way to looking at x% weight on specificity? $\endgroup$
    – shawn
    Commented Mar 27, 2023 at 2:02
  • $\begingroup$ No, that doesn't follow at all. $\endgroup$ Commented Mar 27, 2023 at 2:09
  • $\begingroup$ Then, can you please elaborate your #2 point? thanks $\endgroup$
    – shawn
    Commented Mar 27, 2023 at 2:12
  • $\begingroup$ If your data are bad enough, you can't have 90% confidence in anything, no matter what your class balance is or your loss function or your algorithm. The point of weights in the loss is to prioritise the use of whatever information you do have; it's a separate question how much information that is $\endgroup$ Commented Mar 27, 2023 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.