0
$\begingroup$

Context: Have been trying to create a prediction model for a 1% outcome variable using Random Forest Machine Learning for a large health survey (entirely multi-level categorical data, yes/no outcome, ~230,000 observations initially). Stupidly high amount of missing values, poor data collection and the need to infer between variable. Multiple imputation used, CCA used (as a separate base), k-fold CV, train/test split, the lot. Whilst under-sampling improved things marginally, over-sampling was pretty much consistent with no sampling at all (~140,000 observations now, with perfect sensitivity and no specificity reached - i.e., unhelpful). Whilst Random forest can handle this type of data well, I understand it has its limitations (especially with class imbalance and biasness towards certain predictors).

ANYWAY:

I've read a bit of ML research, and I don't claim to understand all of it, but there's sometimes a pretty solid paper on improper methods for ML. Presently, I understand multiple logistic regression SHOULD be able to handle the class imbalance - but in my case, I still get nearly perfective sensitivity and 0 specificity.

Does this then justify the use of a balancing method in that the imbalance is now a problem, or am I missing something? Have just been using the Caret train package for most training needs.

Improve this question
$\endgroup$
2
  • $\begingroup$ perfect sensitivity and zero specificity (assigning all patterns to the majority class) may well be the optimal solution for the learning task as posed, see stats.stackexchange.com/questions/539638/… . If this is not acceptable, it means that false negatives are worse errors than false positives, and the solution is cost-sensitive learning (and the level of imbalance is irrelevant to the implementation of that). $\endgroup$
    – Dikran Marsupial
    Jan 14 at 6:59
  • 2
    $\begingroup$ When you write about sensitivity and specificity, you are obviously thinking in terms of "hard" classifications, which unfortunately most "ML" methods output by default. Logistic regressions give you probabilistic classifications, which I would argue are much more suitable. To go from probabilistic to hard classifications, you need a threshold, and the trivial hardcoded one of 0.5 is usually not useful at all, especially for "unbalanced" data. See here for more information. $\endgroup$
    – Stephan Kolassa
    Jan 14 at 7:09

1 Answer 1

Reset to default
3
$\begingroup$

No, it doesn't. Think of it this way - I don't even need a model, or any predictors, to get a $99\%$ sensitivity. That's great! I can do this because the prior odds are $99-1$ in favor of one of the classes; call it $A$. To predict the other class ($B$), you'd need your data+model to be strong enough to indicate $> 99-1$ odds in favor of $B$. That's pretty strong. And if you don't have a collection of features that, together, give you those odds or better, then the odds are that $A$ is the correct class. Pretending otherwise (by, for example, oversampling) will likely hurt you when it comes to generalizing to out-of-sample data.

One situation in which this isn't the case is if the losses of misclassification are asymmetric. If misclassifying a $B$ is $10\times$ as expensive as misclassifying an $A$, you'd be better off predicting a $B$ if the odds are better than $10-1$ against. However, you don't need oversampling to do this - you can look at the probabilities, and, instead of using the default $> 50\%$ rule for classifying, use, say, a $> 10\%$ rule.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.