I've been thinking a lot about the "class imbalance problem" in machine/statistical learning lately, and am drawing ever deeper into a feeling that I just don't understand what is going on.
First let me define (or attempt to) define my terms:
The class imbalance problem in machine/statistical learning is the observation that some binary classification(*) algorithms do not perform well when the proportion of 0 classes to 1 classes is very skewed.
So, in the above, for example, if there were one-hundred $0$ classes for every single $1$ class, I would say the class imbalance is $1$ to $100$, or $1\%$.
Most statements of the problem I have seen lack what I would think of as sufficient qualification (what models struggle, how imbalanced is a problem), and this is one source of my confusion.
A survey of the standard texts in machine/statistical learning turns up little:
- Elements of Statistical Leaning and Introduction to Statistical Learning do not contain "class imbalance" in the index.
Machine Learning for Predictive Data Analytics also does not contain"class imbalance" in the index.
Murphy's Machine Learning: A Probabilistic Perspective does contain "class imbalance* in the index. The reference is to a section on SVM's, where I found the following tantalizing comment:
It is worth remembering that all these difficulties, and the plethora of heuristics that have been proposed to fix them, fundamentally arise because SVM's do not model uncertainty using probabilities, so their output scores are not comparable across classes.
This comment does jive with my intuition and experience: at my previous job we would routinely fit logistic regressions and gradient boosted tree models (to minimize binomial log-likelihood) to unbalanced data (on the order of a $1\%$ class imbalance), with no obvious issues in performance.
I have read (somewhere) that classification tree based models (trees themselves and random forest) do also suffer from the class imbalance problem. This muddies the waters a little bit, trees do, in some sense, return probabilities: the voting record for the target class in each terminal node of the tree.
So, to wrap up, what I'm really after is a conceptual understanding of the forces that lead to the class imbalance problem (if it exists).
- Is it something we do to ourselves with badly chosen algorithms and lazy default classification thresholds?
- Does it vanish if we always fit probability models that optimize proper scoring criteria? Said differently, is the cause simply a poor choice of loss function, i.e. evaluating the predictive power of a model based on hard classification rules and overall accuracy?
- If so, are models that do not optimize proper scoring rules then useless (or at least less useful)?
(*) By classification I mean any statistical model fit to binary response data. I am not assuming that my goal is a hard assignment to one class or the other, though it may be.