I've been thinking a lot about the "class imbalance problem" in machine/statistical learning lately, and am drawing ever deeper into a feeling that I just don't understand what is going on.

First let me define (or attempt to) define my terms:

The class imbalance problem in machine/statistical learning is the observation that some binary classification(*) algorithms do not perform well when the proportion of 0 classes to 1 classes is very skewed.

So, in the above, for example, if there were one-hundred $0$ classes for every single $1$ class, I would say the class imbalance is $1$ to $100$, or $1\%$.

Most statements of the problem I have seen lack what I would think of as sufficient qualification (what models struggle, how imbalanced is a problem), and this is one source of my confusion.

A survey of the standard texts in machine/statistical learning turns up little:

  • Elements of Statistical Leaning and Introduction to Statistical Learning do not contain "class imbalance" in the index.
  • Machine Learning for Predictive Data Analytics also does not contain"class imbalance" in the index.

  • Murphy's Machine Learning: A Probabilistic Perspective does contain "class imbalance* in the index. The reference is to a section on SVM's, where I found the following tantalizing comment:

    It is worth remembering that all these difficulties, and the plethora of heuristics that have been proposed to fix them, fundamentally arise because SVM's do not model uncertainty using probabilities, so their output scores are not comparable across classes.

This comment does jive with my intuition and experience: at my previous job we would routinely fit logistic regressions and gradient boosted tree models (to minimize binomial log-likelihood) to unbalanced data (on the order of a $1\%$ class imbalance), with no obvious issues in performance.

I have read (somewhere) that classification tree based models (trees themselves and random forest) do also suffer from the class imbalance problem. This muddies the waters a little bit, trees do, in some sense, return probabilities: the voting record for the target class in each terminal node of the tree.

So, to wrap up, what I'm really after is a conceptual understanding of the forces that lead to the class imbalance problem (if it exists).

  • Is it something we do to ourselves with badly chosen algorithms and lazy default classification thresholds?
  • Does it vanish if we always fit probability models that optimize proper scoring criteria? Said differently, is the cause simply a poor choice of loss function, i.e. evaluating the predictive power of a model based on hard classification rules and overall accuracy?
  • If so, are models that do not optimize proper scoring rules then useless (or at least less useful)?

(*) By classification I mean any statistical model fit to binary response data. I am not assuming that my goal is a hard assignment to one class or the other, though it may be.

  • 1
    $\begingroup$ An obvious problem might arise when the learner penalizes each classes' loss the same. Returning everything the same class could, theoretically, minimize total loss. $\endgroup$ – Firebug Nov 25 '16 at 19:45
  • 1
    $\begingroup$ I forgot to add poor choice of loss function in my list. So, do you think this is true even for proper scoring rules as loss functions? $\endgroup$ – Matthew Drury Nov 25 '16 at 22:22
  • $\begingroup$ I think so. I guess we can formulate a problem where minimizing the loss of the bigger class only minimizes the loss of the whole problem as well, while in general the minority class is of bigger interest. $\endgroup$ – Firebug Nov 27 '16 at 12:37
  • 3
    $\begingroup$ I agree with the sentiments of the question. I've had a working hypothesis (though happy to reject it) that there is no class imbalance problem per se, just that we train with loss functions that don't represent what we will use to measure success on test data. And it's hard to call this a mistake, as it's almost standard practice: e.g. it's not standard to directly optimize AUC or F1 score, but those are common success metrics for problems with class imbalance. So maybe that's the class imbalance problem? $\endgroup$ – DavidR Dec 13 '16 at 14:55
  • 8
    $\begingroup$ The cause of the class imbalance problem is the convention to use accuracy as a loss function. class imbalance is a problem characteristic (rare disease diagnostic for example), that can be dealt with using several strategies. Using a class weight inverse proportional to the class size when computing the loss function is one of them. Other than that, AUC as a loss function is a good idea since it specifically distinguished between true-positive and false-positive. Therefore the core issue of the class imbalance problem is the loss function. Great question though, which I don't dare to answer. $\endgroup$ – Nikolas Rieble Jan 12 '17 at 15:19

Anything that involves optimization to minimize a loss function will, if sufficiently convex, give a solution that is a global minimum of that loss function. I say 'sufficiently convex' since deep networks are not on the whole convex, but give reasonable minimums in practice, with careful choices of learning rate etc.

Therefore, the behavior of such models is defined by whatever we put in the loss function.

Imagine that we have a model, $F$, that assigns some arbitrary real scalar to each example, such that more negative values tend to indicate class A, and more positive numbers tend to indicate class B.

$$y_f = f(\mathbf{x})$$

We use $F$ to create model $G$, which assigns a threshold, $b$, to the output of $F$, implicitly or explicitly, such that when $F$ outputs a value greater than $b$ then model $G$ predicts class B, else it predicts class A.

$$ y_g = \begin{cases} B & \text{if } f(\mathbf{x}) > b \\ A & \text{otherwise}\\ \end{cases} $$

By varying the threshold $b$ that model $G$ learns, we can vary the proportion of examples that are classified as class A or class B. We can move along a curve of precision/recall, for each class. A higher threshold gives lower recall for class B, but probably higher precision.

Imagine that the model $F$ is such that if we choose a threshold that gives equal precision and recall to either class, then the accuracy of model G is 90%, for either class (by symmetry). So, given a training example, $G$ would get the example right 90% of the time, no matter what is the ground truth, A or B. This is presumably where we want to get to? Let's call this our 'ideal threshold', or 'ideal model G', or perhaps $G^*$.

Now, let's say we have a loss function which is:

$$ \mathcal{L} = \frac{1}{N}\sum_{n=1}^N I_{y_i \ne g(x_i)} $$

where $I_c$ is an indicator variable that is $1$ when $c$ is true, else $0$, $y_i$ is the true class for example $i$, and $g(x_i)$ is the predicted class for example $i$, by model G.

Imagine that we have a dataset that has 100 times as many training examples of class A than class B. And then we feed examples through. For every 99 examples of A, we expect to get $99*0.9 = 89.1$ examples correct, and $99*0.1=9.9$ examples incorrect. Similarly, for every 1 example of B, we expect to get $1 * 0.9=0.9$ examples correct, and $1 * 0.1=0.1$ examples incorrect. The expected loss will be:

$ \mathcal{L} = (9.9 + 0.1)/100 = 0.1 $

Now, lets look at a model $G$ where the threshold is set such that class A is systematically chosen. Now, for every 99 examples of A, all 99 will be correct. Zero loss. But each example of B will be systematically not chosen, giving a loss of $1/100$, so the expected loss over the training set will be:

$ \mathcal{L} = 0.01 $

Ten times lower than the loss when setting the threshold such as to assign equal recall and precision to each class.

Therefore, the loss function will drive model $G$ to choose a threshold which chooses A with higher probability than class B, driving up the recall for class A, but lowering that for class B. The resulting model no longer matches what we might hope, no longer matches our ideal model $G^*$.

To correct the model, we'd need to for example modify the loss function such that getting B wrong costs a lot more than getting A wrong. Then this will modify the loss function to have a minimum closer to the earlier ideal model $G^*$, which assigned equal precision/recall to each class.

Alternatively, we can modify the dataset by cloning every B example 99 times, which will also cause the loss function to no longer have a minimum at a position different from our earlier ideal threshold.

  • 4
    $\begingroup$ Can you please try to make your answer a bit more particular to the questions being asked? While clearly thoughtful it reads mostly as commentary rather than an answer. For example, just for commentary purposes one could argue that using an improper scoring rule like the loss function defined is fundamentally wrong and therefore the subsequent analysis is invalid. $\endgroup$ – usεr11852 Mar 19 '17 at 0:29
  • $\begingroup$ I dont think one can say that the loss function is 'right' or 'wrong' without knowing the actual purpose of the model. If the goal is for the machine learning model to 'look cool/useful', then the $G^*$ model is better, but if it's to maximize eg scoring on some test/exam, where 99 of the questions have answer A, and one has answer B, and we only have a 90% chance of predicting the answer correctly, we're better off just choosing A for everything, and that's what the loss function above does. $\endgroup$ – Hugh Perkins Mar 19 '17 at 10:36
  • 2
    $\begingroup$ I generally agree; I am not fully convinced about the proper scoring rule necessity but on the other hand the "actual purpose" of any classification model is the useful prediction of class membership, ie. you need an informed utility function. I would argue that generally for imbalanced problems assigning cost/gain to FP, TP, etc. is probably the best way to have a reasonable utility function; in the absence of relevant domain knowledge this can be hairy. I almost always use as my first choice Cohen's $k$, a somewhat conservative metric of "agreement", because of that reason. $\endgroup$ – usεr11852 Mar 19 '17 at 12:03
  • $\begingroup$ I googled for 'utility function', but nothing came up. Do you have a link/reference? I think from the context, what you are calling a 'utility function' is essentially the model $F$ above? Model $F$ is invariant across the various scenarios. One interesting question perhaps is, if one trains model $G$ directly, using unbalanced data, will the underlying, possibly implicit, model $F$ be similar/identical to a model $F$ trained, via training model $G$, on balanced data? $\endgroup$ – Hugh Perkins Mar 19 '17 at 14:29
  • 3
    $\begingroup$ This presumes implicitly (1) that the KPI we attempt to maximize is accuracy, and (2) that accuracy is an appropriate KPI for classification model evaluation. It isn't. $\endgroup$ – Stephan Kolassa Jul 17 '18 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.