Towards answering my three questions I offer a perspective based on analysis of a toy problem. While it is meant to illuminate a topic I find to be widely discussed but poorly understood, I do not claim my perspective provides uniformly comprehensive, authoritative conclusions. And the answer is geared towards traditional, simple logistic regression models. It should be less relevant to highly flexible ML models where misspecification is less of an issue.
Class reweighting can affect the decision boundary of logistic regression, especially when the model is misspecified (the data doesn't follow the pattern specified by the model). Upweighting one class pushes the model to align its decision boundary with the pattern of that class rather than the other class. As a result, class reweighting can be a good idea if you can't see how to fix the model misspecification and you need high recall on one class. However, the best weighting is use case-dependent, and is sometimes demonstrably not 50-50.
In applications, it's very common that a classifier's purpose is to predict rare but important events. Upweighting the minority class can make the model more fit-to-purpose if accuracy in the high-recall regime is the goal. Upweighting the minority class has the side effect of pushing the weighted class balance towards 50-50. It often increases AUC as well, since AUC is optimized at a 50-50 training balance in some models. (This has been proven for linear discriminant analysis in the infinite sample limit, and LDA is very similar to logistic regression.) However, neither class balance nor maximal AUC necessarily provide the best model for a given use case.
While the weighting that yields best performance is use case-dependent, and the optimal class balance is not necessarily 50-50, balancing might be a decent rule of thumb, especially when the model's desired operating characteristics are not known in advance. In this case, good average performance over a range of thresholds could be desirable, AUC is a metric for exactly this, and 50-50 balance often seems to provide better AUC, both theoretically and empirically.
That said, there is no general guarantee that class balancing maximizes AUC, and definitely no guarantee that it produces the best model for a given use case. Therefore, framing class imbalance as a "problem" that should be "fixed" is misleading and can lead to suboptimal modeling choices.
Class reweighting affects the decision boundary
Let's fire up R and fit a very simple misspecified model. Our data will have two classes, A and B. Class A will be on the x-axis and B on the y-axis.
It's possible to separate these data perfectly (except at the origin), but let's say we don't know that, and we only fit the most obvious logistic regression model with untransformed x and y as predictors. The decision boundary of this model is
Now, let's upweight one class by 10x. This time, the decision boundary aligns more closely with the data on the y-axis.
What's going on here?
From the perspective of the model, the data contains two conflicting patterns. With class A on the x-axis and class B on the y-axis, the classes are not cleanly separated by a single line, as the logistic regression model assumes. In other words, the model is misspecified. The fit must decide how to split the difference.
When class A and B have equal weights, the decision boundary is set halfway between the A pattern and the B pattern. When B is upweighted, it's pushed towards the B pattern. Weighting does not just change the intercept, but also aligns the decision boundary more with the distribution of class B.
In retrospect, this makes sense. When errors on one class become much more costly, the model fitting process will prefer a model which is highly accurate on that class by keeping it to one side of the decision boundary as much as possible. Changing the slope of the boundary accomplishes this more effectively than just changing the intercept. Similar phenomena should also occur in more complex problems with more predictors and nonlinear decision boundaries.
What does this suggest about class balancing?
Models are often misspecified in real-world applications, especially when linear-decision-boundary models are used, because the data rarely conform to simple linear formulas. However, misspecified models can still be useful, and we can use weighting to make the model more fit for purpose. In this case, if the purpose of the model is to find all possible elements of class B, and some false positives from class A are tolerable, the model weighted 10x towards class B is probably preferable. It has the same recall, and better precision, than simply shifting the threshold of the unweighted model. (This can be seen by thinking geometrically: if you move the y=x line down until all of B is on one side, more A points are on the wrong side than in the weighted model.) This is essentially importance weighting, as explained here.
This means 50-50 weighting is not ideal in this use case. And it doesn't matter if it maximizes AUC. It may, but that would just illustrate how maximizing AUC doesn't always get us what we really want.
Reweighting for class balance: a satisficing choice?
While reweighting for class balance doesn't always get the best result, it could perhaps be "good enough" in many applications. Very commonly, classifiers are designed to pick out a rare but important event. Upweighting the rare event may improve the fitness of the model for this purpose, and balancing classes can improve AUC, which can be a decent rough proxy for what is really desired.
In particular, when the desired operating characteristics of the model are not known in advance, maximizing AUC is arguably a good satisficing choice, because AUC represents average performance over a range of thresholds, and good average performance makes better performance around the operating threshold somewhat more likely.
On the other hand, a bit of thought about the use case often reveals a lot about the desired performance characteristics, and could yield better final results than spending 100% of effort towards maximizing AUC.
R Code
library(tidyverse)
ggplot2::theme_set(theme_minimal())
vals <- seq(-1, 3, by = 1/10)
n <- length(vals)
ones <- 1 + numeric(n)
df <- tibble(
x = c(1 * vals, 0 * vals),
y = c(0 * vals, 1 * vals),
class = c(0 * ones, 1 * ones)) %>%
mutate(class = factor(class, c(0, 1), labels = c('A', 'B')))
# plot data
df %>%
ggplot(aes(x = x, y = y, group = class, color = class)) +
geom_point(size = 3) +
ggtitle('Training data')
# Model 1: equal weights on classes ---------------
model <- glm(class ~ x + y, family = 'binomial', data = df)
beta_0 <- coef(model)['(Intercept)']
beta_x <- coef(model)['x']
beta_y <- coef(model)['y']
# plot data + decision boundary
# 0 = beta_0 + beta_x * x + beta_y * y
# y = (-beta_x / beta_y) * x + (-beta_0 / beta_y)
df %>%
ggplot(aes(x = x, y = y, group = class, color = class)) +
geom_point(size = 3) +
geom_abline(slope = -beta_x/beta_y,
intercept = -beta_0/beta_y,
color = 'black', linetype = 'dashed') +
ggtitle('Unweighted model')
# Model 2: upweight class B by 10x -----------------
df <- df %>% mutate(w = if_else(class == 'B', 10, 1))
model <- glm(class ~ x + y, family = 'binomial', data = df, weights = w)
beta_0 <- coef(model)['(Intercept)']
beta_x <- coef(model)['x']
beta_y <- coef(model)['y']
# plot data + decision boundary
df %>%
ggplot(aes(x = x, y = y, group = class, color = class)) +
geom_point(size = 3) +
geom_abline(slope = -beta_x/beta_y,
intercept = -beta_0/beta_y,
color = 'black', linetype = 'dashed') +
ggtitle('Fit with 10x weight on class B')
