I was wondering if anyone could help me understand resampling for class imbalance. From what I have learned, class imbalance is usually a small data problem where the less prevalent class usually cannot be observed enough to really help inform a model to separate it. If the number of observations were sufficiently large such that the less prevalent class had sufficient density, I imagined that any of these resampling methods wouldn't help too much. However, many blog posts and fellow Data Scientists insist to resample.
If anything, I am concerned that resampling the minority class will bias the model too much toward the observations. In an extreme example, you can imagine having just one point that is a minority class. This one point is unlikely to represent the whole space that can be the minority class, but if you were to upsample it 1000x let's say, the model may mistakenly have too much confidence in this specific area (and also not others?).
So here is my python code on resampling. I did it pretty roughly so any comments to make it more genuine for the theoretical problem is appreciated. I didn't run the full gamut of all resampling methods because I think they are all somewhat similar in the application of the theory. I am using the AUC of the ROC as a gauge of separation. Upsampling the minority class does not seem to help much except in the case of very low noise volatility (noise_var == 1 for example) where the space occupied by the observed minority class can seem more reliable to set strong thresholds.
import numpy as np
from sklearn.linear_model import LogisticRegressionCV
from sklearn.metrics import roc_auc_score, roc_curve
from sklearn.utils import resample
import matplotlib.pyplot as plt
# training
n = 1000
x1 = np.random.gamma(1, size=n)
x2 = np.random.uniform(0,100, n)
x3 = np.random.normal(0,3, n)
noise_var = 100
y = 2 + .025*x1 - 5*x2 + 6*x3
y_with_noise = y + np.random.normal(0,noise_var,n)
print('Variance ratios:',np.var(y),np.var(y_with_noise),np.var(y_with_noise)/ np.var(y))
p = np.exp(y)/(1+np.exp(y))
obs = np.random.binomial(1, p, n)
X = np.column_stack((np.ones(n),x1,x2,x3))
model1 = LogisticRegressionCV().fit(X,obs)
print('Observed Training Targets Original:', sum(obs)/len(obs))
# train with upsampling
upsample = obs == 1
upsample_X = X[upsample]
newX = resample(upsample_X, n_samples=sum(obs!=1))
X = np.concatenate((X[obs!=1],newX))
obs = np.concatenate((obs[obs!=1],np.ones(len(newX))))
model2 = LogisticRegressionCV().fit(X,obs)
print('Observed Training Targets Resampled:', sum(obs)/len(obs))
# prediction
n = 1000000
x1 = np.random.gamma(1, size=n)
x2 = np.random.uniform(0,100, n)
x3 = np.random.normal(0,3, n)
y = 2 + .025*x1 - 5*x2 + 6*x3 + np.random.normal(0,noise_var,n)
p = np.exp(y)/(1+np.exp(y))
obs = np.random.binomial(1, p, n)
print('Observed Holdout Targets:',sum(obs), sum(obs)/1e6)
X = np.column_stack((np.ones(n),x1,x2,x3))
prediction = model1.predict(X)
prediction_prob = model1.predict_proba(X)[:,1]
fpr1, tpr1, _ = roc_curve(obs, prediction_prob)
print('ROC AUC plain:',roc_auc_score(obs, prediction_prob))
prediction = model2.predict(X)
prediction_prob = model2.predict_proba(X)[:,1]
fpr2, tpr2, _ = roc_curve(obs, prediction_prob)
print('ROC AUC resampled:',roc_auc_score(obs, prediction_prob))
plt.plot(fpr1, tpr1, ':', label='original')
plt.plot(fpr2, tpr2, '--', label='resample')
plt.legend()
