# Choosing class-balance of training dataset for unbalanced binary classification problem

There are many discussions on here about techniques for handling unbalanced datasets, eg. Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?. My question is different in that it pertains to a scenario where we have an unbalanced population but we have the ability to construct a training dataset with whatever class-balance we decide.

Assume we have a binary classification problem with fractions $$P$$ and $$1-P$$ of the population falling into the positive/negative classes. Let us say we want to create a labeled dataset of size $$N$$ and that we can control the fraction of positives in our labeled dataset ($$P'$$). In the case $$P' \neq P$$ we will apply some sort of correction (eg. importance sampling, optimizing threshold) to try and correct for the bias from the improperly proportioned training set in whatever way is appropriate for our task (eg. prediction/classification). In this scenario, how do we go about choosing $$P'$$ so as to maximize the expected efficiency of our model with respect to some metric?

As a motivating example, if $$P=0.05$$ and $$N=20$$, it seems like it would be exceedingly difficult to learn a useful model since we will only have one example of the positive class in our training dataset. I can imagine how a model trained on a dataset with $$P=0.50$$ and then afterward adjusted would perform better, but I'm not sure how to think about this systematically.

Edit: Here is a simulated example that demonstrates that over-representing the minority class in the training dataset can lead to greater model efficiency. In this case, we perform weighted logistic regression where the negative class is given a weight of 1.0 and the positive, minority class is given a weight of $$P'/P$$. When the positive class is represented in the same proportion as in the population ($$P'=P=0.05%$$) then this is the same as unweighted logistic regression. When the positive class is overrepresented (eg. $$0.5=P'>P=0.05$$) then the positive class is down-weighted ($$P'/P=0.1$$). The results demonstrate that higher model performance is achieved with the same number of samples when we oversample the positive class ($$P' \geq 0.10$$).

import random
import pandas as pd
from sklearn.metrics import roc_auc_score
import seaborn as sns
from matplotlib import pyplot as plt
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

B0 = 4.9
B1 = np.array([.1,1,.8,.5])

X = np.random.normal(-4,1.6, [1_000_000,4])
log_odds = B0+np.sum(B1*X,axis=1)
y_prob = np.exp(log_odds)/(1+np.exp(log_odds))
y = y_prob>np.random.uniform(0,1, 1_000_000)
print(np.mean(y))

X, X_test, y, y_test = train_test_split(X, y, test_size=0.001)

true_idx = sorted(list(np.where(y)[0]))
false_idx = sorted(list(np.where(~y)[0]))

ratios = [0.05, 0.1, 0.2, 0.5]
n_repititions = 1000
N_sampels_per_experiment_list = list(np.arange(20,120,20))+list(np.arange(100,1100,100))

results_df = pd.DataFrame(columns=['N','ratio','brier','auc'])

i = 0
for N_sampels_per_experiment in N_sampels_per_experiment_list:
for ratio in ratios:
for repitition in range(n_repititions):
idx_samp_true = random.sample(true_idx, int(ratio*N_sampels_per_experiment))
idx_samp_false = random.sample(false_idx, int((1-ratio)*N_sampels_per_experiment))
idx_samp = idx_samp_true+idx_samp_false
y_samp = y[idx_samp].ravel()
x_samp = X[idx_samp,:]

m = LogisticRegression(class_weight={0:1, 1:(0.05/ratio)})
m.fit(x_samp, y_samp)

y_test_pred_prob = m.predict_proba(X_test)[:,1]
y_test_pred = y_test_pred_prob>0.5

results_df.loc[i,'N'] = N_sampels_per_experiment
results_df.loc[i,'ratio'] = ratio
results_df.loc[i,'brier'] = np.mean((y_test-y_test_pred_prob)**2)
i+=1

# Plot results
plt.figure(figsize=(10,6))
sns.lineplot(data=results_df, x='N',y='brier',hue='ratio',palette='tab10')
plt.xlim(0,1000)
plt.ylabel('Brier Score')
plt.xlabel('Training Set Size')
plt.title('Efficiency of Weighted Logisitic Regression \n Varying Training Class Balance')
plt.legend(title='Fraction positives \n in training set')
plt.show()


• Per the thread you link, I suspect the main difference for a model trained on selected rather than randomly sampled data will be that it will wildly overestimate the prevalence of the minority class, and/or be far too confident about its estimate and predictions, since it believes it has leveraged much more information than there is in the wild. But I am aware that others think differently. Commented Jul 2, 2022 at 13:47
• @StephanKolassa I've updated my question with the results of a simulation that I think contradicts your suspicion. A key difference between this case and the one linked is that we are not just resampling from a constant dataset to change the proportion of the minority class. Commented Jul 4, 2022 at 13:58

Say you know the outcomes for all observations but have a limit on the number of how many, $$N$$, that you can evaluate in your model. That might be because of the expense of collecting data on some predictor variables, for example an expensive analytical test. Then what you describe are the issues faced by a case-control-study. See this page. Your observations with the rare outcome would be considered "cases," and you choose a number of observations without that outcome, subject to the economic or other constraint.
This page describes some tradeoffs in case/control ratios. If a fixed $$N$$ is the only constraint, then your sense is correct that equal numbers in each group might be optimal in terms of predictor coefficients. But that might not be the only constraint, as discussed in the answer from @MaxGordon. You will still be faced with the difficulty of correcting back to the population prevalence of the outcome.