Choosing class-balance of training dataset for unbalanced binary classification problem

Question

There are many discussions on here about techniques for handling unbalanced datasets, eg. Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?. My question is different in that it pertains to a scenario where we have an unbalanced population but we have the ability to construct a training dataset with whatever class-balance we decide.

Assume we have a binary classification problem with fractions $P$ and $1-P$ of the population falling into the positive/negative classes. Let us say we want to create a labeled dataset of size $N$ and that we can control the fraction of positives in our labeled dataset ($P'$). In the case $P' \neq P$ we will apply some sort of correction (eg. importance sampling, optimizing threshold) to try and correct for the bias from the improperly proportioned training set in whatever way is appropriate for our task (eg. prediction/classification). In this scenario, how do we go about choosing $P'$ so as to maximize the expected efficiency of our model with respect to some metric?

As a motivating example, if $P=0.05$ and $N=20$, it seems like it would be exceedingly difficult to learn a useful model since we will only have one example of the positive class in our training dataset. I can imagine how a model trained on a dataset with $P=0.50$ and then afterward adjusted would perform better, but I'm not sure how to think about this systematically.

Edit: Here is a simulated example that demonstrates that over-representing the minority class in the training dataset can lead to greater model efficiency. In this case, we perform weighted logistic regression where the negative class is given a weight of 1.0 and the positive, minority class is given a weight of $P'/P$. When the positive class is represented in the same proportion as in the population ($P'=P=0.05%$) then this is the same as unweighted logistic regression. When the positive class is overrepresented (eg. $0.5=P'>P=0.05$) then the positive class is down-weighted ($P'/P=0.1$). The results demonstrate that higher model performance is achieved with the same number of samples when we oversample the positive class ($P' \geq 0.10$).

import random 
import pandas as pd
from sklearn.metrics import roc_auc_score
import seaborn as sns
from matplotlib import pyplot as plt
import numpy as np 
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

B0 = 4.9
B1 = np.array([.1,1,.8,.5])

X = np.random.normal(-4,1.6, [1_000_000,4])
log_odds = B0+np.sum(B1*X,axis=1)
y_prob = np.exp(log_odds)/(1+np.exp(log_odds))
y = y_prob>np.random.uniform(0,1, 1_000_000)
print(np.mean(y))

X, X_test, y, y_test = train_test_split(X, y, test_size=0.001)

true_idx = sorted(list(np.where(y)[0]))
false_idx = sorted(list(np.where(~y)[0]))

ratios = [0.05, 0.1, 0.2, 0.5]
n_repititions = 1000
N_sampels_per_experiment_list = list(np.arange(20,120,20))+list(np.arange(100,1100,100))

results_df = pd.DataFrame(columns=['N','ratio','brier','auc'])

i = 0
for N_sampels_per_experiment in N_sampels_per_experiment_list:
    for ratio in ratios:
        for repitition in range(n_repititions):
            idx_samp_true = random.sample(true_idx, int(ratio*N_sampels_per_experiment))
            idx_samp_false = random.sample(false_idx, int((1-ratio)*N_sampels_per_experiment))
            idx_samp = idx_samp_true+idx_samp_false
            y_samp = y[idx_samp].ravel()
            x_samp = X[idx_samp,:]

            m = LogisticRegression(class_weight={0:1, 1:(0.05/ratio)})
            m.fit(x_samp, y_samp)
            
            y_test_pred_prob = m.predict_proba(X_test)[:,1]
            y_test_pred = y_test_pred_prob>0.5
            
            results_df.loc[i,'N'] = N_sampels_per_experiment
            results_df.loc[i,'ratio'] = ratio
            results_df.loc[i,'brier'] = np.mean((y_test-y_test_pred_prob)**2)
            i+=1

# Plot results
plt.figure(figsize=(10,6))
sns.lineplot(data=results_df, x='N',y='brier',hue='ratio',palette='tab10')
plt.xlim(0,1000)
plt.ylabel('Brier Score')
plt.xlabel('Training Set Size')
plt.title('Efficiency of Weighted Logisitic Regression \n Varying Training Class Balance')
plt.legend(title='Fraction positives \n in training set')
plt.show()

Answer 1

Say you know the outcomes for all observations but have a limit on the number of how many, $N$, that you can evaluate in your model. That might be because of the expense of collecting data on some predictor variables, for example an expensive analytical test. Then what you describe are the issues faced by a case-control-study. See this page. Your observations with the rare outcome would be considered "cases," and you choose a number of observations without that outcome, subject to the economic or other constraint.

This page describes some tradeoffs in case/control ratios. If a fixed $N$ is the only constraint, then your sense is correct that equal numbers in each group might be optimal in terms of predictor coefficients. But that might not be the only constraint, as discussed in the answer from @MaxGordon. You will still be faced with the difficulty of correcting back to the population prevalence of the outcome.

As the answer from @Fomite on that page says:

There isn't necessarily an optimal case-control study ratio, otherwise it would be the one we all used. Generally, [it] is argued that a higher ratio of controls to cases results in greater study power, though at the cost of a more expensive study.

Absent such constraints, use all the data you have.