4
$\begingroup$

Here are some commonly seen statements about the importance of prevalence in the train and test sets when developing a classifier:

"Another reason not to rebalance datasets is that models should be trained on datasets whose distributions will reflect the future, real-world test cases for which they will ultimately be applied." https://towardsdatascience.com/why-balancing-classes-is-over-hyped-e382a8a410f7

"It is simply the case that a classifier trained to a 1/2 prevalence situation will not be applicable to a population with a 1/1000 prevalence." https://www.fharrell.com/post/classification/

"fans of “classifiers” sometimes subsample from observations in the most frequent outcome category (here Y=1) to get an artificial 50/50 balance of Y=0 and Y=1 when developing their classifier. Fans of such deficient notions of accuracy fail to realize that their classifier will not apply to a population when a much different prevalence of Y=1 than 0.5." https://www.fharrell.com/post/classification/*

These statements are so obvious that there is no explanation for it. But I struggle to understand, why is that the case?

In the typical situation that we are using a deterministic model + a decision boundary to perform classification (e.g. logistic regression, or a neural network), how does class prevalence in the test set really affecting the model results?

Prevalence might affect the training of a model if there are too few samples to be able to learn relevant features to distinguish a given class. Here we have the common example of a naive classifier trained on a heavily imbalanced training set measured on accuracy, where it will learn to only predict the majority class.

But once the classifier has learnt relevant features, predictions will be deterministic. It does not matter if we feed a single example in testing (prevalence = 100%), two examples of each class (prevalence = 50%) or 99:1 examples (prevalence 1%). The prediction for that one exemplar will always be the same.

If we have a very imbalanced data set (say 99:1 for two classes), I dont see why balancing the training set would introduce any problems. The training set would have an artificial prevalence of 50/50, to make sure the model learns relevant features for both classes (given there are still enough training examples for each one), and then this model can be deployed in a natural test set. If the test set has a prevalence of 99:1, the model has no awareness of it. Is this thinking wrong?

As pointed out here Different number of samples (observations) per class (one vs. all classification), a classifier is the likelihood function in a Bayesian model, and it is independent of prior probabilities (prevalence). Specifically in machine learning we don't use probabilistic models, but deterministic, where the Likelihood just gives out a point estimate for the model parameters. If we are interested in modeling a posterior probability, then we can combine the likelihood with priors, but that is a different statement and apporach that the usual approach in classification algorithms.

Are the initial statements about prevalence really applicable to deterministic models, or is there something else I am missing? Does a deterministic model really learn to mimic the prevalence of a training population?

$\endgroup$

2 Answers 2

5
$\begingroup$

"Why exactly does a classifier need the same prevalence in the train and test sets?"

Perhaps my answer to a related question on the DS SE might help

Doesn't over(/under)sampling an imbalanced dataset cause issues?

Yes, the classifier will expect the relative class frequencies in operation to be the same as those in the training set. This means that if you over-sample the minority class in the training set, the classifier is likely to over-predict that class in operational use.

To see why it is best to consider probabilistic classifiers, where the decision is based on the posterior probability of class membership p(C_i|x), but this can be written using Bayes' rule as

$p(C_i|x) = \frac{p(x|C_i)p(C_i)}{p(x)}\qquad$ where $\qquad p(x) = > \sum_j p(x|C_j)p(c_j)$,

so we can see that the decision depends on the prior probabilities of the classes, $p(C_i)$, so if the prior probabilities in the training set are different than those in operation, the operational performance of our classifier will be suboptimal, even if it is optimal for the training set conditions.

Some classifiers have a problem learning from imbalanced datasets, so one solution is to oversample the classes to ameliorate this bias in the classifier. There are to approaches. The first is to oversample by just the right amount to overcome this (usually unknown) bias and no more, but that is really difficult. The other approach is to balance the training set and then post-process the output to compensate for the difference in training set and operational priors. We take the output of the classifier trained on an oversampled dataset and multiply by the ratio of operational and training set prior probabilities,

$q_o(C_i|x) \propto p_t(x|C_i)p_t(C_i) \times \frac{p_o(C_i)}{p_t(C_i} > = p_t(x|C_i)p_o(C_i)$

Quantities with the o subscript relate to operational conditions and those wit the t subscript relate to training set conditions. I have written this as $q_o(C_i|x)$ as it is an un-normalised probability, but it is straight forward to renormalise them by dividing by the sum of $q_o(C_i|x)$ over all classes. For some problems it may be better to use cross-validation to chose the correction factor, rather than the theoretical value used here, as it depends on the bias in the classifier due to the imbalance.

So in short, for imbalanced datasets, use a probabilistic classifier and oversample (or reweight) to get a balanced dataset, in order to overcome the bias a classifier may have for imbalanced datasets. Then post-process the output of the classifier so that it doesn't over-predict the minority class in operation.

Specific issues:

If we have a very imbalanced data set (say 99:1 for two classes), I dont see why balancing the training set would introduce any problems.

It doesn't present a problem, provided you post-process the output of the model to compensate for the difference in training set and operational class frequencies. If you don't perform that adjustment (or you use a discrete yes-no classifier) you will over-predict the minority class for the reason given above.

"fans of “classifiers” sometimes subsample from observations in the most frequent outcome category (here Y=1) to get an artificial 50/50 balance of Y=0 and Y=1 when developing their classifier. Fans of such deficient notions of accuracy fail to realize that their classifier will not apply to a population when a much different prevalence of Y=1 than 0.5."

I don't think this accurately represents the situation. The reason for balancing is actually because the majority class is "more important" in some sense than the minority class, and the rebalancing is an attempt to include misclassification costs so that it does work better in operational conditions. However a lot of blogs don't explain that properly, so a lot of practitioners are rather misinformed about it.

$\endgroup$
9
  • 1
    $\begingroup$ @hirschme the decision boundary of a deterministic classifier is essentially an estimate of the contour of the probability of class membership being 0.5. The likelihood and priors do play a role, they just don't appear explicitly in the maths. $\endgroup$ May 9 at 16:00
  • 1
    $\begingroup$ The optimal decision boundary (assuming equal misclassification costs) will be where the posterior probability of class membership is 0.5. So any state-of-the art classifier is going to give a decision boundary in approximately that location, or it would give poor results. The position of the 0.5 contour depends on the priors (relative class frequencies) so if a deterministic classifier is not sensitive to those priors it won't work well in imbalanced conditions (but they do - classifying everything as belonging to the majority class is often optimal for accuracy). $\endgroup$ May 9 at 16:05
  • 1
    $\begingroup$ Try using a linear SVM to classify data from two overlapping Gaussians with equal covariance structures and then vary the proportions of positive and negative patterns. You should find that the direction of the decision boundary is fairly constant, but the position will shift towards the most common class in the training set. This demonstrates that they are sensitive to the priors/relative class frequencies. $\endgroup$ May 9 at 16:08
  • 1
    $\begingroup$ "That is the rationale for why an imbalanced training set is bad" no, quite the opposite, it shows that deterministic classifiers work just fine on imbalanced data and the only reason for resampling is that it is one way of implementing cost sensitive learning, in which case you can use a balanced training set and imbalanced test set, but you are looking at the expected loss on that test set (a weighted accuracy) not the ordinary accuracy. Personally I think it is better to weight the training criterion rather than re-balancing, but it is the same thing asymptotically. $\endgroup$ May 9 at 16:23
  • 1
    $\begingroup$ Hi, it doesn't show up explicitly in the maths, but the training criterion is usually the sum of losses for individual training patterns, so the priors affect the focus on positive and negative patterns via the number of terms they have in that summation. $\endgroup$ May 10 at 15:18
3
$\begingroup$

I would proceed with caution. The only time I would rebalance would be if I knew the characteristics of the original population, so that when I did rebalanced it would reflect the proportion of the true population. In other words treat it as a true sample, and weight it. But, arbitrarily adding or subtracting from groups may remove important data from the majority classes, and multiply error or introduce bias if adding to minority classes. If you received new data later which didn't match the class breakout of the old data, it would be a headache to have to go make and refine your analysis, since you would have made incorrect assumptions about the class breakouts.

However, having said that, it is true that some ML algorithms, like Decision Tree algorithms, output trivial results with imbalanced classes, and sometimes it is OK to balance to equal classes if your goal is understand the model rules rather than making predictions.

$\endgroup$
3
  • $\begingroup$ "you would have made incorrect assumptions about the class breakouts": What I don't understand is where and how does a model make assumptions about the class breakouts. The model would just either learn to distinguish classes or not, is this incorrect? $\endgroup$
    – hirschme
    May 9 at 15:56
  • 2
    $\begingroup$ "how does a model make assumptions about the class breakouts. " they don't make assumptions, they estimate them from the training data. So if the training data is not representative of operational conditions, it may not work very well. BTW here is a nice paper on estimating the operational priors while in operation, which I have found effectvie ieeexplore.ieee.org/abstract/document/6789744 $\endgroup$ May 9 at 16:10
  • 1
    $\begingroup$ @hirschme yes, but HOW it distinguishes between classes is important. e.g If you are eliminating data, in logistic regression there could be an outlier which is removed which might swing the decision rule to the other class. If you replicated observations to increase a count, the relationships of the variables that you added would be overweighted, but not necessarily be as important as they now appear to be $\endgroup$ May 9 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.