126
$\begingroup$

Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.

My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate).

But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?

Improve this question
$\endgroup$
5
  • 7
    $\begingroup$ It is hard to imagine how this could be. Perhaps you are cutting the predicted probability at 0.5? If so, try varying the cutoff. $\endgroup$
    – Aniko
    Commented Jan 7, 2011 at 20:30
  • 5
    $\begingroup$ The area under the ROC-curve is .585, a rather poor result. This implies that there really isn't a cutoff value where the specificity/sensitivity trade-off is worth it. Fiddling with the cutoff won't improve classifications much, as it would just decrease the specificity by roughly as much as it increases the sensitivity. $\endgroup$
    – Michiel
    Commented Jan 11, 2011 at 7:52
  • 3
    $\begingroup$ Are any of the Variables coefficients significantly different from $0$ (say more than five standard errors)? If not your problem Could be that you just don't have much explanatory power with your set of variables. $\endgroup$
    – probabilityislogic
    Commented Jan 21, 2013 at 11:20
  • 3
    $\begingroup$ Note also that your sample size in terms of making good predictions is really the number of unique patterns in the predictor variable, and not the number of sampled individuals. For example, a model with a single categorical predictor variable with two levels can only fit a logistic regression model with two parameters (one for each category), even if there are millions people in the sample. $\endgroup$
    – probabilityislogic
    Commented Jan 21, 2013 at 11:32
  • $\begingroup$ Related: stats.stackexchange.com/questions/67903 $\endgroup$
    – amoeba
    Commented Nov 6, 2018 at 21:33

3 Answers 3

Reset to default
101
$\begingroup$

Balance in the Training Set

For logistic regression models unbalanced training data affects only the estimate of the model intercept (although this of course skews all the predicted probabilities, which in turn compromises your predictions). Fortunately the intercept correction is straightforward: Provided you know, or can guess, the true proportion of 0s and 1s and know the proportions in the training set you can apply a rare events correction to the intercept. Details are in King and Zeng (2001) [PDF].

These 'rare event corrections' were designed for case control research designs, mostly used in epidemiology, that select cases by choosing a fixed, usually balanced number of 0 cases and 1 cases, and then need to correct for the resulting sample selection bias. Indeed, you might train your classifier the same way. Pick a nice balanced sample and then correct the intercept to take into account the fact that you've selected on the dependent variable to learn more about rarer classes than a random sample would be able to tell you.

Making Predictions

On a related but distinct topic: Don't forget that you should be thresholding intelligently to make predictions. It is not always best to predict 1 when the model probability is greater 0.5. Another threshold may be better. To this end you should look into the Receiver Operating Characteristic (ROC) curves of your classifier, not just its predictive success with a default probability threshold.

Improve this answer
$\endgroup$
8
  • 10
    $\begingroup$ If you don't know the operational class frequencies, they can be estimated by EM without knowing the labels of the test/operational samples. The details are in Saerens et al. "Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure", Neural Computation, vol. 14, no. 1, pp. 21-41, 2002 ( dx.doi.org/10.1162/089976602753284446 ). I've used this a couple of times and was impressed at how well it worked. Note however that the theoretical correction is not normally optimal, and setting it via e.g. cross-validation is often better. $\endgroup$
    – Dikran Marsupial
    Commented Jan 8, 2011 at 17:17
  • $\begingroup$ Yeah, I should have mentioned that the results from the ROC-curve weren't convincing also. In this case I think there isn't a threshold that gives satisfactory results. $\endgroup$
    – Michiel
    Commented Jan 11, 2011 at 7:58
  • 2
    $\begingroup$ @Perlnika The details are in the paper link (in the simplest case you change the estimated intercept). To threshold not at 0.5, just get the predicted probabilities using predict and compute for each whether it is greater than the new threshold. $\endgroup$
    – conjugateprior
    Commented Jun 4, 2015 at 11:19
  • 2
    $\begingroup$ What's the difference between adjusting Intercept and adjusting threshold? The way I see it, adjusting Intercept changes where logistic function produces 0.5. But why would that matter, if someone adjusts threshold - that is, which value of logistic function separates the classes. $\endgroup$
    – Sassa NF
    Commented May 24, 2017 at 9:11
  • 3
    $\begingroup$ @SassaNF It's true that an intercept shift can be offset by by a threshold change. However, that couples your probability estimation (inference) to relative costliness of errors (loss function), whereas the latter might differ in applications. For example, when the cost of mistaking a 1 for a 0 is C times the cost of mistaking a 0 for a 1, then you'd want to threshold your estimated probability at 1/(1+C). $\endgroup$
    – conjugateprior
    Commented May 25, 2017 at 18:16
55
$\begingroup$

The problem is not that the classes are imbalanced per se, it is that there may not be sufficient patterns belonging to the minority class to adequately represent its distribution. This means that the problem can arise for any classifier (even if you have a synthetic problem and you know you have the true model), not just logistic regression. The good thing is that as more data become available, the "class imbalance" problem usually goes away. Having said which, 4:1 is not all that imbalanced.

If you use a balanced dataset, the important thing is to remember that the output of the model is now an estimate of the a-posteriori probability, assuming the classes are equally common, and so you may end up biasing the model too far. I would weight the patterns belonging to each class differently and choose the weights by minimising the cross-entropy on a test set with the correct operational class frequencies.

Alternatively (see the comments) it might be better to weight the positive and negative classes so they contribute equally to the training criterion (so there isn't a class imbalance problem in the estimation of the model parameters), but afterwards to rescale the posterior probabilities estimated by the classifier in order to compensate for the difference in the (effective) training set class frequencies and those in operational conditions (see this answer to a related question)

Improve this answer
$\endgroup$
8
  • 11
    $\begingroup$ +1 If you use a balanced dataset, the important thing is to remember that the output of the model is now an estimate of the a-posteriori probability $\endgroup$
    – Zhubarb
    Commented Jan 8, 2015 at 12:17
  • 2
    $\begingroup$ This is good point: [...] there may not be sufficient patterns belonging to the minority class to adequately represent its distribution. $\endgroup$
    – lsdr
    Commented Dec 6, 2019 at 16:39
  • $\begingroup$ Why is it if the dataset is balanced, the output of the model is an estimate of the a-posteriori probability? And why can this end up biasing the model too far? $\endgroup$
    – roulette01
    Commented Jun 29, 2020 at 16:45
  • $\begingroup$ @dd22205 by Bayes rule, $p(y|x) \propto p(x|y)p(y)$, so the a-posteriori probability estimate depends on the prior probability $p(y)$. If you use a balanced dataset where $p(y)=0.5$ (for a two class problem) then it only gives good estimates of $p(y|x)$ in operational use IF $p(y)=0.5$ in operational conditions as well as in the training data. Now some classifiers have a problem with imbalanced classes, so reducing the imbalance can help correct that bias in the model, but fully balancing the dataset is usually over-correcting as this bias is usually fairly small. $\endgroup$
    – Dikran Marsupial
    Commented Jun 30, 2020 at 8:02
  • $\begingroup$ Could you elaborate / explain in detail what you mean by "I would weight the patterns belonging to each class differently and choose the weights by minimising the cross-entropy on a test set with the correct operational class frequencies."? $\endgroup$
    – elexhobby
    Commented Jul 29, 2021 at 18:15
3
$\begingroup$

Think about the underlying distributions of the two samples. Do you have enough sample to measure both sub- populations without a massive amount of bias in the smaller sample?

See here for a longer explanation.

https://statisticalhorizons.com/logistic-regression-for-rare-events

Improve this answer
$\endgroup$
3
  • 7
    $\begingroup$ This does not seem to answer the question. $\endgroup$
    – Michael R. Chernick
    Commented May 28, 2017 at 18:13
  • $\begingroup$ That is because there is no definite answer! It is about how you apply it and the amount of bias one is willing to allow into the estimation process. $\endgroup$
    – Paul Tulloch
    Commented May 28, 2017 at 18:53
  • 1
    $\begingroup$ I think this is great answer. As far as I understand all attempts to correct imbalance rely on some external knowledge not captured in the experiment. In particular knowing underlying distribution would help with corrections. $\endgroup$
    – user1700890
    Commented Jun 2, 2017 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.