Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.

My problem is the following: whatever set of predictor variables I use, the classifications never get better than a specificity of 100% and a sensitivity of 0%. However unsatisfactory, this could actually be the best possible result, given the set of candidate predictor variables (from which I can't deviate).

But, I couldn't help but think I could do better, so I noticed that the categories of the dependent variable were quite unevenly balanced, almost 4:1. Could a more balanced subsample improve classifications?

  • 5
    $\begingroup$ It is hard to imagine how this could be. Perhaps you are cutting the predicted probability at 0.5? If so, try varying the cutoff. $\endgroup$ – Aniko Jan 7 '11 at 20:30
  • 4
    $\begingroup$ The area under the ROC-curve is .585, a rather poor result. This implies that there really isn't a cutoff value where the specificity/sensitivity trade-off is worth it. Fiddling with the cutoff won't improve classifications much, as it would just decrease the specificity by roughly as much as it increases the sensitivity. $\endgroup$ – Michiel Jan 11 '11 at 7:52
  • 3
    $\begingroup$ Are any of the Variables coefficients significantly different from $0$ (say more than five standard errors)? If not your problem Could be that you just don't have much explanatory power with your set of variables. $\endgroup$ – probabilityislogic Jan 21 '13 at 11:20
  • 2
    $\begingroup$ Note also that your sample size in terms of making good predictions is really the number of unique patterns in the predictor variable, and not the number of sampled individuals. For example, a model with a single categorical predictor variable with two levels can only fit a logistic regression model with two parameters (one for each category), even if there are millions people in the sample. $\endgroup$ – probabilityislogic Jan 21 '13 at 11:32
  • $\begingroup$ Related: stats.stackexchange.com/questions/67903 $\endgroup$ – amoeba Nov 6 '18 at 21:33

Balance in the Training Set

For logistic regression models unbalanced training data affects only the estimate of the model intercept (although this of course skews all the predicted probabilities, which in turn compromises your predictions). Fortunately the intercept correction is straightforward: Provided you know, or can guess, the true proportion of 0s and 1s and know the proportions in the training set you can apply a rare events correction to the intercept. Details are in King and Zeng (2001) [PDF].

These 'rare event corrections' were designed for case control research designs, mostly used in epidemiology, that select cases by choosing a fixed, usually balanced number of 0 cases and 1 cases, and then need to correct for the resulting sample selection bias. Indeed, you might train your classifier the same way. Pick a nice balanced sample and then correct the intercept to take into account the fact that you've selected on the dependent variable to learn more about rarer classes than a random sample would be able to tell you.

Making Predictions

On a related but distinct topic: Don't forget that you should be thresholding intelligently to make predictions. It is not always best to predict 1 when the model probability is greater 0.5. Another threshold may be better. To this end you should look into the Receiver Operating Characteristic (ROC) curves of your classifier, not just its predictive success with a default probability threshold.

| cite | improve this answer | |
  • 8
    $\begingroup$ If you don't know the operational class frequencies, they can be estimated by EM without knowing the labels of the test/operational samples. The details are in Saerens et al. "Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure", Neural Computation, vol. 14, no. 1, pp. 21-41, 2002 ( dx.doi.org/10.1162/089976602753284446 ). I've used this a couple of times and was impressed at how well it worked. Note however that the theoretical correction is not normally optimal, and setting it via e.g. cross-validation is often better. $\endgroup$ – Dikran Marsupial Jan 8 '11 at 17:17
  • $\begingroup$ Yeah, I should have mentioned that the results from the ROC-curve weren't convincing also. In this case I think there isn't a threshold that gives satisfactory results. $\endgroup$ – Michiel Jan 11 '11 at 7:58
  • $\begingroup$ Regarding making predictions: how can I take into account size of my training set for 0 and 1 outcomes? I indeed don't want to use threshold of 0.5, but not sure how to do this in R. $\endgroup$ – Perlnika Jun 3 '15 at 15:39
  • 1
    $\begingroup$ @Perlnika The details are in the paper link (in the simplest case you change the estimated intercept). To threshold not at 0.5, just get the predicted probabilities using predict and compute for each whether it is greater than the new threshold. $\endgroup$ – conjugateprior Jun 4 '15 at 11:19
  • 1
    $\begingroup$ @SassaNF It's true that an intercept shift can be offset by by a threshold change. However, that couples your probability estimation (inference) to relative costliness of errors (loss function), whereas the latter might differ in applications. For example, when the cost of mistaking a 1 for a 0 is C times the cost of mistaking a 0 for a 1, then you'd want to threshold your estimated probability at 1/(1+C). $\endgroup$ – conjugateprior May 25 '17 at 18:16

The problem is not that the classes are imbalanced per se, it is that there may not be sufficient patterns belonging to the minority class to adequately represent its distribution. This means that the problem can arise for any classifier (even if you have a synthetic problem and you know you have the true model), not just logistic regression. The good thing is that as more data become available, the "class imbalance" problem usually goes away. Having said which, 4:1 is not all that imbalanced.

If you use a balanced dataset, the important thing is to remember that the output of the model is now an estimate of the a-posteriori probability, assuming the classes are equally common, and so you may end up biasing the model too far. I would weight the patterns belonging to each class differently and choose the weights by minimising the cross-entropy on a test set with the correct operational class frequencies.

| cite | improve this answer | |
  • 9
    $\begingroup$ +1 If you use a balanced dataset, the important thing is to remember that the output of the model is now an estimate of the a-posteriori probability $\endgroup$ – Zhubarb Jan 8 '15 at 12:17
  • $\begingroup$ This is good point: [...] there may not be sufficient patterns belonging to the minority class to adequately represent its distribution. $\endgroup$ – lsdr Dec 6 '19 at 16:39
  • $\begingroup$ Why is it if the dataset is balanced, the output of the model is an estimate of the a-posteriori probability? And why can this end up biasing the model too far? $\endgroup$ – roulette01 Jun 29 at 16:45
  • $\begingroup$ @dd22205 by Bayes rule, $p(y|x) \propto p(x|y)p(y)$, so the a-posteriori probability estimate depends on the prior probability $p(y)$. If you use a balanced dataset where $p(y)=0.5$ (for a two class problem) then it only gives good estimates of $p(y|x)$ in operational use IF $p(y)=0.5$ in operational conditions as well as in the training data. Now some classifiers have a problem with imbalanced classes, so reducing the imbalance can help correct that bias in the model, but fully balancing the dataset is usually over-correcting as this bias is usually fairly small. $\endgroup$ – Dikran Marsupial Jun 30 at 8:02

Think about the underlying distributions of the two samples. Do you have enough sample to measure both sub- populations without a massive amount of bias in the smaller sample?

See here for a longer explanation.


| cite | improve this answer | |
  • 6
    $\begingroup$ This does not seem to answer the question. $\endgroup$ – Michael R. Chernick May 28 '17 at 18:13
  • $\begingroup$ That is because there is no definite answer! It is about how you apply it and the amount of bias one is willing to allow into the estimation process. $\endgroup$ – Paul Tulloch May 28 '17 at 18:53
  • 1
    $\begingroup$ I think this is great answer. As far as I understand all attempts to correct imbalance rely on some external knowledge not captured in the experiment. In particular knowing underlying distribution would help with corrections. $\endgroup$ – user1700890 Jun 2 '17 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.