Flexible version of logistic regression

Question

I'm trying to fit a logistic regression where there is a huge difference in the number of data points in either group (70 Vs 10,000). A statistician friend of mine has told me that this is a known problem with logistic regression and that for those kinds of numbers it overfits the data and basically doesn't work. When I bin the data and compare to the model, it is fairly obvious that this is definitely the case.

I'm wondering if anyone is aware of a better / more flexible method for fitting this kind of binary response data?

(I'm not a statistician by the way so go easy on me!)

Answer 1

This problem surfaces in virtually all classification approaches, whether logistic regression, support vector classification, or Naive Bayes classification. There are two intertwined issues:

• A model trained on an imbalanced dataset may overfit in the sense of acquiring a bias in favour of the majority class.

• When evaluating this model on a test dataset with the same degree of imbalance, classification accuracy can be a hugely misleading performance measure.

The literature on these issues has come up with three solution strategies:

1. You can restore balance on the training set by undersampling the large class or by oversampling the small class, to prevent bias from arising in the first place (see the response by @grotos).

2. Alternatively, you can modify the costs of misclassification to prevent the model from acquiring a bias in the first place.

3. An additional safeguard is to replace the accuracy by the so-called balanced accuracy. It is defined as the arithmetic mean of the class-specific accuracies, $\phi := \frac{1}{2}\left(\pi^+ + \pi^-\right),$ where $\pi^+$ and $\pi^-$ represent the accuracy obtained on positive and negative examples, respectively. If the classifier performs equally well on either class, this term reduces to the conventional accuracy (i.e., the number of correct predictions divided by the total number of predictions). In contrast, if the conventional accuracy is above chance only because the classifier takes advantage of an imbalanced test set, then the balanced accuracy, as appropriate, will drop to chance (see sketch below which I have taken from my response to a related question).

As detailed in my previous response, I would recommend to consider at least two of the above approaches in conjunction. For example, you could oversample your minority class to prevent your classifier from acquiring a bias in favour the majority class. Following this, when evaluating the performance of your classifier, you could replace the accuracy by the balanced accuracy.

Answer 2

That it doesn't work does not come from the unbalanced size of the groups, but from the smallness of one of the groups. Downsampling the larger group is ok but does not help with overfitting. (BTW, there is an easy and elegant way to correct the predictions from the downsampled model, by adding ±log(r) to the linear terms where r is the downsampling ratio.)

If overfitting really is the problem, you need to either decrease the number of variables, or regularize the model.

Answer 3

Do you mean the distribution of response, i.e. you have 70 cases of "YES" and 10000 of "NO"?

If so, that is a common problem in data mining applications. Imagine a database with 1,000,000 instances, where only about 1,000 cases are "YES". Response rate of 1% and even less is a common thing in a business predictive modeling. And if you pick a sample to train a model that is a huge problem, especially with assessing stability of given model.

What we do is pick a sample with different proportions. In aforementioned example, that would be 1000 cases of "YES" and, for instance, 9000 of "NO" cases. This approach gives more stable models. However, it have to be tested on a real sample (that with 1,000,000 rows).

I've tested it with data mining models, such as logistic regression, decision trees, etc. However, I haven't used it with "proper" [1] statistic models.

You can search it as "oversampling in statistics", the first result is pretty good: http://www.statssa.gov.za/isi2009/ScientificProgramme/IPMS/1621.pdf

[1] "proper" in meaning "not data mining".

Answer 4

If you want a classification technique that is insensitive to the relative proportion of examples from different classes, Support Vector Machines have that property as do decision trees.