Suppose the event of interest occurs in approximately $10 \%$ of the cases where the number of cases is around $5,000$. Should you use a penalized logistic regression for this or is regular logistic regression okay? In other words, what qualifies something as a rare event?
-
3$\begingroup$ Do you have a reason for thinking that the frequency with which the event occurs has particular bearing on whether one should use penalized regression? $\endgroup$– rolando2Commented Oct 3, 2017 at 19:22
-
2$\begingroup$ How many variables do you have / intend to use in the model? Are you interested in testing hypotheses about the relationships b/t your variables and the event, or do you want to predict the probability of the event for future cases? $\endgroup$– gung - Reinstate MonicaCommented Oct 3, 2017 at 19:25
-
$\begingroup$ @gung: I cannot comment, but it is for prediction. $\endgroup$– guestguy4332Commented Oct 3, 2017 at 20:48
-
$\begingroup$ Please merge your accounts stats.stackexchange.com/users/179319/guestguy4332 and stats.stackexchange.com/users/179327/guestguy4332. Then you'll be able to comment in the threads for your own posts. $\endgroup$– Glen_bCommented Oct 3, 2017 at 21:14
2 Answers
Don't do anything special.
However, and this is crucial: choose a good quality measure. And that is not classification accuracy, sensitivity, specificity or similar measures, such as ROC curves. These can be very misleading in the case of unbalanced data, "identifying" that simply labeling everything as the majority class is "optimal". Which it isn't.
Oversampling the minority class or undersampling the majority class won't solve this problem, because it amounts to biasing your model and pretending that the population is different than it truly is. Neither will collecting more data solve your problem, since the relation between majority and minority classes won't change.
Instead, use probabilistic models instead of hard thresholded 0-1 classification, and then use proper scoring-rules. ("Proper" is really part of the term. There are proper and non-proper scoring rules. Classification accuracy is a non-proper scoring rule, and that is why it is not useful.)
Frank Harrell, who knows what he is talking about, has written extensively on the topic:
What you are asking here is basically food for numerous posts, opinions and a lot of research on the field of imbalanced classes. To answer your main question, there is not a straight answer of what a “rare” event is. I personally would say that your case is basically in the boundary. I will list some approaches here but seriously this will be the tip of the iceberg you will have to go back to google and look for more info:
- Do nothing. Sometimes it’s fine to just model the data as it is. Especially for an algorithm like logistic regression which not a classifier and predicts probabilities, these class imbalances will be taken into account.
In the case you want to use the model as a classifier you might consider to:
- Undersample the majority class. Sample observations so that you make the two classes balanced
- Oversample the minority class. Similarly sample some observations from the minority class, obviously with replacement
That being said, I believe that these 3 techniques can get you started. Given that you want prediction accuracy, I would suggest to look into other techniques as well. If you really want to use logistic regression then what I would do is:
- Split the data into train and test sets
- Define a loss function
- Check which approach minimises the loss function in the test set
-
2$\begingroup$ What problem would oversampling and undersampling address? The logistic regression model predicts a probability, and these probabilities will be calibrated to the class balance in the data the model is trained on. I think @ronaldo2 's comment is the salient point, why does the OP think there is a connection between the class balance and the decision to use a regularized model? $\endgroup$ Commented Oct 3, 2017 at 21:39
-
2$\begingroup$ The reason why I said that is because some people use logistic regression as a classifier (although it’s not!). The OP asks for prediction and my mind’s eye, he might use it as a classifier. In that context there are some papers suggesting to train your model in a 50-50 dataset. Thanks for the feedback, I will add it in my answer. $\endgroup$ Commented Oct 3, 2017 at 21:49
-
2$\begingroup$ Even if one were to use it as a classifier, you would need to choose a threshold to achieve the hard classification. The threshold needs to be based on correct classification benefits and misclassification costs. Even in the reductive (and often unrealistic) case of optimizing for classification accuracy with a balanced data set, it is not true that a 0.5 threshold is always optimal. $\endgroup$ Commented Oct 3, 2017 at 21:54
-
3$\begingroup$ I completely agree and that’s why I didn’t put this option as my optimal choice in the first place. Additionally, I recommended to try these approaches while trying alternative techniques and I wanted to give an approach that is generic enough for everything and not only for logistic regression. $\endgroup$ Commented Oct 3, 2017 at 22:11
-
$\begingroup$ Cool. May I suggest in your 1. you add something along the lines of "logistic regression predicts probabilities and these will reflect the class balance in your data"? $\endgroup$ Commented Oct 3, 2017 at 22:39