1
$\begingroup$

I'm working on a simple classification project with an imbalanced (minority-to-majority-ratio ~ 0.2) dataset that has ~4000 rows and ~200 features.

I noticed that, for my dataset, a simple logistic regression significantly outperforms most other classification algorithms. The ROC AUC score for the validation data in my LR model is ~0.8, compare to 0.52-0.62 for other algorithms. I tried many different algorithms such as RF, GBM, XGBoost, LighGBM, SVM, etc. and used SkOpt's Bayesian optimization to tune the hyperparameter in each algorithm.

I'm trying to understand what intrinsically is different about my data and was wondering if anyone has encountered such superior performance from LR and what were their thoughts.

Improve this question
$\endgroup$
3
  • 5
    $\begingroup$ I'd say it was not "normal", but at the same time it is not unusual. It is always a good idea to try linear regression, if only as a baseline, and that advice goes back at least as far as the initial neural network boom of the late 1980s. $\endgroup$
    – Dikran Marsupial
    May 15, 2022 at 8:26
  • 4
    $\begingroup$ So many things can be done in a suboptimal way with each of the methods, like using bad metrics, applying oversampling, having dependent rows but assuming independence etc. $\endgroup$
    – Michael M
    May 15, 2022 at 13:25
  • $\begingroup$ I don't think it's possible to say that one method works because the data looks like such. What is important is that it is the correct method for the problem and that you are not violating any of the assumptions. I also would not assume AUC is the perfect evaluation measure and would examine others $\endgroup$
    – Ralph Winters
    May 15, 2022 at 22:07

2 Answers 2

Reset to default
1
$\begingroup$

My informal answer is that maximum likelihood estimation, the method behind logistic regression, finds the set of parameters that fit the data the best given some assumptions. If your dataset satisfies those assumptions very well and you have lots of data, then it is difficult to do better.

This paper about logistic regression vs random forest (I just found the paper by Googling) reports that RF performed better than LR according to the considered accuracy measured in approximately 69% of the datasets. So it suggests that is not unusual for logistic regression to beat random forest.

Also, Wikipedia reports that on the MNIST dataset, the linear classifier has error rate of 7.6% which is higher than other methods but I would say it is pretty good in absolute terms.

My impression is that older techniques like logistic regression are a bit underrated relative to modern ones like random forest or SVM but in many cases they are still preferable. My 2p-

Improve this answer
$\endgroup$
5
  • $\begingroup$ I'll wager that the paper to which you referred used logistic regression in the same way as when it was invented in 1958, i.e, assumed effects of all continuous predictors are linear. That's not a valid comparison. $\endgroup$
    – Frank Harrell
    May 15, 2022 at 21:53
  • $\begingroup$ @FrankHarrell: assumed effects of all continuous predictors are linear - would you mind elaborating on this? Isn't linearity between predictors and $f(response)$ ($f$ typically being the logit function in logistic regression) an assumption of all linear models (hence the name linear)? $\endgroup$
    – dariober
    May 16, 2022 at 8:25
  • $\begingroup$ No linear models assume linearity in X. They are linear only in $\beta$. It is very commonplace to include square, cube, logarithmic, and spline terms in X. Nonlinearity in X is the most common assumption violation that matters in prediction. Splines have been used with logistic regression since 1984. $\endgroup$
    – Frank Harrell
    May 16, 2022 at 11:18
  • $\begingroup$ @FrankHarrell Thanks for replying - Yes, in my comment I meant linearity in $\beta$. As I understand it, your criticism about the paper is that it didn't allow LR to be flexible by applying e.g. splines to improve the prediction. If so, the quoted 69% in favor of random forest is probably an overestimate. $\endgroup$
    – dariober
    May 16, 2022 at 11:44
  • $\begingroup$ Exactly. LR was handcuffed using completely out-of-date statistical approaches. $\endgroup$
    – Frank Harrell
    May 16, 2022 at 21:24
0
$\begingroup$

I haven't done enough projects where I've compared different models to say whether Logistic Regression usually outperforms other ML algorithms so unfortunately I can't answer that part of the question.

I wanted to note though that in imbalanced datasets maximum likelihood estimates of logistic regression coefficients can be biased (King and Zheng, 2001).

(King and Zheng, 2001) provide methods of estimating the bias and adjusting the coefficients to remove the estimated bias.

I'm not overly familiar with the ML algorithms you mentioned in the question however a I believe some of them have a way of weighting the loss function to account for the imbalance in the classes.

Sorry that this answer is a bit vague but hopefully you've got something new to explore.

References

[1] Gary King and Langche Zeng. 2001. “Logistic Regression in Rare Events Data.” Political Analysis, 9, Pp. 137–163. Copy at https://tinyurl.com/y463rgub

Improve this answer
$\endgroup$
2
  • 4
    $\begingroup$ The paper you cite is for situations where there are "dozens to thousands of times fewer ones...than zeros." In the example of this question it's only a 4/1 ratio, which isn't necessarily that bad. $\endgroup$
    – EdM
    May 15, 2022 at 11:59
  • $\begingroup$ Fair comment, I may have jumped the gun a little. However in the simulation studies presented in section 6 the authors show the effect that varying sample size and proportion of 1's have on the difference between the corrected and non corrected models. This is something the original poster can take into consideration when interpreting their model and decide if it is necessary to make the correction $\endgroup$
    – dy_1992
    May 21, 2022 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.