Is it normal for simple logistic regression to significantly outperform any other statistical ML algorithm? I'm working on a simple classification project with an imbalanced (minority-to-majority-ratio ~ 0.2) dataset that has ~4000 rows and ~200 features.
I noticed that, for my dataset, a simple logistic regression significantly outperforms most other classification algorithms. The ROC AUC score for the validation data in my LR model is ~0.8, compare to 0.52-0.62 for other algorithms. I tried many different algorithms such as RF, GBM, XGBoost, LighGBM, SVM, etc. and used SkOpt's Bayesian optimization to tune the hyperparameter in each algorithm.
I'm trying to understand what intrinsically is different about my data and was wondering if anyone has encountered such superior performance from LR and what were their thoughts.
 A: My informal answer is that maximum likelihood estimation, the method behind logistic regression, finds the set of parameters that fit the data the best given some assumptions. If your dataset satisfies those assumptions very well and you have lots of data, then it is difficult to do better.
This paper about logistic regression vs random forest (I just found the paper by Googling) reports that RF performed better than LR according to the considered accuracy measured in approximately 69% of the datasets. So it suggests that is not unusual for logistic regression to beat random forest.
Also, Wikipedia reports that on the MNIST dataset, the linear classifier has error rate of 7.6% which is higher than other methods but I would say it is pretty good in absolute terms.
My impression is that older techniques like logistic regression are a bit underrated relative to modern ones like random forest or SVM but in many cases they are still preferable. My 2p-
A: I haven't done enough projects where I've compared different models to say whether Logistic Regression usually outperforms other ML algorithms so unfortunately I can't answer that part of the question.
I wanted to note though that in imbalanced datasets maximum likelihood estimates of logistic regression coefficients can be biased (King and Zheng, 2001).
(King and Zheng, 2001) provide methods of estimating the bias and adjusting the coefficients to remove the estimated bias.
I'm not overly familiar with the ML algorithms you mentioned in the question however a I believe some of them have a way of weighting the loss function to account for the imbalance in the classes.
Sorry that this answer is a bit vague but hopefully you've got something new to explore.
References
[1] Gary King and Langche Zeng. 2001. “Logistic Regression in Rare Events Data.” Political Analysis, 9, Pp. 137–163. Copy at https://tinyurl.com/y463rgub
