Why use logistic regression instead of a neural network?

Question

I have an AI course in the which we studied and worked on logistic regression. After having read and studied a bit about neural networks I'm starting to ask myself why on earth would we use logistic regression instead of neural networks.

Is it just an academic standard way of introducing the topic of loss-function minimization based ML techniques ?

Answer 1

Logistic regression is basically a neural network with one layer.

So you can equivalently ask, why do people use networks with 12 layers and not 24 layers ? or why do people use 5-th order polynomials for curve fitting and not 10-th order ?

The question of how to choose to right model for your data is very broad. But in general, a model with more parameters is not necessarily better, and in many cases will be worse (think over-fitting). This is why for example convolutional networks work better for image data then fully-connected networks. It all depends on what type of data you have.

Answer 2

Well, that's like asking: why are people still use statistical models? Off the top of my head:

• Interpretable effect estimates with CIs and p-values, i.e. you can use the model for (causal) inference
• Parametric CIs on predictions
• Faster, and better use of your data, because (if you trust the maths) no need to set data aside for validation
• Ability to include prior knowledge on structure and parameters
• Due to the simpler model structure you can have smaller predictive error if the training dataset is small

If you are looking from a purely predictive ML viewpoint (and don't care about inference at all), I would say the logistic regression is a simple benchmark that you should run to see if you super shiny DNN does anything useful at all - if you can't beat the logistic regression, go home!

Answer 3

Even if you are only interested in prediction and do not care about inference (like identification of signifcant influence factors), logistic regression can be a reasonable approach: it depends on the amount of training data that you have available.

In the study listed below (sorry, it is in German), logistic regression outperformed neural networks for training set sizes below 20 000 items. Beyond that size, neural networks became better.

Note that there are many application areas where data is not as abundant as in data mining by social network companies. In medicine, e.g., it is not uncommon to have less than 50 test persons. Good luck with training a neural network on that data...

M. Kass: "Textklassifikation mit neuronalen Netzen und klassischen Modellen." Technischer Bericht Nr. 2019-01, Hochschule Niederrhein, Fachbereich Elektrotechnik und Informatik, 2019