# Difference between logistic regression and neural networks

How do we explain the difference between logistic regression and neural network to an audience that have no background in statistics?

• The first is a trick to use linear regression method for binary classification, the second is a group of obsolete, overcomplex machine learning methods. – mbq Nov 15 '12 at 13:23
• Would anyone with no background in statistics really want to know? And, what would constitute an acceptable explanation of the difference? Perhaps a metaphor. Certainly not any of the answers below (to date), all of which entirely miss the "no background" requirement. – rolando2 Dec 24 '16 at 20:07
• I second what @rolando2 said. This question is almost impossible to answer. How can you explain what LR and MLPs do to people with no background in statistics? Even worse, compare those. – Firebug Sep 19 '17 at 22:50
• "Logistic regression is boring, neural networks are cool." If they want more details: "Neural networks can do more things than logistic regression, and... Tensor Flow!" – Wayne Sep 20 '17 at 13:15
• Q: "How do we explain the difference between logistic regression and neural network to an audience that have no background in statistics?" A: First you have to give them a background in statistics. – Firebug Sep 20 '17 at 13:48

I assume you're thinking of what used to be, and perhaps still are referred to as 'multilayer perceptrons' in your question about neural networks. If so then I'd explain the whole thing in terms of flexibility about the form of the decision boundary as a function of explanatory variables. In particular, for this audience, I wouldn't mention link functions / log odds etc. Just keep with the idea that the probability of an event is being predicted on the basis of some observations.

Here's a possible sequence:

• Make sure they know what a predicted probability is, conceptually speaking. Show it as a function of one variable in the context of some familiar data. Explain the decision context that will be shared by logistic regression and neural networks.
• Start with logistic regression. State that it is the linear case but show the linearity of the resulting decision boundary using a heat or contour plot of the output probabilities with two explanatory variables.
• Note that two classes may not be well-separated by the boundary they see and motivate a more flexible model to make a more curvy boundary. If necessary show some data that would be well distinguished this way. (This is why you start with 2 variables)
• Note that you could start complicating the original linear model with extra terms, e.g. squares or other transformations, and maybe show the boundaries that these generate.
• But then discard these, observing that you don't know in advance what the function form ought to be and you'd prefer to learn it from the data. Just as they get enthusiastic about this, note the impossibility of this in complete generality, and suggest that you are happy to assume that it should at least be 'smooth' rather than 'choppy', but otherwise determined by the data. (Assert that they were probably already thinking of only smooth boundaries, in the same way as they'd been speaking prose all their lives).
• Show the output of a generalized additive model where the output probability is a joint function of the pair of the original variables rather than a true additive combination - this is just for demonstration purposes. Importantly, call it a smoother because that's nice and general and describes things intuitively. Demonstrate the non-linear decision boundary in the picture as before.
• Note that this (currently anonymous) smoother has a smoothness parameter that controls how smooth it actually is, refer to this in passing as being like a prior belief about smoothness of the function turning the explanatory variables into the predicted probability. Maybe show the consequences of different smoothness settings on the decision boundary.
• Now introduce the neural net as a diagram. Point out that the second layer is just a logistic regression model, but also point out the non-linear transformation that happens in the hidden units. Remind the audience that this is just another function from input to output that will be non-linear in its decision boundary.
• Note that it has a lot of parameters and that some of them need to be constrained to make a smooth decision boundary - reintroduce the idea of a number that controls smoothness as the same (conceptually speaking) number that keeps the parameters tied together and away from extreme values. Also note that the more hidden units it has, the more different types of functional forms it can realise. To maintain intuition, talk about hidden units in terms of flexibility and parameter constraint in terms of smoothness (despite the mathematical sloppiness of this characterization)
• Then surprise them by claiming since you still don't know the functional form so you want to be infinitely flexible by adding an infinite number of hidden units. Let the practical impossibility of this sink in a bit. Then observe that this limit can be taken in the mathematics, and ask (rhetorically) what such a thing would look like.
• Answer that it would be a smoother again (a Gaussian process, as it happens; Neal, 1996, but this detail is not important), like the one they saw before. Observe that there is again a quantity that controls smoothness but no other particular parameters (integrated out, for those that care about this sort of thing).
• Conclude that neural networks are particular, implicitly limited, implementations of ordinary smoothers, which are the non-linear, not necessarily additive extensions of the logistic regression model. Then do it the other way, concluding that logistic regression is equivalent to a neural network model or a smoother with the smoothing parameter set to 'extra extra smooth' i.e. linear.

The advantages of this approach is that you don't have to really get into any mathematical detail to give the correct idea. In fact they don't have to understand either logistic regression or neural networks already to understand the similarities and differences.

The disadvantage of the approach is that you have to make a lot of pictures, and strongly resist the temptation to drop down into the algebra to explain things.

I am going to take the question literally: Someone with no background in statistics. And I'm not going to try to give that person a background in statistics. For instance, suppose you have to explain the difference to the CEO of a company or something like that.

So: Logistic regression is a tool for modeling a categorical variable in terms of other variables. It gives you ways to find out how changes in each of the "other" variables affects the odds of different outcomes in the first variable. The output is fairly easy to interpret.

Neural networks are a set of methods to let a computer try to learn from examples in ways that vaguely resemble how humans learn about things. It may result in models that are good predictors, but they are usually much more opaque than those from logistic regression.

• +1 This is a good initial effort to rise to the original challenge of providing an explanation that could be understood by a layperson, yet is reasonably clear and accurate. – whuber Sep 20 '17 at 13:10
• You'll have to explain what "categorical", "variable", "odds" are. Also, Artificial Neural Networks are merely inspired by real neural networks. Our brain can't learn by back propagation as far as we know. So yeah, it's mostly a cool term for a relatively simplified concept. Also, logistic regression is a form of neural network, so there's that as well. – Firebug Sep 20 '17 at 13:46

I was taught that you can think of neural networks (with logistic activation functions) as as a weighted average of logit functions, with the weights themselves estimated. By choosing a large number of logits, you can fit any functional form. There's some graphical intuition in the Econometric Sense blog post.

The other answers are great. I would simply add some pictures showing that you can think of logistic regression and multi-class logistic regression (a.k.a. maxent, multinomial logistic regression, softmax regression, maximum entropy classifier) as a special architecture of neural networks.

A few more illustration for multi-class logistic regression:

A similar illustration taken from http://www.deeplearningbook.org/ chapter 1:

And one more from TensorFlow tutorials:

E.g. in Caffe, you would implement logistic regression as follows:

• So does back-propagation on such a neural network compute the same weights as logistic regression? – Mitch Sep 9 '15 at 21:13
• @ Mitch - I may be too late to the game to contribute. One key difference is that for a logistic regressioin one uses the mle to get the coefficients. In essence that is the choice of a specific error or loss function. For a neural net, the loss function is one of the choices. So with the correct loss fn (I think off the top of my head it's the standard L^2 norm) this is the case. – aginensky Mar 4 '16 at 16:06
• So logistic regression can be formulated exactly like ADALINE (single layer neural network that uses batch/stochastic gradient descent), with the only key differences being the activation function being changed to sigmoid instead of linear, and the prediction function changing to >=0.5 with 0,1 labels instead of >=0 with -1,1 labels. Another strongly preferred, but optional difference is changing the cost function from RSS to logistic cost function because the sigmoid activation causes RSS to be non-convex so RSS can get stuck in local minimas. – Austin Dec 20 '17 at 2:22

For a simpler summary:

Logistic regression: The simplest form of Neural Network, that results in decision boundaries that are a straight line

Neural Networks: A superset that includes Logistic regression and also other classifiers that can generate more complex decision boundaries.

(note: I'm referring to "plain" logistic regression, without the assistance of integral kernels)

(reference: deeplearning.ai courses by Andrew Ng, "Logistic regression as a neural network" and "Planar data classification with one hidden layer")

• From all the current answers I think this is the most realistically close to explaining the concepts to a person with no statistical background. – Firebug Sep 20 '17 at 13:47

I would use an example of a complicated but concrete problem the audience understands. Use hidden nodes whose interpretations are not trained, but have particular meanings.

If you use chess positions (predicting whether white will win), you could let the inputs be a representation of the board (ignore whether you can castle or capture en passant, or even whose move it is), say $64 \times 12$ binary inputs indicating whether there is a piece of each type on each square.

Linear regression determines how good it is to have a white knight on h4. It might not be obvious that it is good at all, but if it is on h4 it hasn't been captured, which probably outweighs other considerations. Linear regression probably recovers the rough values of the pieces, and that it is better to have your pieces toward the center of the board, and on your opponent's side of the board. Linear regression is unable to value combinations, such as that your queen on b2 is suddenly more valuable if the opposing king is on a1.

A neural network could have hidden nodes for concepts, such as "material advantage," "black king safety," "control of the center," "both rooks on the d-file," "isolated queen rook pawn," or "bishop mobility." Some of these can be estimated just from the board inputs, while others might have to be in a second or later hidden layer. The neural network can use these as inputs to the final evaluation of the position. These concepts help an expert to assess a position, so a neural network should be capable of more accurate assessments than a linear regression. However, it takes more work to create the neural network since you have to choose its structure and it has many more parameters to train.