What is the difference between logistic regression and neural networks?

Question

How do we explain the difference between logistic regression and neural network to an audience that have no background in statistics?

Answer 1

I assume you're thinking of what used to be, and perhaps still are referred to as 'multilayer perceptrons' in your question about neural networks. If so then I'd explain the whole thing in terms of flexibility about the form of the decision boundary as a function of explanatory variables. In particular, for this audience, I wouldn't mention link functions / log odds etc. Just keep with the idea that the probability of an event is being predicted on the basis of some observations.

Here's a possible sequence:

• Make sure they know what a predicted probability is, conceptually speaking. Show it as a function of one variable in the context of some familiar data. Explain the decision context that will be shared by logistic regression and neural networks.
• Start with logistic regression. State that it is the linear case but show the linearity of the resulting decision boundary using a heat or contour plot of the output probabilities with two explanatory variables.
• Note that two classes may not be well-separated by the boundary they see and motivate a more flexible model to make a more curvy boundary. If necessary show some data that would be well distinguished this way. (This is why you start with 2 variables)
• Note that you could start complicating the original linear model with extra terms, e.g. squares or other transformations, and maybe show the boundaries that these generate.
• But then discard these, observing that you don't know in advance what the function form ought to be and you'd prefer to learn it from the data. Just as they get enthusiastic about this, note the impossibility of this in complete generality, and suggest that you are happy to assume that it should at least be 'smooth' rather than 'choppy', but otherwise determined by the data. (Assert that they were probably already thinking of only smooth boundaries, in the same way as they'd been speaking prose all their lives).
• Show the output of a generalized additive model where the output probability is a joint function of the pair of the original variables rather than a true additive combination - this is just for demonstration purposes. Importantly, call it a smoother because that's nice and general and describes things intuitively. Demonstrate the non-linear decision boundary in the picture as before.
• Note that this (currently anonymous) smoother has a smoothness parameter that controls how smooth it actually is, refer to this in passing as being like a prior belief about smoothness of the function turning the explanatory variables into the predicted probability. Maybe show the consequences of different smoothness settings on the decision boundary.
• Now introduce the neural net as a diagram. Point out that the second layer is just a logistic regression model, but also point out the non-linear transformation that happens in the hidden units. Remind the audience that this is just another function from input to output that will be non-linear in its decision boundary.
• Note that it has a lot of parameters and that some of them need to be constrained to make a smooth decision boundary - reintroduce the idea of a number that controls smoothness as the same (conceptually speaking) number that keeps the parameters tied together and away from extreme values. Also note that the more hidden units it has, the more different types of functional forms it can realise. To maintain intuition, talk about hidden units in terms of flexibility and parameter constraint in terms of smoothness (despite the mathematical sloppiness of this characterization)
• Then surprise them by claiming since you still don't know the functional form so you want to be infinitely flexible by adding an infinite number of hidden units. Let the practical impossibility of this sink in a bit. Then observe that this limit can be taken in the mathematics, and ask (rhetorically) what such a thing would look like.
• Answer that it would be a smoother again (a Gaussian process, as it happens; Neal, 1996, but this detail is not important), like the one they saw before. Observe that there is again a quantity that controls smoothness but no other particular parameters (integrated out, for those that care about this sort of thing).
• Conclude that neural networks are particular, implicitly limited, implementations of ordinary smoothers, which are the non-linear, not necessarily additive extensions of the logistic regression model. Then do it the other way, concluding that logistic regression is equivalent to a neural network model or a smoother with the smoothing parameter set to 'extra extra smooth' i.e. linear.

The advantages of this approach is that you don't have to really get into any mathematical detail to give the correct idea. In fact they don't have to understand either logistic regression or neural networks already to understand the similarities and differences.

The disadvantage of the approach is that you have to make a lot of pictures, and strongly resist the temptation to drop down into the algebra to explain things.

Answer 2

For a simpler summary:

Logistic regression: The simplest form of Neural Network, that results in decision boundaries that are a straight line

Neural Networks: A superset that includes Logistic regression and also other classifiers that can generate more complex decision boundaries.

(note: I'm referring to "plain" logistic regression, without the assistance of integral kernels)

(reference: deeplearning.ai courses by Andrew Ng, "Logistic regression as a neural network" and "Planar data classification with one hidden layer")

Answer 3

I am going to take the question literally: Someone with no background in statistics. And I'm not going to try to give that person a background in statistics. For instance, suppose you have to explain the difference to the CEO of a company or something like that.

So: Logistic regression is a tool for modeling a categorical variable in terms of other variables. It gives you ways to find out how changes in each of the "other" variables affects the odds of different outcomes in the first variable. The output is fairly easy to interpret.

Neural networks are a set of methods to let a computer try to learn from examples in ways that vaguely resemble how humans learn about things. It may result in models that are good predictors, but they are usually much more opaque than those from logistic regression.

Answer 4

I was taught that you can think of neural networks (with logistic activation functions) as as a weighted average of logit functions, with the weights themselves estimated. By choosing a large number of logits, you can fit any functional form. There's some graphical intuition in the Econometric Sense blog post.

Answer 5

The other answers are great. I would simply add some pictures showing that you can think of logistic regression and multi-class logistic regression (a.k.a. maxent, multinomial logistic regression, softmax regression, maximum entropy classifier) as a special architecture of neural networks.

From Sebastian Raschka, Michigan State University, on KDnuggets:

---

A few more illustration for multi-class logistic regression:

A similar illustration taken from http://www.deeplearningbook.org/ chapter 1:

And one more from TensorFlow tutorials:

E.g. in Caffe, you would implement logistic regression as follows:

Answer 6

I would use an example of a complicated but concrete problem the audience understands. Use hidden nodes whose interpretations are not trained, but have particular meanings.

If you use chess positions (predicting whether white will win), you could let the inputs be a representation of the board (ignore whether you can castle or capture en passant, or even whose move it is), say $64 \times 12$ binary inputs indicating whether there is a piece of each type on each square.

Linear regression determines how good it is to have a white knight on h4. It might not be obvious that it is good at all, but if it is on h4 it hasn't been captured, which probably outweighs other considerations. Linear regression probably recovers the rough values of the pieces, and that it is better to have your pieces toward the center of the board, and on your opponent's side of the board. Linear regression is unable to value combinations, such as that your queen on b2 is suddenly more valuable if the opposing king is on a1.

A neural network could have hidden nodes for concepts, such as "material advantage," "black king safety," "control of the center," "both rooks on the d-file," "isolated queen rook pawn," or "bishop mobility." Some of these can be estimated just from the board inputs, while others might have to be in a second or later hidden layer. The neural network can use these as inputs to the final evaluation of the position. These concepts help an expert to assess a position, so a neural network should be capable of more accurate assessments than a linear regression. However, it takes more work to create the neural network since you have to choose its structure and it has many more parameters to train.