# What is the difference between logistic regression and neural networks?

How do we explain the difference between logistic regression and neural network to an audience that have no background in statistics?

• Would anyone with no background in statistics really want to know? And, what would constitute an acceptable explanation of the difference? Perhaps a metaphor. Certainly not any of the answers below (to date), all of which entirely miss the "no background" requirement. Dec 24, 2016 at 20:07
• Q: "How do we explain the difference between logistic regression and neural network to an audience that have no background in statistics?" A: First you have to give them a background in statistics. Sep 20, 2017 at 13:48
• I see no reason this shouldn't stay open. We needn't take "explain ... no background in statistics" so literally. It is common to ask for explanations that would work for 'a 5 year old' or 'your grandmother'. These are just colloquial ways of asking for non- (or at least less) technical answers. To put it more explicitly, answers always seek to satisfy multiple constraints simultaneously, such as accuracy & brevity; here we add minimizing how technical it is. There is no reason we can't have a question seeking a less technical explanation of the difference b/t LR & ANNs. Sep 20, 2017 at 16:06
• @mbq It's funny that in November 2012 it was possible to describe neural networks as obsolete. May 7, 2018 at 2:52
• @littleO This pretty much still stands; compare NNs'18 with NNs'12 and you'll see the progress came from removing similarity to actual networks and actual neurons, instead going further into ensembles of algebraic operations with stochastic optimisation. But sure, apparently NN trademark proved so powerful it will live long and prosper, regardless of what it means.
– user88
May 23, 2018 at 15:41

I assume you're thinking of what used to be, and perhaps still are referred to as 'multilayer perceptrons' in your question about neural networks. If so then I'd explain the whole thing in terms of flexibility about the form of the decision boundary as a function of explanatory variables. In particular, for this audience, I wouldn't mention link functions / log odds etc. Just keep with the idea that the probability of an event is being predicted on the basis of some observations.

Here's a possible sequence:

• Make sure they know what a predicted probability is, conceptually speaking. Show it as a function of one variable in the context of some familiar data. Explain the decision context that will be shared by logistic regression and neural networks.
• Start with logistic regression. State that it is the linear case but show the linearity of the resulting decision boundary using a heat or contour plot of the output probabilities with two explanatory variables.
• Note that two classes may not be well-separated by the boundary they see and motivate a more flexible model to make a more curvy boundary. If necessary show some data that would be well distinguished this way. (This is why you start with 2 variables)
• Note that you could start complicating the original linear model with extra terms, e.g. squares or other transformations, and maybe show the boundaries that these generate.
• But then discard these, observing that you don't know in advance what the function form ought to be and you'd prefer to learn it from the data. Just as they get enthusiastic about this, note the impossibility of this in complete generality, and suggest that you are happy to assume that it should at least be 'smooth' rather than 'choppy', but otherwise determined by the data. (Assert that they were probably already thinking of only smooth boundaries, in the same way as they'd been speaking prose all their lives).
• Show the output of a generalized additive model where the output probability is a joint function of the pair of the original variables rather than a true additive combination - this is just for demonstration purposes. Importantly, call it a smoother because that's nice and general and describes things intuitively. Demonstrate the non-linear decision boundary in the picture as before.
• Note that this (currently anonymous) smoother has a smoothness parameter that controls how smooth it actually is, refer to this in passing as being like a prior belief about smoothness of the function turning the explanatory variables into the predicted probability. Maybe show the consequences of different smoothness settings on the decision boundary.
• Now introduce the neural net as a diagram. Point out that the second layer is just a logistic regression model, but also point out the non-linear transformation that happens in the hidden units. Remind the audience that this is just another function from input to output that will be non-linear in its decision boundary.
• Note that it has a lot of parameters and that some of them need to be constrained to make a smooth decision boundary - reintroduce the idea of a number that controls smoothness as the same (conceptually speaking) number that keeps the parameters tied together and away from extreme values. Also note that the more hidden units it has, the more different types of functional forms it can realise. To maintain intuition, talk about hidden units in terms of flexibility and parameter constraint in terms of smoothness (despite the mathematical sloppiness of this characterization)
• Then surprise them by claiming since you still don't know the functional form so you want to be infinitely flexible by adding an infinite number of hidden units. Let the practical impossibility of this sink in a bit. Then observe that this limit can be taken in the mathematics, and ask (rhetorically) what such a thing would look like.
• Answer that it would be a smoother again (a Gaussian process, as it happens; Neal, 1996, but this detail is not important), like the one they saw before. Observe that there is again a quantity that controls smoothness but no other particular parameters (integrated out, for those that care about this sort of thing).
• Conclude that neural networks are particular, implicitly limited, implementations of ordinary smoothers, which are the non-linear, not necessarily additive extensions of the logistic regression model. Then do it the other way, concluding that logistic regression is equivalent to a neural network model or a smoother with the smoothing parameter set to 'extra extra smooth' i.e. linear.

The advantages of this approach is that you don't have to really get into any mathematical detail to give the correct idea. In fact they don't have to understand either logistic regression or neural networks already to understand the similarities and differences.

The disadvantage of the approach is that you have to make a lot of pictures, and strongly resist the temptation to drop down into the algebra to explain things.

For a simpler summary:

Logistic regression: The simplest form of Neural Network, that results in decision boundaries that are a straight line

Neural Networks: A superset that includes Logistic regression and also other classifiers that can generate more complex decision boundaries.

(note: I'm referring to "plain" logistic regression, without the assistance of integral kernels)

(reference: deeplearning.ai courses by Andrew Ng, "Logistic regression as a neural network" and "Planar data classification with one hidden layer")

• From all the current answers I think this is the most realistically close to explaining the concepts to a person with no statistical background. Sep 20, 2017 at 13:47
• So a logistic logistic regression classifier IS A neural network? That makes a lot of sense. Apr 23, 2019 at 3:24

I am going to take the question literally: Someone with no background in statistics. And I'm not going to try to give that person a background in statistics. For instance, suppose you have to explain the difference to the CEO of a company or something like that.

So: Logistic regression is a tool for modeling a categorical variable in terms of other variables. It gives you ways to find out how changes in each of the "other" variables affects the odds of different outcomes in the first variable. The output is fairly easy to interpret.

Neural networks are a set of methods to let a computer try to learn from examples in ways that vaguely resemble how humans learn about things. It may result in models that are good predictors, but they are usually much more opaque than those from logistic regression.

• +1 This is a good initial effort to rise to the original challenge of providing an explanation that could be understood by a layperson, yet is reasonably clear and accurate.
– whuber
Sep 20, 2017 at 13:10
• You'll have to explain what "categorical", "variable", "odds" are. Also, Artificial Neural Networks are merely inspired by real neural networks. Our brain can't learn by back propagation as far as we know. So yeah, it's mostly a cool term for a relatively simplified concept. Also, logistic regression is a form of neural network, so there's that as well. Sep 20, 2017 at 13:46

I was taught that you can think of neural networks (with logistic activation functions) as as a weighted average of logit functions, with the weights themselves estimated. By choosing a large number of logits, you can fit any functional form. There's some graphical intuition in the Econometric Sense blog post.

The other answers are great. I would simply add some pictures showing that you can think of logistic regression and multi-class logistic regression (a.k.a. maxent, multinomial logistic regression, softmax regression, maximum entropy classifier) as a special architecture of neural networks.

A few more illustration for multi-class logistic regression:

A similar illustration taken from http://www.deeplearningbook.org/ chapter 1:

And one more from TensorFlow tutorials:

E.g. in Caffe, you would implement logistic regression as follows:

• So does back-propagation on such a neural network compute the same weights as logistic regression? Sep 9, 2015 at 21:13
• @ Mitch - I may be too late to the game to contribute. One key difference is that for a logistic regressioin one uses the mle to get the coefficients. In essence that is the choice of a specific error or loss function. For a neural net, the loss function is one of the choices. So with the correct loss fn (I think off the top of my head it's the standard L^2 norm) this is the case.
– meh
Mar 4, 2016 at 16:06
• So logistic regression can be formulated exactly like ADALINE (single layer neural network that uses batch/stochastic gradient descent), with the only key differences being the activation function being changed to sigmoid instead of linear, and the prediction function changing to >=0.5 with 0,1 labels instead of >=0 with -1,1 labels. Another strongly preferred, but optional difference is changing the cost function from RSS to logistic cost function because the sigmoid activation causes RSS to be non-convex so RSS can get stuck in local minimas. Dec 20, 2017 at 2:22

I would use an example of a complicated but concrete problem the audience understands. Use hidden nodes whose interpretations are not trained, but have particular meanings.

If you use chess positions (predicting whether white will win), you could let the inputs be a representation of the board (ignore whether you can castle or capture en passant, or even whose move it is), say $64 \times 12$ binary inputs indicating whether there is a piece of each type on each square.

Linear regression determines how good it is to have a white knight on h4. It might not be obvious that it is good at all, but if it is on h4 it hasn't been captured, which probably outweighs other considerations. Linear regression probably recovers the rough values of the pieces, and that it is better to have your pieces toward the center of the board, and on your opponent's side of the board. Linear regression is unable to value combinations, such as that your queen on b2 is suddenly more valuable if the opposing king is on a1.

A neural network could have hidden nodes for concepts, such as "material advantage," "black king safety," "control of the center," "both rooks on the d-file," "isolated queen rook pawn," or "bishop mobility." Some of these can be estimated just from the board inputs, while others might have to be in a second or later hidden layer. The neural network can use these as inputs to the final evaluation of the position. These concepts help an expert to assess a position, so a neural network should be capable of more accurate assessments than a linear regression. However, it takes more work to create the neural network since you have to choose its structure and it has many more parameters to train.