# Equivalence of neural networks to linear regression

Are neural networks equivalent to linear regression if the activation function is linear (g(x) = x), and back propagation is basically just SGD for a least squares problem? Or is that only true for single layer neural networks?

I'm very new to neural networks and I basically have very little idea what's going on, so any intuition anyone could give would be appreciated. Thanks!

Edit: I'm going to write some math to establish some things.

Assume that I have a simple network with two hidden layers, and each hidden layer has K units. My input has M features, and my output is one of P classes.

let's just consider 1 sample for now.

Input vector: $x$, of length M

Hidden layer 1: $a_1 = \phi(W_1 [x;1])$ where $W_1$ is a K x (M+1) matrix and $a_1$ is the first activation vector of length K. All the activation functions are the same, and are denoted $\phi(x)$.

Hidden layers 2 ... K: $a_k = \phi(W_k [a_{k-1};1])$ where $W_k$ is a K x (K+1) matrix and $a_k$ is the $k$th activation vector of length K.

Output: $y = \phi(W_{K+1}[a_K;1])$ where $W_{K+1}$ is a P x (K+1) matrix and $y$ is the output of length P.

So, as one involved function, it's

$y = \phi(W_{K+1}[\phi(W_K [\phi(W_{K-1} ... [\phi(W_1 [x;1]);1] ... ; 1] ;1])$.

Clearly this is a very nonlinear and nonconvex function of the $W_k$s, as pointed out in the comments, so indeed it cannot be like linear regression.

• Saxe and McClelland have a paper on this arxiv.org/abs/1312.6120 basically they look at deep linear network dynamics to understand deep nonlinear nets Nov 1, 2015 at 0:57
• You can be more general here. Neural networks are equivalent (reinvented) to generalized linear model. Those kind of things are being reinvented all the times all over the place, I am sure econometrics and engineers have their own name for same problems. There is a medical doctor, who published his reinvention of integration (true story) Nov 3, 2015 at 11:53
• Yeah I think the thing that was tripping me up is that NNs are a (generalized) linear model of the input, but not of the weights. I'm still learning about this now but I agree, the foundational idea seems super simple but I think most of the recent breakthroughs must be in successful large scale implementations, and any tricks of the trade developed to make these things efficient. Nov 4, 2015 at 16:00

• Think about it this way: Suppose you have 2 hidden neurons and you have a set of globally optimal weights $w$. But then you could exchange the roles of the hidden neurons (since they are generic) and get the exact same value for the cost function. In this case, though, maybe all local optima are globally optimal? I don't know that for sure, but there certainly is more than one global optimum. Oct 31, 2015 at 0:47