Are neural networks equivalent to linear regression if the activation function is linear (g(x) = x), and back propagation is basically just SGD for a least squares problem? Or is that only true for single layer neural networks?

I'm very new to neural networks and I basically have very little idea what's going on, so any intuition anyone could give would be appreciated. Thanks!

Edit: I'm going to write some math to establish some things.

Assume that I have a simple network with two hidden layers, and each hidden layer has K units. My input has M features, and my output is one of P classes.

let's just consider 1 sample for now.

Input vector: $x$, of length M

Hidden layer 1: $a_1 = \phi(W_1 [x;1])$ where $W_1$ is a K x (M+1) matrix and $a_1$ is the first activation vector of length K. All the activation functions are the same, and are denoted $\phi(x)$.

Hidden layers 2 ... K: $a_k = \phi(W_k [a_{k-1};1])$ where $W_k$ is a K x (K+1) matrix and $a_k$ is the $k$th activation vector of length K.

Output: $y = \phi(W_{K+1}[a_K;1])$ where $W_{K+1}$ is a P x (K+1) matrix and $y$ is the output of length P.

So, as one involved function, it's

$y = \phi(W_{K+1}[\phi(W_K [\phi(W_{K-1} ... [\phi(W_1 [x;1]);1] ... ; 1] ;1])$.

Clearly this is a very nonlinear and nonconvex function of the $W_k$s, as pointed out in the comments, so indeed it cannot be like linear regression.

  • 1
    Saxe and McClelland have a paper on this arxiv.org/abs/1312.6120 basically they look at deep linear network dynamics to understand deep nonlinear nets – seanv507 Nov 1 '15 at 0:57
  • I'll take a look, thanks for the tip! – Y. S. Nov 3 '15 at 1:38
  • 1
    You can be more general here. Neural networks are equivalent (reinvented) to generalized linear model. Those kind of things are being reinvented all the times all over the place, I am sure econometrics and engineers have their own name for same problems. There is a medical doctor, who published his reinvention of integration (true story) – rep_ho Nov 3 '15 at 11:53
  • Yeah I think the thing that was tripping me up is that NNs are a (generalized) linear model of the input, but not of the weights. I'm still learning about this now but I agree, the foundational idea seems super simple but I think most of the recent breakthroughs must be in successful large scale implementations, and any tricks of the trade developed to make these things efficient. – Y. S. Nov 4 '15 at 16:00
up vote 2 down vote accepted

Yes they are equivalent, but more expensive to compute and train. The thing is that combining linear regressions in a linear fashion does not change anything. I'm not absolutely sure that backprop gets you the exact same result (I'd think it would), but it can certainly perform no better in that particular setting than ordinary regression.

  • Hmm, ok that makes sense, but just to expand a little, in a system with many hidden layers and no nonlinear activation, using the typical back propagation algorithm, is that equivalent to SGD for minimizing the MSE? Thanks! – Y. S. Oct 30 '15 at 16:53
  • I am not sure, but what I do know is that the objective function is non-convex (even with lineal activation) and hence you could arrive at different minima. Just don't use neural networks with an all-linear link, it's just not a good idea! – Felipe Gerard Oct 30 '15 at 20:16
  • haha it definitely seems fruitless to use an all linear link! I just thought it would be easier to analyze with this simplification. If the cost function is not convex it would definitely not be the case, but I think I still need to convince myself of that point! (MSE is convex, and if weights are all linearly appiled, shouldn't it be convex? hmm...) – Y. S. Oct 30 '15 at 23:16
  • Think about it this way: Suppose you have 2 hidden neurons and you have a set of globally optimal weights $w$. But then you could exchange the roles of the hidden neurons (since they are generic) and get the exact same value for the cost function. In this case, though, maybe all local optima are globally optimal? I don't know that for sure, but there certainly is more than one global optimum. – Felipe Gerard Oct 31 '15 at 0:47
  • So I extended my post a bit to actually work out the details. I think it IS convex, actually, and I think it IS just SGD for a convex function. Without the assumption that the activations are convex functions, then it becomes sgd for a nonconvex function. I think. But it's possible my interpretation is somehow wrong, so I'm curious to know your thoughts! – Y. S. Oct 31 '15 at 2:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.