Are neural networks equivalent to linear regression if the activation function is linear (g(x) = x), and back propagation is basically just SGD for a least squares problem? Or is that only true for single layer neural networks?
I'm very new to neural networks and I basically have very little idea what's going on, so any intuition anyone could give would be appreciated. Thanks!
Edit: I'm going to write some math to establish some things.
Assume that I have a simple network with two hidden layers, and each hidden layer has K units. My input has M features, and my output is one of P classes.
let's just consider 1 sample for now.
Input vector: $x$, of length M
Hidden layer 1: $a_1 = \phi(W_1 [x;1])$ where $W_1$ is a K x (M+1) matrix and $a_1$ is the first activation vector of length K. All the activation functions are the same, and are denoted $\phi(x)$.
Hidden layers 2 ... K: $a_k = \phi(W_k [a_{k-1};1])$ where $W_k$ is a K x (K+1) matrix and $a_k$ is the $k$th activation vector of length K.
Output: $y = \phi(W_{K+1}[a_K;1])$ where $W_{K+1}$ is a P x (K+1) matrix and $y$ is the output of length P.
So, as one involved function, it's
$y = \phi(W_{K+1}[\phi(W_K [\phi(W_{K-1} ... [\phi(W_1 [x;1]);1] ... ; 1] ;1])$.
Clearly this is a very nonlinear and nonconvex function of the $W_k$s, as pointed out in the comments, so indeed it cannot be like linear regression.
