I am new to Deep Learning, and I was going through some lecture notes I found online. It said that a feed-forward neural network with $n$ hidden layers and only linear activation functions is equivalent to a linearly activated neural network with no hidden layers. I am having some trouble proving this assertion.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This follows from the fact that matrix multiplications by matrix A and then by matrix B is equivalent to multiplication by matrix C = AB. Linear activations are matrix multiplications (by the weight matrix). So several linear layers are equivalent to one linear layer with parameters given by multiplication of the layers weight matrices. (The same argument holds also for affine transformations, if you consider the bias parameters of the linear layer.)
