4
$\begingroup$

Probably a dumb question, obviously not seen in practice, but after reviewing linear algebra, I can't pinpoint the misunderstanding in this logic:

  • We can represent input data to the network as a vector
  • The vector is passed through weights in layers, which can be seen as transforms to the vector, which is passed on to the next transform
  • In linear algebra, we can compose multiple transforms like a shear and rotation into an equivalent single transform
  • So in a simple case where subsequent layers are of the same size or other ideal conditions, we should be able to compose multiple layers into a single layer, since really they're just transforms

My only guess is that it's because we have to use nonlinear activation functions to model nonlinear distributions, and this composition equivalence doesn't apply to nonlinear transforms?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Your guess is right. Using only linear activation functions, you could indeed reduce a however deep NN into a single layer.

However to be able to use backpropagation you need an activation function which does not have a constant gradient. Therefore in practice, linear activation functions are not used and NN are not reducible to a single layer.

Read more here

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Thanks, understood. That prompts an adjustment to my intuition: - Instead of seeing the NN as creating a unique high dimensional space to distribute the data for best inference of patterns, really each layer creates it's own unique high dimensional space, each independent of the other? $\endgroup$
    – user
    Commented Aug 26, 2020 at 17:53
  • 1
    $\begingroup$ I would not say that the layer level embedding is independent of the other layers. During backpropagation the error gets propagated backwards thus each layer is influenced/adapted in accordance to the error from the following layer. $\endgroup$
    – Nikolas Rieble
    Commented Aug 26, 2020 at 19:06
  • $\begingroup$ Backpropagation can still be done with a constant activation function, no? It doesn't yield a nonlinear function, but it can still be done. $\endgroup$
    – Simon Alford
    Commented Aug 26, 2020 at 21:12
  • 1
    $\begingroup$ Right, backpropagation can still be done but it does not make sense, since the derivative of a linear function is a constant and the weight adaption thus independent of the input. $\endgroup$
    – Nikolas Rieble
    Commented Aug 27, 2020 at 5:36
  • 1
    $\begingroup$ You could probably say that each layer is an embedding of the previous one. $\endgroup$
    – Nikolas Rieble
    Commented Aug 28, 2020 at 6:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.