I know that if the input space is not linearly separable, we can map it to a higher dimensional space with a hidden layer. I am learning about handwritten digit classification, but I am seeing that most of the neural networks have an input space like 400 (20x20 pixel), but hidden layer has lower units, less than 400. Why? Is not 400 the dimension of the input space? What am I missing?

  • $\begingroup$ Are you asking why the hidden layer has lower number of units than the input layers? This is not always the case, anyway. $\endgroup$ – nbro Feb 1 at 16:50

The nonlinearity of neural nets comes from the activation function(s) used, not so much from embedding in higher dimensional space and cutting there.

Most modern image-classification nets though are convolutional, and the first layer will have many more nodes: each filter has (almost, depending on padding) as many individual nodes as the input has pixels, and you will have several filters.

  • $\begingroup$ You confuse number of outputs of CNNs with nodes. In CNNs kernel weights correspond to nodes. $\endgroup$ – Jakub Bartczuk Feb 1 at 16:50
  • $\begingroup$ @JakubBartczuk, I chose the word "node" thinking it didn't have a specific meaning other than the graph-theoretic one. In the OP's sense, we do get a higher-dimensional output (though the outputs from one filter are determined using a common weight matrix). Should I have said just "output" instead of "node"? Anyway, that's not enough in the sense of Cover's Theorem, because it requires a nonlinear kernel trick. (Otherwise, regardless of dimension, composing linear functions can't give you anything new like separation.) So maybe I should just remove the second paragraph entirely. $\endgroup$ – Ben Reiniger Feb 1 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.