I know that if the input space is not linearly separable, we can map it to a higher dimensional space with a hidden layer. I am learning about handwritten digit classification, but I am seeing that most of the neural networks have an input space like 400 (20x20 pixel), but hidden layer has lower units, less than 400. Why? Is not 400 the dimension of the input space? What am I missing?
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The nonlinearity of neural nets comes from the activation function(s) used, not so much from embedding in higher dimensional space and cutting there.
Most modern image-classification nets though are convolutional, and the first layer will have many more nodes: each filter has (almost, depending on padding) as many individual nodes as the input has pixels, and you will have several filters.
answered Feb 1, 2019 at 16:45
-
$\begingroup$ You confuse number of outputs of CNNs with nodes. In CNNs kernel weights correspond to nodes. $\endgroup$ Feb 1, 2019 at 16:50
-
$\begingroup$ @JakubBartczuk, I chose the word "node" thinking it didn't have a specific meaning other than the graph-theoretic one. In the OP's sense, we do get a higher-dimensional output (though the outputs from one filter are determined using a common weight matrix). Should I have said just "output" instead of "node"? Anyway, that's not enough in the sense of Cover's Theorem, because it requires a nonlinear kernel trick. (Otherwise, regardless of dimension, composing linear functions can't give you anything new like separation.) So maybe I should just remove the second paragraph entirely. $\endgroup$ Feb 1, 2019 at 18:42