I know that if the input space is not linearly separable, we can map it to a higher dimensional space with a hidden layer. I am learning about handwritten digit classification, but I am seeing that most of the neural networks have an input space like 400 (20x20 pixel), but hidden layer has lower units, less than 400. Why? Is not 400 the dimension of the input space? What am I missing?
The nonlinearity of neural nets comes from the activation function(s) used, not so much from embedding in higher dimensional space and cutting there.
Most modern image-classification nets though are convolutional, and the first layer will have many more nodes: each filter has (almost, depending on padding) as many individual nodes as the input has pixels, and you will have several filters.