8 x 1 x 8
Lets say there are 8 classes. The network has one hidden layer with only one neuron in it. Can the neuron learn to map the identity function between the 8 inputs and 8 outputs? Will gradient descent learn the weights of the identity? What is the difference between this network and a softmax classifier?
If you didn't have the 1-hidden layer, then the mapping between the input and the output would be a single-layer neural network. No, your network wouldn't learn to map the identity function between the 8 inputs and 8 outputs unless you work with Autoencoders. This network is indeed the softmax classifier except that we have 8 output classes. It is no longer a network of complex non-linear computations except that the non-linearity stems from the single neuron that we have.
