8 x 1 x 8
Lets say there are 8 classes. The network has one hidden layer with only one neuron in it. Can the neuron learn to map the identity function between the 8 inputs and 8 outputs? Will gradient descent learn the weights of the identity? What is the difference between this network and a softmax classifier?