Softmax in multi-class in deep NNs

Question

I don't understand how to use the cross-entropy loss function and softmax activation function for multi-class classification in NNs.

Say I have a 2 layer NN (1 output layer and 1 hidden layer). For the hidden-layer, I apply sigmoid function to the pre-activation from the input layer. Now, I am not sure when I feed the pre-activation from hidden layer to output layer what activation function should I use? Should I first employ the sigmoid function to output-layer's pre-activation, then the softmax function? But that means using 2 activation functions for the same layer.

Answer 1

The hidden layer can use sigmoid or any other activation function.

To perform multi-class classification, it's the architecture of the output layer that is the most important. It must have the same number of units (neurons) as the number of categories. And it must use the softmax activation function. This will ensure that the output will be one-hot encoded (at most one unit activates). The index of the unit that activates is the index of the inferred class.

Hope this makes sense.

Answer 2

Seeing softmax as an activation function always seemed a bit weird to me, for the reason that you highlight. It's usually used on w^Tx (+bias). The softmax is a normalized sigmoid, so it makes little sense to do softmax(sigmoid(pre-activation)).

Answer 3

I think you could choose to use or not use. It sometimes matters while in most cases doesn't. I normally don't do the "pre-activation" but I saw somebody do it and get better performance on specific tasks. I think you try both and see which one get better performance.