1
$\begingroup$

I don't understand how to use the cross-entropy loss function and softmax activation function for multi-class classification in NNs.

Say I have a 2 layer NN (1 output layer and 1 hidden layer). For the hidden-layer, I apply sigmoid function to the pre-activation from the input layer. Now, I am not sure when I feed the pre-activation from hidden layer to output layer what activation function should I use? Should I first employ the sigmoid function to output-layer's pre-activation, then the softmax function? But that means using 2 activation functions for the same layer.

$\endgroup$
1
$\begingroup$

The hidden layer can use sigmoid or any other activation function.

To perform multi-class classification, it's the architecture of the output layer that is the most important. It must have the same number of units (neurons) as the number of categories. And it must use the softmax activation function. This will ensure that the output will be one-hot encoded (at most one unit activates). The index of the unit that activates is the index of the inferred class.

Hope this makes sense.

$\endgroup$
  • $\begingroup$ In output layer, you apply softmax on w^Tx or sigmoid(w^Tx)? Where I get confused is activation (softmax here) includes an interaction btw neurons in the same layer. $\endgroup$ – Abhishek Bhatia Sep 14 '16 at 12:38
  • $\begingroup$ Hi again. You apply softmax on w^Tx. No more sigmoid. That would be applying two activation functions on this layer, and you don't want that. $\endgroup$ – ociule Sep 14 '16 at 14:39
1
$\begingroup$

Seeing softmax as an activation function always seemed a bit weird to me, for the reason that you highlight. It's usually used on w^Tx (+bias). The softmax is a normalized sigmoid, so it makes little sense to do softmax(sigmoid(pre-activation)).

$\endgroup$
0
$\begingroup$

I think you could choose to use or not use. It sometimes matters while in most cases doesn't. I normally don't do the "pre-activation" but I saw somebody do it and get better performance on specific tasks. I think you try both and see which one get better performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.