0
$\begingroup$

I'm having a really hard time picturing a neural network with softmax activation function on its last layer. Sorry for dumb question.

For example, say if the last hidden layer has 100 nodes and the output layer had 10 nodes, then each of the output nodes would receive a sum of 100 inputs multiplied by their weights. The softmax activation function would then be applied to sumand and that would form the output.

However, the output is meant to represent the probability of a classification. But how can the probability ever sum up to 1 since the softmax is applied only to the inputs of each node.

Or is it applied to the whole output from the neural network. But if this is the case, then what activation function is applied to the input of each node.

$\endgroup$
1
$\begingroup$

The network outputs a $k$-dimensional vector for $k$ classes. Each one of the elements of that vector could just be the linear combination of the weights and inputs going into a particular node, or it could be the linera combination of the weights and inputs going into a particular node passed through an activation function, if you want (I have not heard of this second way because the softmax itself kind of serves as the activation function, so having another activation function right before it might not be a good thing---it would definitely make it a little harder to derive the gradients. It is ill-advised here). Either way, use the softmax to convert the entries of that vector to probabilities.

$\endgroup$
  • $\begingroup$ So with keras, for example, when you build a sequential neural network the last thing you add is a dense layer, in the examples I've seen it always sets the activation to be softmax. Does it know that to only apply the softmax function to the output of the whole network. So the API is implicit in this matter? (Sorry for making the question explicitly about keras) $\endgroup$ – John Paul Aug 5 '17 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.