Difficulty picturing neural network with softmax activation

Question

I'm having a really hard time picturing a neural network with softmax activation function on its last layer.

For example, say if the last hidden layer has 100 nodes and the output layer had 10 nodes, then each of the output nodes would receive a sum of 100 inputs multiplied by their weights. The softmax activation function would then be applied to sumand and that would form the output.

However, the output is meant to represent the probability of a classification. But how can the probability ever sum up to 1 since the softmax is applied only to the inputs of each node.

Or is it applied to the whole output from the neural network. But if this is the case, then what activation function is applied to the input of each node.

Answer 1

The network outputs a $k$-dimensional vector for $k$ classes. Each one of the elements of that vector could just be the linear combination of the weights and inputs going into a particular node, or it could be the linera combination of the weights and inputs going into a particular node passed through an activation function, if you want (I have not heard of this second way because the softmax itself kind of serves as the activation function, so having another activation function right before it might not be a good thing---it would definitely make it a little harder to derive the gradients. It is ill-advised here). Either way, use the softmax to convert the entries of that vector to probabilities.