I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So I have been wondering:
Should one use exp(log_softmax) or softmax as activation function for the output layer?

  • $\begingroup$ What is your loss function? Note that you can work through the algebra to combine cross entropy loss and log_softmax, or cross entropy loss and logits. That’s what makes for improved numerical stability: you aren’t round-tripping log and exp. Using exp(log(soft max)) discards the benefits of log(softmax) or logits. $\endgroup$
    – Sycorax
    Nov 27 '21 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.