In the machine learning literature, to represent a probability distribution, the softmax function is often used. Is there a reason for this? Why isn't another function used?


From an optimization perspective it has some nice properties in terms of differentiability. For a lot of machine learning problems it's a good fit for 1-of-N classification.

From a deep learning perspective: One could also argue that in theory, using a deep network with a softmax classifier on top can represent any N-class probability function over the feature space as MLPs have the Universal Approximation property.

  • 1
    $\begingroup$ So the main reason for popularity of Softmax is it's nice differentiation properties which are helpful in Gradient Based learning setting. That's it, right? $\endgroup$ – SHASHANK GUPTA Jan 6 '16 at 18:42
  • $\begingroup$ Yeap, in my opinion anyway. Softmax is a simple with nice derivatives and is attractive for gradient based learning. Agree with everything you said. $\endgroup$ – Indie AI Jan 6 '16 at 18:46
  • $\begingroup$ You can think softmax as probability mass/density function of the function you are going to optimize. In my opinion, softmax is just a convenience way to model a probability mass/density function. $\endgroup$ – Charles Chow Dec 22 '18 at 21:55

Softmax is also a generalization of the logistic sigmoid function and therefore it carries the properties of the sigmoid such as ease of differentiation and being in the range 0-1. The output of a logistic sigmoid function is also between 0 and 1 and therefore naturally a suitable choice for representing probability. Its derivative is also exoressed in terms of its own output. However, if your function has a vector output you need to use the Softmax function to get the probability distribution over the output vector. There are some other advantages of using Softmax which Indie AI has mentioned, although it does not necessarily has anything to do with the Universal Approximation theory since Softmax is not a function only used for Neural Networks.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.