20
$\begingroup$

Applying the softmax function on a vector will produce "probabilities" and values between $0$ and $1$.

But we can also divide each value by the sum of the vector and that will produce probabilities and values between $0$ and $1$.

I read the answer on here but it says that the reason is because it's differentiable, although Both functions are differentiable.

$\endgroup$
  • 1
    $\begingroup$ I think its better if you first look at logistic regression. your 'goal' is to monotonically transform $(-\infty, \infty)$ to (0,1). This is what the logistic function does. Note that any cumulative (probability) distribution function on the real line also works - see probit regression which uses the normal distribution function. $\endgroup$ – seanv507 Jul 30 at 20:09
36
$\begingroup$

The function you propose has a singularity whenever the sum of the elements is zero.

Suppose your vector is $[-1, \frac{1}{3}, \frac{2}{3}]$. This vector has a sum of 0, so division is not defined. The function is not differentiable here.

Additionally, if one or more of the elements of the vector is negative but the sum is nonzero, your result is not a probability.

Suppose your vector is $[-1, 0, 2]$. This has a sum of 1, so applying your function results in $[-1, 0, 2]$, which is not a probability vector because it has negative elements, and elements exceeding 1.

Taking a wider view, we can motivate the specific form of the softmax function from the perspective of extending binary logistic regression to the case of three or more categorical outcomes.

Doing things like taking absolute values or squares, as suggested in comments, means that $-x$ and $x$ have the same predicted probability; this means the model is not identified. By contrast, $\exp(x)$ is monotonic and positive for all real $x$, so the softmax result is (1) a probability vector and (2) the multinomial logistic model is identified.

$\endgroup$
  • $\begingroup$ Thank you so much. We can solve both issues by dividing by the sum of absolute values, Right? $\endgroup$ – floyd Jul 30 at 1:19
  • 2
    $\begingroup$ No. What happens if you sum the absolute values of both of my examples and then divide by that sum? $\endgroup$ – Reinstate Monica Jul 30 at 1:20
  • $\begingroup$ really thank you. I get it now. but we can solve this issue by taking the absolute value of the numerator or maybe calculating $x_i^2/sum(X^2)$ for each value in the vector. I am not trying to be stubborn, I just find it weird that people invented a complex function although there are simpler ones to calculate probabilities. I don't know a lot of math so maybe there are other mathematical properties $\endgroup$ – floyd Jul 30 at 1:34
  • 8
    $\begingroup$ Your proposal still fails for $[0,0,0]$. Additional reasons for the softmax function relate to its properties as a generalization of binary logistic regression to the case of multiple outcomes. We have a number of threads about this such as stats.stackexchange.com/questions/349418/… $\endgroup$ – Reinstate Monica Jul 30 at 1:52
  • 8
    $\begingroup$ In addition to Sycorax's point, applying $x_i^2 / \sum_j x_j^2$ or $|x_i| / \sum_j |x_j|$ does not have the desired property that reducing a vector element will always reduce its probability contribution. Reducing negative elements would increase their contribution. $\exp(x)$ has the nice property that its output is positive for all real inputs, and is monotonic on the whole real line. $\endgroup$ – Bridgeburners Jul 30 at 13:21
4
$\begingroup$

Softmax has two components:

  1. Transform the components to e^x. This allows the neural network to work with logarithmic probabilities, instead of ordinary probabilities. This turns the common operation of multiplying probabilities into addition, which is far more natural for the linear algebra based structure of neural networks.

  2. Normalize their sum to 1, since that's the total probability we need.

One important consequence of this is that bayes' theorem is very natural to such a network, since it's just multiplication of probabilities normalized by the denominator.

The trivial case of a single layer network with softmax activation is equivalent to logistic regression.

The special case of two component softmax is equivalent to sigmoid activation, which is thus popular when there are only two classes. In multi class classification softmax is used if the classes are mutually exclusive and component-wise sigmoid is used if they are independent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.