I already search for this question but I can't find any convincing explanation so I want to ask it here. my problem is with softmax activation function and cross-entropy.why they can produce a better estimation for multi-classification problems? I search for it and people keep talking about the advantage of softmax like : Each value ranges between 0 and 1 or The sum of all values is always 1 and etc.

but this paper directly said: It is unclear why the log-softmax loss would perform better than other loss alternatives.

I want to know is there any proof for better performances of soft-max and cross-entropy function or just we conclude it by experimental.


2 Answers 2


The below answer is copy and paste from arxiv.org/pdf/1711.07758.pdf

Supposing the dataset has input $X$ and label $Y$,
the task is to find a good prediction of $Y$ using $X$.

The prediction $\hat{Y}$ needs to maximize the conditional entropy $H(\hat{Y} \mid X)$ while preserving the same distribution with data $(X, Y)$.

This is formulated as:

$$ \begin{gathered} \min -H(\hat{Y} \mid X) \\ \text { s.t. } P(X, Y)=P(X, \hat{Y}), \sum_{\hat{Y}} P(\hat{Y} \mid X)=1 \end{gathered} $$

This optimization question can be solve by lagrangian multiplier method:

$$ \begin{aligned} L = & \sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) \log (P(\hat{Y} \mid X))+\omega_0\left(1-\sum_{\hat{Y}} P(\hat{Y} \mid X)\right) \\ & +\sum_{X, Y} \omega_i(P(X, Y)-P(X) P(\hat{Y}=Y \mid X)) \end{aligned} $$

The above equation can be equivalently written with the original defined predicate function in (Berger et al.,

$$ \begin{aligned} L = & \sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) \log (P(\hat{Y} \mid X))+\omega_0\left(1-\sum_{\hat{Y}} P(\hat{Y} \mid X)\right) \\ & +\sum_i \omega_i\left(\sum_{X, Y} P(X, Y) f_i(X, Y)-\sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) f_i(X, \hat{Y})\right) \end{aligned} $$

where $f_i(X, Y)$ is predicate function,
which equalizes 1 when $(X, Y)$ satisfies a certain status:

$$ f_i(X, Y)=\left\{\begin{array}{lr} 1, & X=x_i, Y=y_i \\ 0, & \text { others } \end{array}\right. $$

The solution to the above problem is:

$$ \begin{gathered} P_\omega(\hat{Y}=y \mid X=x)=\frac{1}{Z_\omega(x)} \exp \left(\sum_i \omega_i f_i(x, y)\right) \\ Z_\omega(x)=\sum_y \exp \left(\sum_i \omega_i f_i(x, y)\right) \end{gathered} $$


As far as I know, there is no proof published.

  • $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. $\endgroup$ Commented Aug 11, 2018 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.