# Proof for the efficiency of Softmax in multi-classification

I already search for this question but I can't find any convincing explanation so I want to ask it here. my problem is with softmax activation function and cross-entropy.why they can produce a better estimation for multi-classification problems? I search for it and people keep talking about the advantage of softmax like : Each value ranges between 0 and 1 or The sum of all values is always 1 and etc.

but this paper directly said: It is unclear why the log-softmax loss would perform better than other loss alternatives.

I want to know is there any proof for better performances of soft-max and cross-entropy function or just we conclude it by experimental.

• softmax is derived by the maxium entropy theory. Dec 11, 2022 at 10:44
• arxiv.org/pdf/1711.07758.pdf here is the proof Dec 11, 2022 at 11:02

The below answer is copy and paste from arxiv.org/pdf/1711.07758.pdf

Supposing the dataset has input $$X$$ and label $$Y$$,
the task is to find a good prediction of $$Y$$ using $$X$$.

The prediction $$\hat{Y}$$ needs to maximize the conditional entropy $$H(\hat{Y} \mid X)$$ while preserving the same distribution with data $$(X, Y)$$.

This is formulated as:

$$\begin{gathered} \min -H(\hat{Y} \mid X) \\ \text { s.t. } P(X, Y)=P(X, \hat{Y}), \sum_{\hat{Y}} P(\hat{Y} \mid X)=1 \end{gathered}$$

This optimization question can be solve by lagrangian multiplier method:

\begin{aligned} L = & \sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) \log (P(\hat{Y} \mid X))+\omega_0\left(1-\sum_{\hat{Y}} P(\hat{Y} \mid X)\right) \\ & +\sum_{X, Y} \omega_i(P(X, Y)-P(X) P(\hat{Y}=Y \mid X)) \end{aligned}

The above equation can be equivalently written with the original defined predicate function in (Berger et al.,
1996):

\begin{aligned} L = & \sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) \log (P(\hat{Y} \mid X))+\omega_0\left(1-\sum_{\hat{Y}} P(\hat{Y} \mid X)\right) \\ & +\sum_i \omega_i\left(\sum_{X, Y} P(X, Y) f_i(X, Y)-\sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) f_i(X, \hat{Y})\right) \end{aligned}

where $$f_i(X, Y)$$ is predicate function,
which equalizes 1 when $$(X, Y)$$ satisfies a certain status:

$$f_i(X, Y)=\left\{\begin{array}{lr} 1, & X=x_i, Y=y_i \\ 0, & \text { others } \end{array}\right.$$

The solution to the above problem is:

$$\begin{gathered} P_\omega(\hat{Y}=y \mid X=x)=\frac{1}{Z_\omega(x)} \exp \left(\sum_i \omega_i f_i(x, y)\right) \\ Z_\omega(x)=\sum_y \exp \left(\sum_i \omega_i f_i(x, y)\right) \end{gathered}$$

As far as I know, there is no proof published.

• This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. Aug 11, 2018 at 16:18