Proof for the efficiency of Softmax in multi-classification I already search for this question but I can't find any convincing explanation so I want to ask it here. my problem is with softmax activation function and cross-entropy.why they can produce a better estimation for multi-classification problems? I search for it and people keep talking about the advantage of softmax like :
Each value ranges between 0 and 1 or
The sum of all values is always 1
and etc.
but this paper directly said: It is unclear why the log-softmax loss would perform better than other loss alternatives.
I want to know is there any proof for better performances of soft-max and cross-entropy function or just we conclude it by experimental.
 A: The below answer is copy and paste from arxiv.org/pdf/1711.07758.pdf
Supposing the dataset has input $X$ and label $Y$,
the task is to find a good prediction of $Y$ using $X$.
The prediction $\hat{Y}$ needs to maximize the conditional entropy $H(\hat{Y} \mid X)$ while preserving the same distribution with data $(X, Y)$.
This is formulated as:

$$
\begin{gathered}
\min -H(\hat{Y} \mid X) \\
\text { s.t. } P(X, Y)=P(X, \hat{Y}), \sum_{\hat{Y}} P(\hat{Y} \mid X)=1
\end{gathered}
$$

This optimization question can be solve by lagrangian multiplier method:

$$
\begin{aligned}
L = & \sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) \log (P(\hat{Y} \mid X))+\omega_0\left(1-\sum_{\hat{Y}} P(\hat{Y} \mid X)\right) \\
& +\sum_{X, Y} \omega_i(P(X, Y)-P(X) P(\hat{Y}=Y \mid X))
\end{aligned}
$$

The above equation can be equivalently written with the original defined predicate function in (Berger et al.,
1996):

$$
\begin{aligned}
L = & \sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) \log (P(\hat{Y} \mid X))+\omega_0\left(1-\sum_{\hat{Y}} P(\hat{Y} \mid X)\right) \\
& +\sum_i \omega_i\left(\sum_{X, Y} P(X, Y) f_i(X, Y)-\sum_{X, \hat{Y}} P(X) P(\hat{Y} \mid X) f_i(X, \hat{Y})\right)
\end{aligned}
$$

where $f_i(X, Y)$ is predicate function,
which equalizes 1 when $(X, Y)$ satisfies a certain status:

$$
f_i(X, Y)=\left\{\begin{array}{lr}
1, & X=x_i, Y=y_i \\
0, & \text { others }
\end{array}\right.
$$

The solution to the above problem is:

$$
\begin{gathered}
P_\omega(\hat{Y}=y \mid X=x)=\frac{1}{Z_\omega(x)} \exp \left(\sum_i \omega_i f_i(x, y)\right) \\
Z_\omega(x)=\sum_y \exp \left(\sum_i \omega_i f_i(x, y)\right)
\end{gathered}
$$
A: As far as I know, there is no proof published.
