I'm recently working on CNN and I want to know what is the function of temperature in the softmax formula? and why should we use high temperatures to see a softer norm in probability distribution?
The formula can be seen below:
$$\large P_i=\frac{e^{\frac{y_i}T}}{\sum_{k=1}^n e^{\frac{y_k}T}}$$
The temperature is a way to control the entropy of a distribution, while preserving the relative ranks of each event.
If two events $i$ and $j$ have probabilities $p_i$ and $p_j$ in your softmax, then adjusting the temperature preserves this relationship, as long as the temperature is finite:
$$p_i > p_j \Longleftrightarrow p'_i > p'_j$$
Heating a distribution increases the entropy, bringing it closer to a uniform distribution. (Try it for yourself: construct a simple distribution like $\mathbf{y}=(3, 4, 5)$, then divide all $y_i$ values by $T=1000000$ and see how the distribution changes.)
Cooling it decreases the entropy, accentuating the common events.
I’ll put that another way. It’s common to talk about the inverse temperature $\beta=1/T$. If $\beta = 0$, then you've attained a uniform distribution. As $\beta \to \infty$, you reach a trivial distribution with all mass concentrated on the highest-probability class. This is why softmax is considered a soft relaxation of argmax.
Temperature will modify the output distribution of the mapping.
For example:
low temperature softmax probs : [0.01,0.01,0.98]
high temperature softmax probs : [0.2,0.2,0.6]
Temperature is a bias against the mapping. Adding noise to the output. The higher the temp, the less it's going to resemble the input distribution.
Think of it vaguely as "blurring" your output.
Increasing T above 1 tends to spread the probability among the inputs, while decreasing T below 1 tends to concentrate all of the probability on the most likely class.
In the limit that T reaches infinity (or even hundreds, practically speaking), all classes will have the same value. In the limit that T reaches zero (or even 0.001), only the maximum value will be selected.
Increasing temperature is a simple way to correct an over-confident network whose maximum output (going into softmax) is too far away from the next closest output.
The 1/T temperature adjustment effectively scales the inputs to the softmax. A T larger than 1 brings the inputs closer together, and a T smaller than 1 drives inputs farther apart. Since Softmax splits the output probability among the inputs that are closer together, driving the inputs closer together spreads the probability among the inputs.
To answer this question in detail, it may be helpful to think about how softmax works without temperature.
Consider a sofmax layer in which only two inputs are not -infinity, and have values 2 and 4.
Then, going through the softmax layer, the smaller element will have value
$$\large P_i=\frac{e^{y_i}}{\sum_{k=1}^n e^{y_k}}=\frac{e^2}{e^2+e^4}=\frac{e^4}{e^4}\frac{e^{-2}}{e^{-2}+1} \approx e^-2 \approx \frac{1}{9}$$
How does this change if the inputs are instead 12 and 14, both shifted the same distance to the right?
$$\large P_i=\frac{e^{y_i}}{\sum_{k=1}^n e^{y_k}}=\frac{e^{12}}{e^{12}+e^{14}}=\frac{e^{14}}{e^{14}}\frac{e^{-2}}{e^{-2}+1} \approx e^{-2} \approx \frac{1}{9}$$
It makes no difference at all -- adding a constant to all the inputs results in a constant multiplier after the exponent that can be pulled out.
Now, how does this change if we increase the distance between the 2 and the 4, e.g. to 2 to 6:
$$\large P_i=\frac{e^{y_i}}{\sum_{k=1}^n e^{y_k}}=\frac{e^2}{e^2+e^6}=\frac{e^6}{e^6}\frac{e^{-4}}{e^{-4}+1} \approx e^{-4} \approx \frac{1}{81}$$
Changing the distance between the maximum input and those close below it has a major input on the output -- the closer the second-maximum value is to the maximum value, the more the two values will be blended in the output.
The "softness" of softmax is controlled by how far apart the maximal inputs are. Once they are in the tens or 100's apart, it becomes a rather hard-max instead of a softmax.
What the temperature field allows us to do is to control the distance between the values entering the softmax by directly scaling the entire axis.
Suppose we wanted to soften the most recent example above and bring the 2 and 6 closer together. We could use a temperature of $T=2$:
$$\large P_i=\frac{e^{y_i}}{\sum_{k=1}^n e^{{y_k/T}}}=\frac{e^{2/2}}{e^{2/2}+e^{6/2}}=\frac{e^{1}}{e^{1}+e^{3}}=\frac{e^3}{e^3}\frac{e^{-2}}{e^{-2}+1} \approx e^{-2} \approx \frac{1}{9}$$
Here, the temperature of $T = 2$ cuts the distance between the 2 and 6 in half, softening the transition from one to another.
We can also use T values less than 1 to INCREASE the distance between the values entering softmax, and in the limit as T->0, we max the maximum value be infinitely far from the rest as it enters the exponent function. In code, this must be handled as a special case that finds the maximum output instead of selecting an output probabilistically.
So the legal values of T range from 0 (pick the maximum) through infinity (where all values would have equal probabilities because they are all 0 as they enter softmax).
I decide to add this answer because temperature in Softmax is a case of wonder:
From How does entropy depend on location and scale?, we would have:
The (differential) entropy of $z=y\beta$ is the entropy of X plus $\log(σ)$.
So, as $\beta \to 0$, the differential entropy of $z$ tends to $-\infty$.
At the same time, as $z$ tends toward a continuous variant of degenerate distribution, the Shannon entropy of the projection $z_i$ now tends toward $\frac{1}{n}$.
So while high temperature increase the Shannon entropy for $z_i$, it actually decreases the differential entropy of the complete embedding vector $z$.
