Significance of Softmax

Question

I am studying neural nets and softmax is showing up a lot.

I understood what softmax does: it quashes a set of real values to a probability distribution i.e. the new set can be interpreted as a probability distribution.

My question is:

1. While training neural nets is the temperature of softmax also a trainable parameter? i.e. is the softmax layer in Neural nets just a pre-defined quashing function or is it trained too?

2. Surely, machine learning people didn't invent softmax as I remember seeing this kind of function in physics classes as well, somewhere in Botlzmann theory of gases probably. What is the physical significance of softmax i.e. the exponentiation function? Why exponentiate? A square function or any function for that matter, can do the quashing job too, when normalized with the denominator. So, why exponentiate, what does it confer to the function.

Cheers :)

Answer 1

Let's look at the definitions:

$\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}} $

Softmax with temperature $T$:

$\sigma (\mathbf {z} )_{j}={\frac {e^{ \frac{z_{j}}{T}}}{\sum _{k=1}^{K}e^{\frac{z_{k}}{T}}}} $

In context of machine learning softmax is used, as you remarked, to define probability distributions ($\mathbf{x} \in \mathbb{R}^D$):

$ P(y=j\mid \mathbf {x} )={\frac {e^{\mathbf {x} ^{\mathsf {T}}\mathbf {w} _{j}}}{\sum _{k=1}^{K}e^{\mathbf {x} ^{\mathsf {T}}\mathbf {w} _{k}}}} $

Where $\mathbf{w_j}$'s are trainable parameters.

---

1. Temperature is just a multiplicative constant, so you can treat it as degree of freedom of $\mathbf{w_j}$'s (if $W = (\mathbf{w_j})_{j < M}$ is weights matrix, then it has $DM$ parameters, so equivalently if you treat $T$ as free parameter $\frac{W}{T}$ will have $DM - 1$ degrees of freedom).

2. I'm not familiar with physical interpretation, but softmax is related to maximum entropy principle, see A Brief Maxent Tutorial