I am studying neural nets and softmax is showing up a lot.
I understood what softmax does: it quashes a set of real values to a probability distribution i.e. the new set can be interpreted as a probability distribution.
My question is:
While training neural nets is the temperature of softmax also a trainable parameter? i.e. is the softmax layer in Neural nets just a pre-defined quashing function or is it trained too?
Surely, machine learning people didn't invent softmax as I remember seeing this kind of function in physics classes as well, somewhere in Botlzmann theory of gases probably. What is the physical significance of softmax i.e. the exponentiation function? Why exponentiate? A square function or any function for that matter, can do the quashing job too, when normalized with the denominator. So, why exponentiate, what does it confer to the function.