How deep is the connection between the softmax function in ML and the Boltzmann distribution in thermodynamics?

Question

The softmax function, commonly used in neural networks to convert real numbers into probabilities, is the same function as the Boltzmann distribution, the probability distribution over energies for en ensemble of particles in thermal equilibrium at a given temperature T in thermodynamics.

I can see some clear heuristical reasons why this is practical:

• No matter if input values are negative, softmax outputs positive values that sum to one.
• It's always differentiable, which is handy for backpropagation.
• It has a 'temperature' parameter controlling how lenient the network should be toward small values (when T is very large, all outcomes are equally likely, when very small, only the value with the largest input is selected).

Is the Boltzmann function only used as softmax for practical reasons, or is there a deeper connection to thermodynamics/statistical physics?

Answer 1

To my knowledge there is no deeper reason, apart from the fact that a lot of the people who took ANNs beyond the Perceptron stage were physicists.

Apart from the mentioned benefits, this particular choice has more advantages. As mentioned, it has a single parameter that determines the output behaviour. Which in turn can be optimized or tuned in itself.

In short, it is a very handy and well known function that achieves a kind of 'regularization', in the sense that even the largest input values are restricted.

Of course there are many other possible functions that fulfill the same requirements, but they are less well known in the world of physics. And most of the time, they are harder to use.

Answer 2

Yes. There is a connection, one that is articulated in the following paper:

Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One. Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, Kevin Swersky. ICLR 2020.

In particular, we can think of neural networks with softmax as inspired by an energy-based formulation, and in particular, by a Boltzmann distribution. Basically, we try to fit a Boltzmann distribution to the data, and then use that construct a classifier.

The paper suggests that we should think about neural networks as follows: we should think of them as a way of defining an energy-based model for estimating the joint probability $p(x,y)$, via the formula

$$p(x,y) = c \cdot e^{f(x)[y]},$$

where here $f(x)[y]$ represents the logit output for class $y$, when neural network $f$ is given input $x$, and $c$ is an appropriate normalizing constant. This is basically a way to parametrize the probability distribution $p(x,y)$ in a learnable way: the problem of learning a classifier is to learn a function $f$ so that $c \cdot e^{f(x)[y]}$ is a good approximation to the true joint distribution $p(x,y)$.

Notice that this formula means that we are approximating $p(x,y)$ by a Boltzmann distribution. In particular, the Boltzmann distribution would be $p(x,y) = c \cdot e^{-\epsilon_{x,y}/(kT)}$. Here we take the special case of $kT=1$. Moreover, we use the neural network to predict the appropriate $\epsilon_{x,y}$ values: in particular, we define $\epsilon_{x,y}$ to be the logit value produced by the neural network.

Given this definition, we can now use this to build a discriminative classifier. In particular, we can calculate

$$\begin{align*} p(y|x) &= {p(x,y) \over p(x)}\\ &={p(x,y) \over \sum_{y'} p(x,y')}\\ &={c \cdot e^{f(x)[y]} \over \sum_{y'} c \cdot e^{f(x)[y']}}\\ &={e^{f(x)[y]} \over \sum_{y'} e^{f(x)[y']}} \end{align*}$$

Looking closely, we see that the latter expression is exactly the softmax applied to the logits $f(x)$. Therefore, using this energy-based approach to fitting the distribution $p(x,y)$ yields exactly the same result as constructing a neural network that ends with a softmax layer. It's just two different ways to look at the same thing.

Hopefully this now makes clear the connection. See the paper above for more details and discussion.

Answer 3

the softmax function is also used in discrete choice modelling, it is same as the logit model, if u assume there is a utility function associated with each class, and the utility function equals to the output of neural network + an error term following the Gumbel distribution, the probability of belonging to a class equals to the softmax function with neural network as input. See: https://eml.berkeley.edu/reprints/mcfadden/zarembka.pdf

there is alternatives to the logit model, such as the probit model, where the error term is assumed to follow standard normal distribution, which is a better assumption. however, the likelihood would be intractable and is computational expensive to solve, therefore not commonly used in neural network

Answer 4

Here is an academic paper published in the Journal of Statistical Physics by various physicists at MIT: https://dspace.mit.edu/handle/1721.1/135715.

The paper discusses the relation of the softmax function to statistical physics. The authors claim that the relation between the Boltzmann distribution from thermodynamics and machine learning is deep.