# Relationship between variational inference and sampling in a Boltmzann-machine-like network

In this paper concerning a Boltzmann-machine-like network and its variational mean field approximation, the authors write

In the stochastic system as well as the deterministic system, units evolve to minimize the free energy, $$F= \langle E\rangle - HT$$...

where the expectation is taken with respect to the fully-factoring joint distribution on the units and $$H$$ is that distribution's entropy. By the "deterministic system" they mean the mean field approximation of their Boltzmann machine. I assume by the stochastic system" they mean the version of their network in which configurations of units are determined by random sampling, in the manner that configurations of a traditional Boltzmann machine are acquired by Gibbs sampling.

I understand variational approximation as a way of avoiding a costly sampling approach, but I don't understand why the authors claim that the sampling approach itself minimizes" variational free energy. A Gibbs sampling chain should explore configurations of the model with both low energy and large volume, since, intuitively, those are regions with high probability. Otherwise, we'd expect sampling to always give us the mode of the distribution, which is decreasingly likely in high dimensions.

Is there another interpretation to the author's statement? And is there, in general, anything we can say about $$$$energy" of the states visited by the states of a Markov chain in the stochastic system? Any literature on this would also be appreciated.

• “A Gibbs sampling chain should explore configurations of the model with both low energy and large volume” doesn't the free energy relate to that? It minimizes $E$ relating to low energy, and maximizes $HT$ relating to large volume/entropy. Commented Feb 2 at 7:37
• The link doesn't work. Commented Feb 2 at 7:42

One possible interpretation is the following: Let $$p$$ be the Boltzmann distribution distribution (i.e. $$p(z)\propto\exp(-E(z))$$), and let $$\mu_t$$ be the distribution over the units after $$t$$ iterations of MCMC, e.g. Gibbs sampling, targeting $$p$$. One can then show under fairly general circumstances that the quantity

$$D_t = KL (\mu_t | p)$$

decreases towards $$0$$ as $$t$$ grows.

Noting the equivalence between the KL divergence and the free energy,

$$KL(q|p) = \mathbf{E}_{q(z)} [ E (z) ] - H (q)$$

where $$H$$ is the entropy, one can thus argue that the stochastic system is also minimising the free energy, through its evolution.

• So this interpretation, that the approach minimized free energy, only works when the distribution $p$ to be approximated is in the exponential family, when there is an interpretation available for 'energy'. Commented Feb 2 at 7:32
• @SextusEmpiricus Sort of, yeah. In various cases, you can safely synthesise the exponential family by picking a dominating measure $\nu$, defining the energy to be $E = - \log \left(\frac{\mathrm{d} p}{\mathrm{d} \nu} \right)$, and proceeding from there. It's perhaps a bit artificial, but also reasonably common in some areas.
– πr8
Commented Feb 3 at 10:45