# Approximating a proxy to the mutual information with Monte-Carlo sampling and Restricted Boltzmann machines

I first asked this in the physics community, to no avail, so now I'm here!

I've been reading through Koch-Janusz and Ringel's, "Mutual Information, Neural Networks, and the Renormalization Group" (check it out here). I'm currently trying to reimplement some results from their methodology and I'm getting stuck in the details. Specifically, I get that the dependence of their cost function on the parameters of their model seems to cancel out.

Below I give an outline of their Appendix A. You can also check out the original here on pages 7-8.

The authors describe a machine learning algorithm (based on restricted Boltzmann machines) that learns general renormalization group transformations for lattice systems.

This algorithm works by maximizing the real-space mutual information between a given block of visible spins and the hidden spins in the next coarse-grained layer. Intuitively, maximizing the real-space mutual information extracts the long-distance fluctuations (those that are physically relevant). Check out the picture below (again from 1).

# Recap of 1's Methodology

Consider a system $$\mathcal{X}=\{x_i,\dots,x_N\}$$ of classical Ising variables $$x_i \in \{0,1\}$$. We divide the system into a "visible" area, $$\mathcal{V}$$, an environment, $$\mathcal{E}$$, and an intermediate buffer, $$\mathcal{B}$$ that separates the two aforementioned parts. Since we care about the long-distance correlations, we ignore $$B$$ in calculating mutual information so as to avoid including short-distance fluctuations. We are given some data distribution $$P(\mathcal{X})=P(\mathcal{V},\mathcal{B},\mathcal{E})$$. $$P(\mathcal{V},\mathcal{E})$$ and $$P(\mathcal{V})$$ are marginalizations over $$P(\mathcal{X})$$, but since marginalizations are compuationally costly, we model these distributions with two standard (contrastive-divergence trained) RBMs, with parameters, $$\Theta$$ and $$\Psi$$, respectively. These distributions we call $$P_\Theta(\mathcal{V},\mathcal{E})$$ and $$P_\Psi(\mathcal{V})$$. Note that this notation is slightly different than the notation used in Koch-Janusz and Ringel.

We define the conditional probability distribution $$P_\Lambda(\mathcal{H}|\mathcal{V})$$ by another RBM parameterized by $$\Lambda$$: \begin{align}% P_\Lambda(\mathcal{H}|\mathcal{V})&=\frac{e^{-E_\Lambda(\mathcal{H}|\mathcal{V})}}{\sum_\mathcal{H}e^{-E_\Lambda(\mathcal{H}|\mathcal{V})}}\\ E_\Lambda(\mathcal{H}|\mathcal{V})&=- \left(\sum_{ij}v_i\lambda_i^jh_j+\sum_ia_iv_i+\sum_jb_jh_j \right). \end{align}%

The goal is to maximize the mutual information between $$\mathcal{E}$$ and $$\mathcal{H}$$, defined: $$$$I_\Lambda(\mathcal{H}:\mathcal{E})=\sum_{\mathcal{H},\mathcal{E}}P_\Lambda(\mathcal{H},\mathcal{E})\log\left(\frac{P_\Lambda(\mathcal{H},\mathcal{E})}{P_\Lambda(\mathcal{H})P(\mathcal{E})}\right).$$$$ Because $$P(\mathcal{E})$$ has no dependence on the parameters, the authors end up maximizing a proxy to the mutual information: $$$$A_\Lambda(\mathcal{H}:\mathcal{E})=\sum_{\mathcal{H},\mathcal{E}}P_\Lambda(\mathcal{H},\mathcal{E})\log\left(\frac{P_\Lambda(\mathcal{H},\mathcal{E})}{P_\Lambda(\mathcal{H})}\right).$$$$

Here, I skip a few steps (you can find them here), but the authors end up with the following equations: $$$$A_\Lambda(\mathcal{H}:\mathcal{E})\approx\sum_{\mathcal{H},\mathcal{E}}P_\Lambda(\mathcal{H},\mathcal{E}) \langle -\Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\rangle_\mathcal{H},$$$$ which they maximize with stochastic gradient descent. Further definitions: $$$$\langle \Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\rangle_\mathcal{H} \equiv \frac{\sum_\mathcal{V} \left(\Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\right) e^{-E_{\Lambda,\Psi}(\mathcal{V},\mathcal{H})}}{\sum_{\mathcal{V}'}e^{-E_{\Lambda,\Psi}(\mathcal{V}',\mathcal{H})}},$$$$ and \begin{align} E_{\Lambda,\Theta}(\mathcal{V},\mathcal{E},\mathcal{H})&= E_\Lambda(\mathcal{V},\mathcal{H})+E_\Theta(\mathcal{V},\mathcal{E})+\sum_j\log[1+\exp(\sum_iv_j\lambda_i^j+b_j)]\\ E_{\Lambda,\Psi}(\mathcal{V},\mathcal{H})&= E_\Lambda(\mathcal{V},\mathcal{H})+E_\Psi(\mathcal{V})+\sum_j\log[1+\exp(\sum_iv_j\lambda_i^j+b_j)]\\ \Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})&=E_{\Lambda,\Theta}(\mathcal{V},\mathcal{E},\mathcal{H}) - E_{\Lambda,\Psi}(\mathcal{V},\mathcal{H}) \end{align}

# The Question

Now, the authors perform two sets of Monte Carlo samples.

1) They approximate $$\langle \Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})\rangle_\mathcal{H}$$ with a Monte-Carlo average over samples of $$\mathcal{V}$$ (holding $$\mathcal{H}$$ and $$\mathcal{E}$$ fixed).

2) Using $$P_\Lambda(\mathcal{H},\mathcal{E})=\sum_{\mathcal{V}'}P_\Lambda(\mathcal{H}|\mathcal{V}')P(\mathcal{V}',\mathcal{E})$$, they replace the sum over $$\mathcal{H}$$ and $$\mathcal{E}$$ in $$A_\Lambda(\mathcal{H}:\mathcal{E})$$ with a Monte-Carlo average over samples of $$(\mathcal{V}',\mathcal{E})$$. Then, $$\mathcal{H}=\mathcal{H}(\mathcal{V}')$$ as given by the $$\Lambda$$-RBM.

If you're still here, thanks for sticking through it with me. My question is, where does the $$\Lambda$$ dependence show up? If you plug in the definition of $$\Delta E$$, you get a bunch of cancellations:

$$$$\Delta E_{\Lambda,\Theta,\Psi}(\mathcal{V},\mathcal{E},\mathcal{H})=E_\Theta(\mathcal{V},\mathcal{E})-E_\Psi(\mathcal{V})$$$$

This obviously has no dependence on $$\Lambda$$. I've been stuck here for a while so any help would be appreciated.For the original go to page 7 here. Thank you!