How to get the derivation for mean-field approximation?

Question

In page 487/1266 of [probabilistic graphic model: principal and technique] book, the authors give the following Lagrange equation:

$$L_i[Q]=\sum_{\phi \in \Phi} \mathbf{E}_{\mathbf{U}_\phi \sim Q}[\ln \phi] + \mathbf{H}_Q (X_i) + \lambda (\sum_{x_i} Q(x_i) - 1)$$

And in page 42/86 of [http://web.eng.tau.ac.il/deep_learn/wp-content/uploads/2017/01/Semantic-Segmentation.pdf], the authors derived the following Lagrange equation for solving a maximum problem for KL divergence using mean-field approximation:

$$\mathbf{D}(Q\|P) = \sum_{\mathbf{x}} Q(\mathbf{x}) \log \frac{Q(\mathbf{x})}{P(\mathbf{x})}=\mathbf{E}_{U\sim Q}[E(\mathbf{U})]+\sum \mathbf{E}_{U_i \sim Q_i }[\log Q_i(U_i)]+\log Z + \lambda (\sum_{x_i} Q(x_i) - 1)$$ Then take the derivation with respect to $Q(x_i)$.

How to get the derivation with respect to $Q(x_i)$. I'm not sure if my understanding is right. Suppose we have $N$ pixels and the label is from $L$ classes. The random variable is $X=(X_1,X_2,\cdots,X_N)$, where $X_i \in {\{0,1\}}^{L}$, i.e., for each pixel, the possible label is from $0,1,2,\cdots,L-1$. The energy is defined:

$$P(\mathbf X) = \frac{1}{Z(\mathbf X)} \exp \{{-E(\mathbf X)}\}$$ $$E(\mathbf X) = \sum_i \phi_u (x_i) + \sum_i \sum_{j\neq i} \phi_p (x_i,x_j)$$.

What does it mean for $Q(\mathbf X)$? How to get the derivation for $Q(x_i)$?

Answer 1

This derivation is a bit tricky. My experience is to use the approach in Bishop's Machine Learning and Pattern Recognition as a starting point. I've tried to derive it here: https://github.com/idnavid/misc/blob/master/variationalbayes_doc1.ipynb