In page 487/1266 of [probabilistic graphic model: principal and technique] book, the authors give the following Lagrange equation:

$$L_i[Q]=\sum_{\phi \in \Phi} \mathbf{E}_{\mathbf{U}_\phi \sim Q}[\ln \phi] + \mathbf{H}_Q (X_i) + \lambda (\sum_{x_i} Q(x_i) - 1)$$

And in page 42/86 of [http://web.eng.tau.ac.il/deep_learn/wp-content/uploads/2017/01/Semantic-Segmentation.pdf], the authors derived the following Lagrange equation for solving a maximum problem for KL divergence using mean-field approximation:

$$\mathbf{D}(Q\|P) = \sum_{\mathbf{x}} Q(\mathbf{x}) \log \frac{Q(\mathbf{x})}{P(\mathbf{x})}=\mathbf{E}_{U\sim Q}[E(\mathbf{U})]+\sum \mathbf{E}_{U_i \sim Q_i }[\log Q_i(U_i)]+\log Z + \lambda (\sum_{x_i} Q(x_i) - 1)$$ Then take the derivation with respect to $Q(x_i)$.

How to get the derivation with respect to $Q(x_i)$. I'm not sure if my understanding is right. Suppose we have $N$ pixels and the label is from $L$ classes. The random variable is $X=(X_1,X_2,\cdots,X_N)$, where $X_i \in {\{0,1\}}^{L}$, i.e., for each pixel, the possible label is from $0,1,2,\cdots,L-1$. The energy is defined:

$$P(\mathbf X) = \frac{1}{Z(\mathbf X)} \exp \{{-E(\mathbf X)}\}$$ $$E(\mathbf X) = \sum_i \phi_u (x_i) + \sum_i \sum_{j\neq i} \phi_p (x_i,x_j)$$.

What does it mean for $Q(\mathbf X)$? How to get the derivation for $Q(x_i)$?


1 Answer 1


This derivation is a bit tricky. My experience is to use the approach in Bishop's Machine Learning and Pattern Recognition as a starting point. I've tried to derive it here: https://github.com/idnavid/misc/blob/master/variationalbayes_doc1.ipynb

  • 1
    $\begingroup$ Hi, @idnavid, thanks a lot for your kind comment and sharing!! $\endgroup$
    – mining
    Commented Jan 10, 2018 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.