Referring to David Blei's notes on variational inference, I wonder how to get the derivative of $q(z_k)$, the distribution of $z_k$, in eq. 23 from eq. 22.

Namely, eq. 22 is $L_k = \int q(z_k) \mathbb{E}_{-k}[\log p(z_k | z_{-k},x)]dz_k - \int q(z_k)\log q(z_k) dz_k$ and,

eq. 23 is $\frac{dL_k}{dq(z_k)} = \mathbb{E}_{-k}[\log p(z_k | z_{-k},x)] - \log q(z_k) - 1$

I suppose the derivation uses functional derivatives, which I know little about. I'd very appreciate a detailed derivation.

  • $\begingroup$ I believe you are correct that this is a functional derivative, but the notation and the language are pure nonsense. This becomes apparent once you realize that all appearances of "$z_k$" are as dummy variables of integration and therefore "$q(z_k)$" is not a part of the formula for $L_k$. (You could just as well replace "$z_k$" by a variable "$y$", making all occurrences of "$z_k$" in the right hand side disappear, and then taking a derivative with respect to "$q(z_k)$" would become truly mysterious indeed.) $\endgroup$
    – whuber
    Sep 21 '16 at 14:47

Let's streamline the notation by fixing a function $f$ and considering a functional

$$\mathcal{L}[q] = \int (q(z) f(z) - q(z) \log(q(z))) dz.$$

A variation $h$ is a function for which $q+h$ is still the same kind of function as $q$ (e.g., continuous or non-negative or whatever you need). The effect of changing $q$ to $q+h$ is found in the usual way: we compare the results of $\mathcal{L}$ by subtracting the original value from the new one:

$$\eqalign{ \frac{\delta\mathcal{L}}{\delta q}[h] &= \mathcal{L}[q+h] - \mathcal{L}[h] \\ &= \int \left((q+h)(z) f(z) - (q+h)(z) \log((q+h)(z)) - \left(q(z) f(z) - q(z) \log(q(z))\right)\right) dz.}$$

Let's now restrict attention to variations $h$ that have infinitesimally small values so that we may be free to neglect second-order terms. Specifically, that means we plan to use a product-like differentiation rule to approximate the product difference

$$(q+h)(z)\log((q+h)(z)) - q(z)\log(q(z)) \approx h(z)\log(q(z)) + q(z) \frac{h(z)}{q(z)}.\tag{1}$$

To see where this came from, compare it to taking the differential of the function (not functional)

$$d(x \log(x)) = (dx) \log(x) + x \left(\frac{dx}{x}\right).$$

Plugging $(1)$ into $\frac{\delta\mathcal{L}}{\delta q}[h]$, simplifying, and factoring out the common factors of $h(z)$ yields

$$\frac{\delta\mathcal{L}}{\delta q}[h] = \int\left(f(z) - \log(q(z)) - 1\right) h(z) dz.$$

If this is to vanish (to second order in $h$, anyway) for all infinitesimal $h$, then--assuming that at each $z$ in the domain of integration there exist some variations $h$ that are nonzero in some neighborhood of $z$--the right hand side can vanish only when the coefficient of $h$ itself vanishes for all $z$; that is,

$$f(z) - \log(q(z)) - 1 = 0.$$

That's Equation 23 in the referenced notes. It means the functional $\mathcal{L}$ is stationary at $q$. That's a necessary (but not sufficient) condition for $\mathcal{L}$ to have a local extremum at $q$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.