# How to obtain the functional derivative in variational inference?

Referring to David Blei's notes on variational inference, I wonder how to get the derivative of $q(z_k)$, the distribution of $z_k$, in eq. 23 from eq. 22.

Namely, eq. 22 is $L_k = \int q(z_k) \mathbb{E}_{-k}[\log p(z_k | z_{-k},x)]dz_k - \int q(z_k)\log q(z_k) dz_k$ and,

eq. 23 is $\frac{dL_k}{dq(z_k)} = \mathbb{E}_{-k}[\log p(z_k | z_{-k},x)] - \log q(z_k) - 1$

I suppose the derivation uses functional derivatives, which I know little about. I'd very appreciate a detailed derivation.

• I believe you are correct that this is a functional derivative, but the notation and the language are pure nonsense. This becomes apparent once you realize that all appearances of "$z_k$" are as dummy variables of integration and therefore "$q(z_k)$" is not a part of the formula for $L_k$. (You could just as well replace "$z_k$" by a variable "$y$", making all occurrences of "$z_k$" in the right hand side disappear, and then taking a derivative with respect to "$q(z_k)$" would become truly mysterious indeed.)
– whuber
Sep 21 '16 at 14:47

Let's streamline the notation by fixing a function $f$ and considering a functional

$$\mathcal{L}[q] = \int (q(z) f(z) - q(z) \log(q(z))) dz.$$

A variation $h$ is a function for which $q+h$ is still the same kind of function as $q$ (e.g., continuous or non-negative or whatever you need). The effect of changing $q$ to $q+h$ is found in the usual way: we compare the results of $\mathcal{L}$ by subtracting the original value from the new one:

\eqalign{ \frac{\delta\mathcal{L}}{\delta q}[h] &= \mathcal{L}[q+h] - \mathcal{L}[h] \\ &= \int \left((q+h)(z) f(z) - (q+h)(z) \log((q+h)(z)) - \left(q(z) f(z) - q(z) \log(q(z))\right)\right) dz.}

Let's now restrict attention to variations $h$ that have infinitesimally small values so that we may be free to neglect second-order terms. Specifically, that means we plan to use a product-like differentiation rule to approximate the product difference

$$(q+h)(z)\log((q+h)(z)) - q(z)\log(q(z)) \approx h(z)\log(q(z)) + q(z) \frac{h(z)}{q(z)}.\tag{1}$$

To see where this came from, compare it to taking the differential of the function (not functional)

$$d(x \log(x)) = (dx) \log(x) + x \left(\frac{dx}{x}\right).$$

Plugging $(1)$ into $\frac{\delta\mathcal{L}}{\delta q}[h]$, simplifying, and factoring out the common factors of $h(z)$ yields

$$\frac{\delta\mathcal{L}}{\delta q}[h] = \int\left(f(z) - \log(q(z)) - 1\right) h(z) dz.$$

If this is to vanish (to second order in $h$, anyway) for all infinitesimal $h$, then--assuming that at each $z$ in the domain of integration there exist some variations $h$ that are nonzero in some neighborhood of $z$--the right hand side can vanish only when the coefficient of $h$ itself vanishes for all $z$; that is,

$$f(z) - \log(q(z)) - 1 = 0.$$

That's Equation 23 in the referenced notes. It means the functional $\mathcal{L}$ is stationary at $q$. That's a necessary (but not sufficient) condition for $\mathcal{L}$ to have a local extremum at $q$.