I want to fit a Gaussian $q$ to a pdf $p$ by minimizing the energy $E = -\int q(x) \log p(x) dx$. This should result in a "delta function" Gaussian with $\sigma \rightarrow 0$ and $\mu \rightarrow x^*$, where $x^*$ is the mode of the target distribution.

If I try to do this via gradient descent, I get

\begin{align} \nabla_\mu E &= - \int \log p \, \nabla_\mu q \,\, dx \\ &= - \int \log p \, \cdot (q \cdot \nabla_\mu (\log q)) \,\, dx \\ &= - \mathbb E_q[ \nabla_\mu (\log q) \log p] \\ &= \mathbb E_q[ \Sigma^{-1}(x-\mu) \log p] \\ \end{align}

where the second line comes from the chain rule.

But according to the last line, if I draw from $q$, I get $\mu$ in expectation, which means my gradient will be usually be zero if I attempt gradient descent. What's going on here?


The last expectation isn't 0. For example, suppose you approximate $\log p$ with a linear function around $\mu$: $\log p \approx (x-\mu)^Tw+c$. Then you have:

$$ \begin{align} &\mathbb{E}_q[\Sigma^{-1}(x-\mu)((x-\mu)^Tw+c)] \\ &\text{now pull out all the terms not involving $x$} \\ &= \Sigma^{-1}\mathbb{E}_q[(x-\mu)(x-\mu)^T]w + c\Sigma^{-1}\mathbb{E}_q[(x-\mu)] \\ &= \Sigma^{-1}\Sigma w + 0 \\ &= w \end{align} $$

So in fact, this is equivalent to gradient ascent on the gradient of $\log p$ (assuming small enough $\sigma)$.


To get some intuition for this problem, write the energy as a moment:

$$E(\mu,\sigma) \equiv \mathbb{E}(-\log p(X) | X \sim \text{N}(\mu, \sigma^2)).$$

Taking $\sigma=0$ and $\mu=x^*$ (where the latter is the mode of $p$) so that the distribution in the expectation is a delta function at the mode of target distribution then you get:

$$E(x^*,0) = \mathbb{E}(-\log p(X) | X = x^*) = \mathbb{E}(-\log p(x^*)) = -\log p(x^*).$$

Now, to show this is the minimum, we note that the mode $x^*$ satisfies $p(x^*) = \max p(x)$, so we have:

$$\begin{align} E(\mu,\sigma) &= \mathbb{E}(-\log p(X) | X \sim \text{N}(\mu, \sigma^2)) \\[12pt] &= - \int \limits_\mathbb{R} \log p(x) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt] &\geqslant - \int \limits_\mathbb{R} \Big( \max_{x} \log p(x) \Big) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt] &= - \int \limits_\mathbb{R} \log \Big( \max_{x} p(x) \Big) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt] &= - \int \limits_\mathbb{R} \log p(x^*) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt] &= - \log p(x^*) \int \limits_\mathbb{R} \text{N}(x|\mu, \sigma^2) \ dx \\[6pt] &= - \log p(x^*). \\[6pt] \end{align}$$

This establishes that $E(x^*,0) \geqslant E(\mu, \sigma^2)$ for all $\mu \in \mathbb{R}$ and $\sigma \geqslant 0$, which means that the delta function at the mode is a minimising input for the energy function. There is not really any need for gradient descent (or any other iterative method) here, except possibly to find the mode of $p$.

  • $\begingroup$ What exactly do you mean by "there is not really any need for gradient descent"? Given an initialization of $\mu$, don't I need to move in the direction of the gradient in order to approach $x^*$? $\endgroup$
    – actinidia
    May 15 at 6:51
  • $\begingroup$ It depends on whether or not you already have an explicit formula for $x^*$. For many distributions the mode has an explicit closed form, in which case iterative methods are not needed. $\endgroup$
    – Ben
    May 15 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.