Why does it appear impossible to fit Gaussians to arbitrary probability density functions $p$? I want to fit a Gaussian $q$ to a pdf $p$ by minimizing the energy $E = -\int q(x) \log p(x) dx$. This should result in a "delta function" Gaussian with $\sigma \rightarrow 0$ and $\mu \rightarrow x^*$, where $x^*$ is the mode of the target distribution.
If I try to do this via gradient descent, I get
\begin{align}
\nabla_\mu E &= - \int \log p \, \nabla_\mu q \,\, dx \\
&= - \int \log p \, \cdot (q \cdot \nabla_\mu (\log q)) \,\,  dx \\
&= - \mathbb E_q[ \nabla_\mu (\log q) \log p] \\
&= \mathbb E_q[ \Sigma^{-1}(x-\mu) \log p] \\
\end{align}
where the second line comes from the chain rule.
But according to the last line, if I draw from $q$, I get $\mu$ in expectation, which means my gradient will be usually be zero if I attempt gradient descent. What's going on here?
 A: To get some intuition for this problem, write the energy as a moment:
$$E(\mu,\sigma) \equiv \mathbb{E}(-\log p(X) | X \sim \text{N}(\mu, \sigma^2)).$$
Taking $\sigma=0$ and $\mu=x^*$ (where the latter is the mode of $p$) so that the distribution in the expectation is a delta function at the mode of target distribution then you get:
$$E(x^*,0) = \mathbb{E}(-\log p(X) | X = x^*) = \mathbb{E}(-\log p(x^*)) = -\log p(x^*).$$
Now, to show this is the minimum, we note that the mode $x^*$ satisfies $p(x^*) = \max p(x)$, so we have:
$$\begin{align}
E(\mu,\sigma) 
&= \mathbb{E}(-\log p(X) | X \sim \text{N}(\mu, \sigma^2)) \\[12pt]
&= - \int \limits_\mathbb{R} \log p(x) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt]
&\geqslant - \int \limits_\mathbb{R} \Big( \max_{x} \log p(x) \Big) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt]
&= - \int \limits_\mathbb{R} \log \Big( \max_{x} p(x) \Big) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt]
&= - \int \limits_\mathbb{R} \log p(x^*) \cdot \text{N}(x|\mu, \sigma^2) \ dx \\[6pt]
&= - \log p(x^*) \int \limits_\mathbb{R} \text{N}(x|\mu, \sigma^2) \ dx \\[6pt]
&= - \log p(x^*). \\[6pt]
\end{align}$$
This establishes that $E(x^*,0) \geqslant E(\mu, \sigma^2)$ for all $\mu \in \mathbb{R}$ and $\sigma \geqslant 0$, which means that the delta function at the mode is a minimising input for the energy function.  There is not really any need for gradient descent (or any other iterative method) here, except possibly to find the mode of $p$.
A: The last expectation isn't 0. For example, suppose you approximate $\log p$ with a linear function around $\mu$: $\log p \approx (x-\mu)^Tw+c$. Then you have:
$$
\begin{align}
&\mathbb{E}_q[\Sigma^{-1}(x-\mu)((x-\mu)^Tw+c)] \\
&\text{now pull out all the terms not involving $x$} \\
&= \Sigma^{-1}\mathbb{E}_q[(x-\mu)(x-\mu)^T]w + c\Sigma^{-1}\mathbb{E}_q[(x-\mu)] \\
&= \Sigma^{-1}\Sigma w + 0 \\
&= w
\end{align}
$$
So in fact, this is equivalent to gradient ascent on the gradient of $\log p$ (assuming small enough $\sigma)$.
