Why is the expected gradient of a density not parallel to the expected gradient of the log density?

Question

I'm confused by a seemingly counter-intuitive property of the interaction between distributions, log transforms, expectations and gradients.

Suppose I have some distribution over random variable $x$ parameterized by $\theta$. (Unless I'm mistaken), the MLE for the distribution is the same as the MLE for the log of the distribution.

$$\theta^* = \underset{\theta}{\operatorname{argmax}} p(x;\theta) = \underset{\theta}{\operatorname{argmax}} \log p(x;\theta)$$

If $p(x;\theta)$ is differentiable with respect to $\theta$, I believe that the gradient of the distribution with respect to $\theta$ is parallel to the gradient of the log distribution since

$$\nabla_{\theta} \log p(x ; \theta) = \frac{1}{p(x ; \theta)} \nabla_{\theta} p(x ; \theta)$$

This means that performing gradient ascent on $\log p(x;\theta)$ is the same as performing gradient ascent on $p(x; \theta)$.

In contrast, if I consider the expected value of the gradient of the distribution $\langle \nabla_{\theta} p(x;\theta) \rangle_{p(x;\theta)}$ against the expected value of the gradient of the log density $\langle \nabla_{\theta} \log p(x;\theta) \rangle_{p(x;\theta)}$ , the two are perpendicular since the mean of the gradient of the log density is 0:

$$ \begin{align*} \langle \nabla_{\theta} \log p(x;\theta) \rangle_{p(x;\theta)} &= \int dx \, p(x;\theta) \nabla_{\theta} \log p(x;\theta)\\ &= \int dx \, p(x;\theta) \frac{1}{p(x;\theta)} \nabla_{\theta} p(x;\theta)\\ &= \int dx \, \nabla_{\theta} p(x;\theta)\\ &= \nabla_{\theta} \int dx \, p(x;\theta)\\ &= \nabla_{\theta} 1\\ &= 0 \end{align*} $$

This seems contradictory. For any specific sample from the density, the two gradients point in the same direction, but averaged over multiple samples, they don't. Why is the expected gradient of a density not parallel to the expected gradient of the log density?

Answer 1

There is a minor typo in one of the equations: $$\nabla_{\theta} \log p(x ; \theta) = \frac{\nabla_{\theta} p(x ; \theta)}{p(x ; \theta)} $$ Nevertheless, I think the confusion is because you are ignoring the effect of the term $p(x;\theta)$.

Consider a simple example, the gradient of the function $x^2 + y^2$ averaged over all the points on the unit circle is zero i.e $2x \hat{i} + 2y\hat{j} $ average over the unit circle is zero vector.

However, the same gradient when weighted by a term, say, $\frac{1}{x}$ is non-zero i.e $2 \hat{i} + 2\frac{y}{x}\hat{j} $ averaged over unit circle has positive component along x-axis.

Notice that in both the cases, the gradients are parallel

Answer 2

Usually when you're doing maximum likelihood estimation what you're really trying to do is maximize the posterior distribution $p(\theta | x)$. That is, you're trying to find the most likely model (or model parameters) given a set of data $x$. By Bayes' theorem, this becomes $$ p(\theta | x) = \frac{p(x | \theta) p(\theta)}{p(x)}. $$ We maximize this by taking the derivative with respect to $\theta$. The derivative $\nabla_\theta p(x)$ is zero, and if the prior is sufficiently weak then its derivative is (approximately) zero as well, so we can approximate the maximum posterior by using maximum likelihood estimation.

Now, consider the form of $p(x|\theta)$ when you have multiple data points. If the points are chosen independently from the underlying (unknown!) distribution $p(x)$, then $$p(\vec{x}|\theta) = p(x_1, x_2, \ldots | \theta) = \prod_i p(x_i|\theta).$$ As you note, $\nabla_\theta f(\theta)$ is parallel to $\nabla_\theta \log f(\theta)$, so we might as well maximize $\log p(\vec{x}|\theta)$ instead. The product becomes a sum, $$ \log p(\vec{x}|\theta) = \sum_i \log p(x_i | \theta), $$ and we can divide by the total number of samples and take the continuum limit to get an expectation over all samples, $$ \log p(\vec{x}|\theta) \rightarrow \langle\log p(x|\theta)\rangle_{p(x)}. $$ Maximizing this expression will be equivalent to maximizing the posterior distribution with a uniform (or weak) prior. Note that the expectation should be over $p(x)$, not $p(x|\theta)$! The derivative goes to zero when $p(x) = p(x|\theta)$, as you noted, which just shows that you've modeled the distribution perfectly and reached the maximum likelihood.

So why not maximize the expectation of $p(x|\theta$) directly? Because that expectation doesn't correspond to joint a probability distribution! You shouldn't expect that maximizing the sum of a bunch of probabilities would yield the same result as maximizing their product, which is really what you want to do at the end of the day.