I am having trouble understanding page 6 of this PDF: https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/lecture-notes/lecture20-2pp.pdf

This is also a question in cs229-autumn2018 PS#3-3d.

In particular, page 6:

$$ D_{KL} \big( P(X;\theta)||P(X;\theta+d) \big) \\ = \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;\theta+d)} \\ \approx \Sigma_x P(x;\theta) \Big( \frac{P(x;\theta)}{P(x;\theta)} - d^T \nabla_{\theta} lnP(x;\theta) - \frac{1}{2} d^T \nabla^2_{\theta} lnP(x;\theta) d \Big) $$

This is where I am having an issue. The approximation is a 2nd order Taylor expansion of $D_{KL}$, but instead of variable $x$ or $\theta$, we have a function $P(x)$ parameterized by $\theta$. I do not understand how this works.

To further illustrate where exactly my confusion lies, my intuition is that in normal Taylor expansion, e.g. $f(x)$ at $a=5$, our expansion would simply be $$\frac{1}{0!} f(5) (x-5)^0 + \frac{1}{1!} \frac{df}{dx}(5) (x-5)^1 + \frac{1}{2!} \frac{d^2f}{dx^2}(5) (x-5)^2$$

Now, for $D_{KL}$, we treat $(\theta+d)$ as a variable and $\theta$ as a constant. We take the derivative w.r.t to $(\theta+d)$. For where we would replace $x$ with $a=5$ in normal Taylor expansion, we replace $P(\theta+d)$ with $P(\theta)$.

For simplicity's sake, we denote $(\theta + d)$ as $t$. Thereby, we get:

$$ D_{KL} \big( P(X;\theta)||P(X;t) \big) \\ = \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;t)} \\ $$

The first term in the expansion is simple, no derivatives are involved. We simply replace $t$ with $\theta$. $$ln\frac{P(x;\theta)}{P(x;\theta)}$$

The second term, ignoring the $(x-a)$ part, is

$$ \nabla_t ln\frac{P(x;\theta)}{P(x;t)} \\ = \nabla_t lnP(x;\theta) - \nabla_t lnP(x;t) \\ = 0 - \nabla_t lnP(x;t) \\ = \nabla_{\theta} lnP(x;\theta) $$

And baam... How is $\nabla_t lnP(x;t)$ supposed to be equal to $\nabla_{\theta} lnP(x;\theta)$?

If so, why doesn't $$\nabla_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$ or $$\nabla^2_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla^2_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$ which the second one is clearly wrong?

I felt I am missing some fundamental concepts here. Thank you for your help.


2 Answers 2


I believe the main reason for the confusion is the notation $P(x;\theta)$, which refers to the probability density function for the random variable $x$, which is parameterized by $\theta$. This notation is used to show that it is a probability density function for $x$ but not $\theta$.

Yet, this function can be simply understood as a multivariate function $f(x, \theta)$ (so as $\ln P(x;\theta)$), where it depends on both $\theta$ and $x$. And the Taylor expansion of it w.r.t. $\theta$ is simply given by $$f(x, \theta + \delta\theta) \approx f(x, \theta) + \nabla_{\theta}f(x,\theta)\delta\theta + \frac{1}{2}\delta\theta^{\rm T}\nabla^2_{\theta}f(x, \theta)\delta\theta + \dots,$$ where $\nabla_{\theta}$ is the partial derivative w.r.t. to $\theta$ while keeping $x$ constant.

By considering the relevant term in the KL divergence and by substituting $f(x,\theta) = \ln P(x;\theta)$ into the above equation, you will get $$ \begin{aligned} \ln\frac{P(x;\theta)}{P(x;\theta+\delta\theta)} &= \ln P(x;\theta) -\ln P(x;\theta+\delta\theta)\\ &\approx\ln P(x;\theta) - \left(\ln P(x;\theta) + \nabla_{\theta}\ln P(x;\theta) \delta\theta + \frac{1}{2}\delta\theta^{\rm T}\nabla^2_{\theta}\ln P(x;\theta)\delta\theta\right), \end{aligned} $$ which is the equation shown in the slides.

I hope this will clear things up for you.


I believe this: Information Geometry and Natural Gradients --- by Nathan Ratlif can answer your question. Please find the detailed derivation in Section 3 on Page 6-8. I just copy the part below.

enter image description here enter image description here enter image description here

Update: Since the original link i posted become invalid, I found the updated link:https://drive.google.com/file/d/1Fx_ZyKZbV_geKQ6DhW-wiK8oJtxhlLIz/view

  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Mar 11, 2022 at 18:26
  • $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Mar 11, 2022 at 18:48
  • $\begingroup$ @wanxin-jin I can't access the website. Would you please post the whole pdf? $\endgroup$
    – psimeson
    Feb 7, 2023 at 5:26
  • 1
    $\begingroup$ @psimeson, please see the updated link: drive.google.com/file/d/1Fx_ZyKZbV_geKQ6DhW-wiK8oJtxhlLIz/view $\endgroup$
    – Wanxin Jin
    Mar 8, 2023 at 3:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.