2nd order Taylor expansion of KL divergence

I am having trouble understanding page 6 of this PDF: https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/lecture-notes/lecture20-2pp.pdf

This is also a question in cs229-autumn2018 PS#3-3d.

In particular, page 6:

$$D_{KL} \big( P(X;\theta)||P(X;\theta+d) \big) \\ = \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;\theta+d)} \\ \approx \Sigma_x P(x;\theta) \Big( \frac{P(x;\theta)}{P(x;\theta)} - d^T \nabla_{\theta} lnP(x;\theta) - \frac{1}{2} d^T \nabla^2_{\theta} lnP(x;\theta) d \Big)$$

This is where I am having an issue. The approximation is a 2nd order Taylor expansion of $$D_{KL}$$, but instead of variable $$x$$ or $$\theta$$, we have a function $$P(x)$$ parameterized by $$\theta$$. I do not understand how this works.

To further illustrate where exactly my confusion lies, my intuition is that in normal Taylor expansion, e.g. $$f(x)$$ at $$a=5$$, our expansion would simply be $$\frac{1}{0!} f(5) (x-5)^0 + \frac{1}{1!} \frac{df}{dx}(5) (x-5)^1 + \frac{1}{2!} \frac{d^2f}{dx^2}(5) (x-5)^2$$

Now, for $$D_{KL}$$, we treat $$(\theta+d)$$ as a variable and $$\theta$$ as a constant. We take the derivative w.r.t to $$(\theta+d)$$. For where we would replace $$x$$ with $$a=5$$ in normal Taylor expansion, we replace $$P(\theta+d)$$ with $$P(\theta)$$.

For simplicity's sake, we denote $$(\theta + d)$$ as $$t$$. Thereby, we get:

$$D_{KL} \big( P(X;\theta)||P(X;t) \big) \\ = \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;t)} \\$$

The first term in the expansion is simple, no derivatives are involved. We simply replace $$t$$ with $$\theta$$. $$ln\frac{P(x;\theta)}{P(x;\theta)}$$

The second term, ignoring the $$(x-a)$$ part, is

$$\nabla_t ln\frac{P(x;\theta)}{P(x;t)} \\ = \nabla_t lnP(x;\theta) - \nabla_t lnP(x;t) \\ = 0 - \nabla_t lnP(x;t) \\ = \nabla_{\theta} lnP(x;\theta)$$

And baam... How is $$\nabla_t lnP(x;t)$$ supposed to be equal to $$\nabla_{\theta} lnP(x;\theta)$$?

If so, why doesn't $$\nabla_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$ or $$\nabla^2_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla^2_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$ which the second one is clearly wrong?

I felt I am missing some fundamental concepts here. Thank you for your help.

I believe the main reason for the confusion is the notation $$P(x;\theta)$$, which refers to the probability density function for the random variable $$x$$, which is parameterized by $$\theta$$. This notation is used to show that it is a probability density function for $$x$$ but not $$\theta$$.

Yet, this function can be simply understood as a multivariate function $$f(x, \theta)$$ (so as $$\ln P(x;\theta)$$), where it depends on both $$\theta$$ and $$x$$. And the Taylor expansion of it w.r.t. $$\theta$$ is simply given by $$f(x, \theta + \delta\theta) \approx f(x, \theta) + \nabla_{\theta}f(x,\theta)\delta\theta + \frac{1}{2}\delta\theta^{\rm T}\nabla^2_{\theta}f(x, \theta)\delta\theta + \dots,$$ where $$\nabla_{\theta}$$ is the partial derivative w.r.t. to $$\theta$$ while keeping $$x$$ constant.

By considering the relevant term in the KL divergence and by substituting $$f(x,\theta) = \ln P(x;\theta)$$ into the above equation, you will get \begin{aligned} \ln\frac{P(x;\theta)}{P(x;\theta+\delta\theta)} &= \ln P(x;\theta) -\ln P(x;\theta+\delta\theta)\\ &\approx\ln P(x;\theta) - \left(\ln P(x;\theta) + \nabla_{\theta}\ln P(x;\theta) \delta\theta + \frac{1}{2}\delta\theta^{\rm T}\nabla^2_{\theta}\ln P(x;\theta)\delta\theta\right), \end{aligned} which is the equation shown in the slides.

I hope this will clear things up for you.

I believe this: Information Geometry and Natural Gradients --- by Nathan Ratlif can answer your question. Please find the detailed derivation in Section 3 on Page 6-8. I just copy the part below.