2nd order Taylor expansion of KL divergence I am having trouble understanding page 6 of this PDF: https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/lecture-notes/lecture20-2pp.pdf
This is also a question in cs229-autumn2018 PS#3-3d.
In particular, page 6:
$$
D_{KL} \big( P(X;\theta)||P(X;\theta+d) \big) \\
= \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;\theta+d)} \\
\approx \Sigma_x P(x;\theta) \Big(
\frac{P(x;\theta)}{P(x;\theta)} 
- d^T \nabla_{\theta} lnP(x;\theta)
- \frac{1}{2} d^T \nabla^2_{\theta} lnP(x;\theta) d
\Big)
$$
This is where I am having an issue. The approximation is a 2nd order Taylor expansion of $D_{KL}$, but instead of variable $x$ or $\theta$, we have a function $P(x)$ parameterized by $\theta$. I do not understand how this works.

To further illustrate where exactly my confusion lies, my intuition is that in normal Taylor expansion, e.g. $f(x)$ at $a=5$, our expansion would simply be
$$\frac{1}{0!} f(5) (x-5)^0 + \frac{1}{1!} \frac{df}{dx}(5) (x-5)^1 + \frac{1}{2!} \frac{d^2f}{dx^2}(5) (x-5)^2$$
Now, for $D_{KL}$, we treat $(\theta+d)$ as a variable and $\theta$ as a constant. We take the derivative w.r.t to $(\theta+d)$. For where we would replace $x$ with $a=5$ in normal Taylor expansion, we replace $P(\theta+d)$ with $P(\theta)$.
For simplicity's sake, we denote $(\theta + d)$ as $t$. Thereby, we get:
$$
D_{KL} \big( P(X;\theta)||P(X;t) \big) \\
= \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;t)} \\
$$
The first term in the expansion is simple, no derivatives are involved. We simply replace $t$ with $\theta$.
$$ln\frac{P(x;\theta)}{P(x;\theta)}$$
The second term, ignoring the $(x-a)$ part, is
$$
\nabla_t ln\frac{P(x;\theta)}{P(x;t)} \\
= \nabla_t lnP(x;\theta) - \nabla_t lnP(x;t) \\
= 0 - \nabla_t lnP(x;t) \\
= \nabla_{\theta} lnP(x;\theta)
$$
And baam... How is $\nabla_t lnP(x;t)$ supposed to be equal to $\nabla_{\theta} lnP(x;\theta)$?
If so, why doesn't
$$\nabla_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$
or
$$\nabla^2_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla^2_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$
which the second one is clearly wrong?
I felt I am missing some fundamental concepts here. Thank you for your help.
 A: I believe the main reason for the confusion is the notation $P(x;\theta)$, which refers to the probability density function for the random variable $x$, which is parameterized by $\theta$. This notation is used to show that it is a probability density function for $x$ but not $\theta$.
Yet, this function can be simply understood as a multivariate function $f(x, \theta)$ (so as $\ln P(x;\theta)$), where it depends on both $\theta$ and $x$. And the Taylor expansion of it w.r.t. $\theta$ is simply given by
$$f(x, \theta + \delta\theta) \approx f(x, \theta) + \nabla_{\theta}f(x,\theta)\delta\theta + \frac{1}{2}\delta\theta^{\rm T}\nabla^2_{\theta}f(x, \theta)\delta\theta + \dots,$$
where $\nabla_{\theta}$ is the partial derivative w.r.t. to $\theta$ while keeping $x$ constant.
By considering the relevant term in the KL divergence and by substituting $f(x,\theta) = \ln P(x;\theta)$ into the above equation, you will get
$$
\begin{aligned}
\ln\frac{P(x;\theta)}{P(x;\theta+\delta\theta)}
&= \ln P(x;\theta)  -\ln P(x;\theta+\delta\theta)\\
&\approx\ln P(x;\theta) - \left(\ln P(x;\theta) + \nabla_{\theta}\ln P(x;\theta) \delta\theta + \frac{1}{2}\delta\theta^{\rm T}\nabla^2_{\theta}\ln P(x;\theta)\delta\theta\right),
\end{aligned}
$$
which is the equation shown in the slides.
I hope this will clear things up for you.
A: I believe this: Information Geometry and Natural
Gradients  --- by Nathan Ratlif can answer your question. Please find the detailed derivation in Section 3 on Page 6-8. I just copy the part below.



