1
$\begingroup$

We exchange derivative and integral while proving that the expected value of the score is 0. Here is the proof which does so: \begin{align*} \int \left( \frac{\partial}{\partial \theta} \log p(x;\theta) \right) p(x;\theta) \,dx &= \int \frac{1}{p(x;\theta)} \frac{\partial}{\partial \theta} p(x;\theta) \,p(x;\theta)\,dx \\ &= \int \frac{\partial}{\partial \theta} p(x;\theta)\,dx \\ &= \frac{\partial}{\partial \theta} \int p(x;\theta)\,dx \\ &= \frac{\partial}{\partial \theta} 1 = 0 \end{align*} So $E_{\theta}\left[\frac{\partial}{\partial \theta} \log p(x;\theta)\right] = 0$. What's wrong with the following argument? \begin{align*} & E_{\theta}\left[ \frac{\partial}{\partial \theta} \log p(x;\theta) \right] = 0 \\ \Rightarrow & \frac{\partial}{\partial \theta}E_{\theta}\left[ \frac{\partial}{\partial \theta} \log p(x;\theta) \right] = 0 \\ \Rightarrow & E_{\theta} \left[ \frac{\partial}{\partial \theta} \frac{\partial}{\partial \theta} \log p(x;\theta)\right] = 0 \\ \Rightarrow & E_{\theta} \left[ \frac{\partial^2}{\partial \theta^2} \log p(x;\theta)\right] = 0 \\ \Rightarrow &\mathcal{I}(\theta) = 0 \end{align*}

$\endgroup$
2
  • 2
    $\begingroup$ If you write out the expectation again as an integral you will see that you haven't actually interchanged integration and differentiation. $\endgroup$
    – whuber
    Commented Nov 22, 2014 at 15:33
  • 2
    $\begingroup$ Ah, I see your point. There's also the $p(x;\theta)$ density term. Thanks whuber! $\endgroup$
    – elexhobby
    Commented Nov 22, 2014 at 15:37

1 Answer 1

3
$\begingroup$

Starting from your third line:

$0 = E_\theta\left(\frac{d}{d\theta}\frac{d}{d\theta}\log(p(x;\theta))\right) = \int \frac{d}{d\theta}\left(\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)\right)dx$

$ = \int \left[\frac{d^2}{d\theta^2}\log(p(x;\theta))p(x;\theta)+\frac{d}{d\theta}p(x;\theta)\frac{d}{d\theta}\log(p(x;\theta))\right]dx$

$ = \int \left[\frac{d^2}{d\theta^2}\log(p(x;\theta))p(x;\theta)+\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)\frac{d}{d\theta}\log(p(x;\theta))\right]dx$

$\implies 0 = E\left(\frac{d^2}{d\theta^2}\log(p(x;\theta))\right) + E\left((\frac{d}{d\theta}\log(p(x;\theta)))^2\right)$

$\implies -E\left(\frac{d^2}{d\theta^2}\log(p(x;\theta))\right) = E\left((\frac{d}{d\theta}\log(p(x;\theta)))^2\right) = I(\theta)$

The problem is that from your third to fourth line the derivative is applied to the entire expression in the expectation. So you need to use the product rule on the term $\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)$.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.