We exchange derivative and integral while proving that the expected value of the score is 0. Here is the proof which does so: \begin{align*} \int \left( \frac{\partial}{\partial \theta} \log p(x;\theta) \right) p(x;\theta) \,dx &= \int \frac{1}{p(x;\theta)} \frac{\partial}{\partial \theta} p(x;\theta) \,p(x;\theta)\,dx \\ &= \int \frac{\partial}{\partial \theta} p(x;\theta)\,dx \\ &= \frac{\partial}{\partial \theta} \int p(x;\theta)\,dx \\ &= \frac{\partial}{\partial \theta} 1 = 0 \end{align*} So $E_{\theta}\left[\frac{\partial}{\partial \theta} \log p(x;\theta)\right] = 0$. What's wrong with the following argument? \begin{align*} & E_{\theta}\left[ \frac{\partial}{\partial \theta} \log p(x;\theta) \right] = 0 \\ \Rightarrow & \frac{\partial}{\partial \theta}E_{\theta}\left[ \frac{\partial}{\partial \theta} \log p(x;\theta) \right] = 0 \\ \Rightarrow & E_{\theta} \left[ \frac{\partial}{\partial \theta} \frac{\partial}{\partial \theta} \log p(x;\theta)\right] = 0 \\ \Rightarrow & E_{\theta} \left[ \frac{\partial^2}{\partial \theta^2} \log p(x;\theta)\right] = 0 \\ \Rightarrow &\mathcal{I}(\theta) = 0 \end{align*}

  • 2
    $\begingroup$ If you write out the expectation again as an integral you will see that you haven't actually interchanged integration and differentiation. $\endgroup$ – whuber Nov 22 '14 at 15:33
  • 2
    $\begingroup$ Ah, I see your point. There's also the $p(x;\theta)$ density term. Thanks whuber! $\endgroup$ – elexhobby Nov 22 '14 at 15:37

Starting from your third line:

$0 = E_\theta\left(\frac{d}{d\theta}\frac{d}{d\theta}\log(p(x;\theta))\right) = \int \frac{d}{d\theta}\left(\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)\right)dx$

$ = \int \left[\frac{d^2}{d\theta^2}\log(p(x;\theta))p(x;\theta)+\frac{d}{d\theta}p(x;\theta)\frac{d}{d\theta}\log(p(x;\theta))\right]dx$

$ = \int \left[\frac{d^2}{d\theta^2}\log(p(x;\theta))p(x;\theta)+\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)\frac{d}{d\theta}\log(p(x;\theta))\right]dx$

$\implies 0 = E\left(\frac{d^2}{d\theta^2}\log(p(x;\theta))\right) + E\left((\frac{d}{d\theta}\log(p(x;\theta)))^2\right)$

$\implies -E\left(\frac{d^2}{d\theta^2}\log(p(x;\theta))\right) = E\left((\frac{d}{d\theta}\log(p(x;\theta)))^2\right) = I(\theta)$

The problem is that from your third to fourth line the derivative is applied to the entire expression in the expectation. So you need to use the product rule on the term $\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.