Fisher information always 0? What's wrong with this argument?

We exchange derivative and integral while proving that the expected value of the score is 0. Here is the proof which does so: \begin{align*} \int \left( \frac{\partial}{\partial \theta} \log p(x;\theta) \right) p(x;\theta) \,dx &= \int \frac{1}{p(x;\theta)} \frac{\partial}{\partial \theta} p(x;\theta) \,p(x;\theta)\,dx \\ &= \int \frac{\partial}{\partial \theta} p(x;\theta)\,dx \\ &= \frac{\partial}{\partial \theta} \int p(x;\theta)\,dx \\ &= \frac{\partial}{\partial \theta} 1 = 0 \end{align*} So $E_{\theta}\left[\frac{\partial}{\partial \theta} \log p(x;\theta)\right] = 0$. What's wrong with the following argument? \begin{align*} & E_{\theta}\left[ \frac{\partial}{\partial \theta} \log p(x;\theta) \right] = 0 \\ \Rightarrow & \frac{\partial}{\partial \theta}E_{\theta}\left[ \frac{\partial}{\partial \theta} \log p(x;\theta) \right] = 0 \\ \Rightarrow & E_{\theta} \left[ \frac{\partial}{\partial \theta} \frac{\partial}{\partial \theta} \log p(x;\theta)\right] = 0 \\ \Rightarrow & E_{\theta} \left[ \frac{\partial^2}{\partial \theta^2} \log p(x;\theta)\right] = 0 \\ \Rightarrow &\mathcal{I}(\theta) = 0 \end{align*}

• If you write out the expectation again as an integral you will see that you haven't actually interchanged integration and differentiation. – whuber Nov 22 '14 at 15:33
• Ah, I see your point. There's also the $p(x;\theta)$ density term. Thanks whuber! – elexhobby Nov 22 '14 at 15:37

$0 = E_\theta\left(\frac{d}{d\theta}\frac{d}{d\theta}\log(p(x;\theta))\right) = \int \frac{d}{d\theta}\left(\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)\right)dx$
$= \int \left[\frac{d^2}{d\theta^2}\log(p(x;\theta))p(x;\theta)+\frac{d}{d\theta}p(x;\theta)\frac{d}{d\theta}\log(p(x;\theta))\right]dx$
$= \int \left[\frac{d^2}{d\theta^2}\log(p(x;\theta))p(x;\theta)+\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)\frac{d}{d\theta}\log(p(x;\theta))\right]dx$
$\implies 0 = E\left(\frac{d^2}{d\theta^2}\log(p(x;\theta))\right) + E\left((\frac{d}{d\theta}\log(p(x;\theta)))^2\right)$
$\implies -E\left(\frac{d^2}{d\theta^2}\log(p(x;\theta))\right) = E\left((\frac{d}{d\theta}\log(p(x;\theta)))^2\right) = I(\theta)$
The problem is that from your third to fourth line the derivative is applied to the entire expression in the expectation. So you need to use the product rule on the term $\frac{d}{d\theta}\log(p(x;\theta))p(x;\theta)$.