I have read two definitions of the Fisher information matrix. Are they equivalent? (The formulas given below are slightly modified versions of those given in the indicated sources, so as to bring out the differences between them.)

  1. Zacks, The Theory of Statistical Inference (1971), p. 194: $$ \left[E_\theta\left(\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right)\cdot\frac{\partial}{\partial\theta_j}\ln f\left(X;\theta_1,\dots,\theta_r\right)\right)\right]_{i,j} $$
  2. Schervish, Theory of Statistics (1997), Definition 2.79, p. 111: $$ \mathrm{Cov}_\theta\left(\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right),\frac{\partial}{\partial\theta_j}\ln f\left(X;\theta_1,\dots,\theta_r\right)\right) $$
  • 4
    $\begingroup$ That's the same as saying "when does $\text{Cov}(X,Y) = \text{E}(XY)$?" ... which is when $\text{E}(X)\text{E}(Y) = 0$ ... so you should think about the circumstances under which they're the same. That is, when does $\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right) = 0$? $\endgroup$
    – Glen_b
    Jun 13 '13 at 8:47
  • 1
    $\begingroup$ @Glen_b: That's a good question! I would say that $\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right)=0$ at a point of extremum of $f$ w.r.t. $\boldsymbol\theta$. $\endgroup$
    – Evan Aad
    Jun 13 '13 at 8:51
  • 1
    $\begingroup$ @EvanAad note quite. Do not forget that you take the expected value over $X$. $\endgroup$
    – gui11aume
    Jun 13 '13 at 9:31

Fortunately they are the same. All we need to prove is $E_{\theta} \left[ \frac{\partial}{\partial \theta} \log f(x|\theta) \right] = 0$, so here we go.

First note that

$$\int_{\mathbb{R}} \frac{\partial}{\partial \theta}\log(f(x|\theta)) \cdot f(x|\theta) dx = \int_{\mathbb{R}} \frac{\partial f(x|\theta)}{\partial \theta}dx. $$

Under the conditions of Leibniz's integral rule you can rewrite the above

$$ \frac{\partial}{\partial \theta} \int_{\mathbb{R}} f(x|\theta)dx = \frac{\partial}{\partial \theta} 1 = 0.$$

So, except pathological cases where the conditions mentioned above do not hold, the expected value of the score is 0, and the covariance is the same as the expected value of the product.

  • 2
    $\begingroup$ Thank you. Could you have meant Leibniz's integral rule rather than Lebesgue's dominated convergence theorem? $\endgroup$
    – Evan Aad
    Jun 13 '13 at 10:45
  • 2
    $\begingroup$ Indeed! I had in mind point 3 of the measure theory statement of the rule, but of course point 1 and 2 are also required. I will edit my answer accordingly. $\endgroup$
    – gui11aume
    Jun 13 '13 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.