6
$\begingroup$

I have read two definitions of the Fisher information matrix. Are they equivalent? (The formulas given below are slightly modified versions of those given in the indicated sources, so as to bring out the differences between them.)

  1. Zacks, The Theory of Statistical Inference (1971), p. 194: $$ \left[E_\theta\left(\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right)\cdot\frac{\partial}{\partial\theta_j}\ln f\left(X;\theta_1,\dots,\theta_r\right)\right)\right]_{i,j} $$
  2. Schervish, Theory of Statistics (1997), Definition 2.79, p. 111: $$ \mathrm{Cov}_\theta\left(\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right),\frac{\partial}{\partial\theta_j}\ln f\left(X;\theta_1,\dots,\theta_r\right)\right) $$
$\endgroup$
3
  • 4
    $\begingroup$ That's the same as saying "when does $\text{Cov}(X,Y) = \text{E}(XY)$?" ... which is when $\text{E}(X)\text{E}(Y) = 0$ ... so you should think about the circumstances under which they're the same. That is, when does $\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right) = 0$? $\endgroup$
    – Glen_b
    Commented Jun 13, 2013 at 8:47
  • 1
    $\begingroup$ @Glen_b: That's a good question! I would say that $\frac{\partial}{\partial\theta_i}\ln f\left(X;\theta_1,\dots,\theta_r\right)=0$ at a point of extremum of $f$ w.r.t. $\boldsymbol\theta$. $\endgroup$
    – Evan Aad
    Commented Jun 13, 2013 at 8:51
  • 1
    $\begingroup$ @EvanAad note quite. Do not forget that you take the expected value over $X$. $\endgroup$
    – gui11aume
    Commented Jun 13, 2013 at 9:31

1 Answer 1

7
$\begingroup$

Fortunately they are the same. All we need to prove is $E_{\theta} \left[ \frac{\partial}{\partial \theta} \log f(x|\theta) \right] = 0$, so here we go.

First note that

$$\int_{\mathbb{R}} \frac{\partial}{\partial \theta}\log(f(x|\theta)) \cdot f(x|\theta) dx = \int_{\mathbb{R}} \frac{\partial f(x|\theta)}{\partial \theta}dx. $$

Under the conditions of Leibniz's integral rule you can rewrite the above

$$ \frac{\partial}{\partial \theta} \int_{\mathbb{R}} f(x|\theta)dx = \frac{\partial}{\partial \theta} 1 = 0.$$

So, except pathological cases where the conditions mentioned above do not hold, the expected value of the score is 0, and the covariance is the same as the expected value of the product.

$\endgroup$
2
  • 2
    $\begingroup$ Thank you. Could you have meant Leibniz's integral rule rather than Lebesgue's dominated convergence theorem? $\endgroup$
    – Evan Aad
    Commented Jun 13, 2013 at 10:45
  • 2
    $\begingroup$ Indeed! I had in mind point 3 of the measure theory statement of the rule, but of course point 1 and 2 are also required. I will edit my answer accordingly. $\endgroup$
    – gui11aume
    Commented Jun 13, 2013 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.