2
$\begingroup$

For a Fisher Information matrix $I(\theta)$ of multiple variables, is it true that $I(\theta) = nI_1(\theta)$? That is, if $\theta = (\theta_1, \ldots, \theta_k)$, will it be the case that the fisher information matrix of multiple parameters for an entire dataset will just be $n$ times the fisher information matrix for the first data point, assuming the data is iid?

Update: As a concrete example, consider a sequence of random variables $y_1, \ldots, y_n$ such that $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ where $\epsilon_i$ is assumed to be i.i.d. $N(0,\sigma^2)$, with $\sigma^2$ known. Additionally, assume that $n$ is even. I am trying to find the Fisher Information Matrix for $\beta = (\beta_1, \beta_2)$. I know that the log-likelihood for one observation is:

$$ l(\beta_0, \beta_1) \propto -\frac{1}{2\sigma^2}(y-\beta_0-\beta_1x)^2 $$

Hence, we have that the observed information matrix (before the expectation), is:

$\frac{\partial^2 l}{\partial \beta_0^2}= \frac{-1}{\sigma^2}$, $\frac{\partial^2 l}{\partial \beta_1^2}= \frac{-x^2}{\sigma^2}$, and $\frac{\partial^2 l}{\partial \beta_0 \partial \beta_1} = \frac{-x}{\sigma^2}$.

Thus the information matrix for a single observation is: $$ -E\left(\frac{\partial^2 l}{\partial \beta^2}\right) =\frac{1}{\sigma^2}\left( \begin{array}{ccc} 1 & x \\ x & x^2 \end{array}\right) $$

and the information matrix for n pairs of observations $(x_i, y_i)$ is given by:

$$ I(\beta_0, \beta_1) =\frac{1}{\sigma^2}\left( \begin{array}{ccc} n & \sum_{i=1}^{n}x_i \\ \sum_{i=1}^{n}x_i & \sum_{i=1}^{n}x_i^2 \end{array}\right) $$

Above, because the fisher information is additive, all we did to move from the single observation to the multiple observation case was just to add entry by entry. HOWEVER, I know that in general if $Y_1, \ldots, Y_n$ are iid, then $I(\theta) = nI_1(\theta)$. My question is, why is it NOT the case above we could have just multiplied each entry by $n$, and instead had to add?

$\endgroup$
  • 1
    $\begingroup$ Is your vector of parameters really of the same dimension as your data set, $n$? $\endgroup$ – Christoph Hanck May 6 '16 at 7:01
  • 1
    $\begingroup$ When you say multiple variables, do you really mean multiple (and i.i.d) observations? Or do you really that $\theta$ contains more than one variable (in which case consider the Fisher information for $N(\mu,\sigma^2)$, which has two variables in it). $\endgroup$ – Glen_b -Reinstate Monica May 6 '16 at 7:05
  • 1
    $\begingroup$ This question should be reopened. It seems clear to me. $\endgroup$ – Neil G May 7 '16 at 5:29
  • 2
    $\begingroup$ Although it could be made more clear by using a different symbol for the observed Fisher information (say J) as the Fisher information (I) as in the wikipedia page: en.wikipedia.org/wiki/Observed_information $\endgroup$ – Neil G May 7 '16 at 5:30
  • 1
    $\begingroup$ Your model is not an iid model if the $x_i$ are to be treated as constants, because $E(y_i)=\beta_0+\beta_1x_i$. $\endgroup$ – Christoph Hanck May 9 '16 at 13:16
3
$\begingroup$

Since the wikipedia article https://en.wikipedia.org/wiki/Fisher_information do not contain a proof, I will write one here. Let $X_1, X_2, \dotsc, X_n$ be independent random variables with density function $f(x;\theta)$ (which might in addition depend on known covariates, so this covers more than the iid case). Then the loglikelihood function is $$ \ell(\theta) = \sum_i \log f(X_i;\theta) $$ and the score function is $$ s(\theta) = \frac{\partial \ell(\theta)}{\partial \theta}= \sum_i\frac{\partial}{\partial\theta}\log f(X;\theta) $$ The Fisher information matrix then can be written $$ \DeclareMathOperator{\E}{\mathbb{E}} I(\theta) = \E\left[\sum_i \left( \frac{\partial}{\partial\theta}\log f(X_i;\theta) \right)\left( \frac{\partial}{\partial\theta}\log f(X_i;\theta) \right)^T \mid \theta\right] $$ and now the result follows by moving the summation sign outside the expectation operator, which shows that $I(\theta)=\sum_i I_i(\theta)$ where $I_i(\theta)$ is the Fisher information from variable $X_i$. In the iid case that becomes $I(\theta)=n I_1(\theta)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.