4
$\begingroup$

For a Fisher Information matrix $I(\theta)$ of multiple variables, is it true that $I(\theta) = nI_1(\theta)$? That is, if $\theta = (\theta_1, \ldots, \theta_k)$, will it be the case that the fisher information matrix of multiple parameters for an entire dataset will just be $n$ times the fisher information matrix for the first data point, assuming the data is iid?

Update: As a concrete example, consider a sequence of random variables $y_1, \ldots, y_n$ such that $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ where $\epsilon_i$ is assumed to be i.i.d. $N(0,\sigma^2)$, with $\sigma^2$ known. Additionally, assume that $n$ is even. I am trying to find the Fisher Information Matrix for $\beta = (\beta_1, \beta_2)$. I know that the log-likelihood for one observation is:

$$ l(\beta_0, \beta_1) \propto -\frac{1}{2\sigma^2}(y-\beta_0-\beta_1x)^2 $$

Hence, we have that the observed information matrix (before the expectation), is:

$\frac{\partial^2 l}{\partial \beta_0^2}= \frac{-1}{\sigma^2}$, $\frac{\partial^2 l}{\partial \beta_1^2}= \frac{-x^2}{\sigma^2}$, and $\frac{\partial^2 l}{\partial \beta_0 \partial \beta_1} = \frac{-x}{\sigma^2}$.

Thus the information matrix for a single observation is: $$ -E\left(\frac{\partial^2 l}{\partial \beta^2}\right) =\frac{1}{\sigma^2}\left( \begin{array}{ccc} 1 & x \\ x & x^2 \end{array}\right) $$

and the information matrix for n pairs of observations $(x_i, y_i)$ is given by:

$$ I(\beta_0, \beta_1) =\frac{1}{\sigma^2}\left( \begin{array}{ccc} n & \sum_{i=1}^{n}x_i \\ \sum_{i=1}^{n}x_i & \sum_{i=1}^{n}x_i^2 \end{array}\right) $$

Above, because the fisher information is additive, all we did to move from the single observation to the multiple observation case was just to add entry by entry. HOWEVER, I know that in general if $Y_1, \ldots, Y_n$ are iid, then $I(\theta) = nI_1(\theta)$. My question is, why is it NOT the case above we could have just multiplied each entry by $n$, and instead had to add?

$\endgroup$
5
  • 1
    $\begingroup$ Is your vector of parameters really of the same dimension as your data set, $n$? $\endgroup$ May 6 '16 at 7:01
  • 1
    $\begingroup$ When you say multiple variables, do you really mean multiple (and i.i.d) observations? Or do you really that $\theta$ contains more than one variable (in which case consider the Fisher information for $N(\mu,\sigma^2)$, which has two variables in it). $\endgroup$
    – Glen_b
    May 6 '16 at 7:05
  • $\begingroup$ Thanks for the comments, I mean't to have the number of parameters be $k$, which is not equal to $n$. As for the multiple variables, I mean is that the observations are iid single observations, ie, $y_i \sim N(\mu, \sigma^2)$, where $y_i \in \mathbb{R}$, and that $\theta$ is the one which contains more than one variable, ie, $\theta = (\mu, \sigma^2)$. Thanks and sorry for the confusion! $\endgroup$ May 6 '16 at 8:34
  • 2
    $\begingroup$ Although it could be made more clear by using a different symbol for the observed Fisher information (say J) as the Fisher information (I) as in the wikipedia page: en.wikipedia.org/wiki/Observed_information $\endgroup$
    – Neil G
    May 7 '16 at 5:30
  • 1
    $\begingroup$ Your model is not an iid model if the $x_i$ are to be treated as constants, because $E(y_i)=\beta_0+\beta_1x_i$. $\endgroup$ May 9 '16 at 13:16
3
$\begingroup$

Since the wikipedia article https://en.wikipedia.org/wiki/Fisher_information do not contain a proof, I will write one here. Let $X_1, X_2, \dotsc, X_n$ be independent random variables with density function $f(x;\theta)$ (which might in addition depend on known covariates, so this covers more than the iid case). Then the loglikelihood function is $$ \ell(\theta) = \sum_i \log f(X_i;\theta) $$ and the score function is $$ s(\theta) = \frac{\partial \ell(\theta)}{\partial \theta}= \sum_i\frac{\partial}{\partial\theta}\log f(X;\theta) $$ The Fisher information matrix then can be written $$ \DeclareMathOperator{\E}{\mathbb{E}} I(\theta) = \E\left[\sum_i \left( \frac{\partial}{\partial\theta}\log f(X_i;\theta) \right)\left( \frac{\partial}{\partial\theta}\log f(X_i;\theta) \right)^T \mid \theta\right] $$ and now the result follows by moving the summation sign outside the expectation operator, which shows that $I(\theta)=\sum_i I_i(\theta)$ where $I_i(\theta)$ is the Fisher information from variable $X_i$. In the iid case that becomes $I(\theta)=n I_1(\theta)$.

$\endgroup$
1
  • $\begingroup$ This is at least incomplete, because covariance of different observations $X_i$ and $X_j$ must be taken into account in general. These covariances are zero, because the observations are independent, which is an important point of the proof. $\endgroup$ Dec 11 '20 at 9:29
1
$\begingroup$

The proof by kjetil b halvorsen is incorrect, you missed the cross-terms. The correct equation for the $I(\theta)$ should read something like this: $$ I(\theta) = \mathbb{E}\left[\left(\sum\limits_i\frac{\partial}{\partial\theta}\log f(X_i;\theta)\right)\left(\sum\limits_j\frac{\partial}{\partial\theta}\log f(X_j;\theta)\right)^T | \theta\right] $$

$\endgroup$
1
  • $\begingroup$ I think that your post would better be located as a comment of the original proof, not as a separate answer. $\endgroup$ Dec 12 '20 at 11:56
0
$\begingroup$

Let $X$ be a random variable with probability density function $f(x;\theta)$. Assume that the observations $x_1,\ldots,x_n$ are independent realizations of $X$. Let us prove that the Fisher matrix is: \begin{align} I(\theta) = n I_1(\theta) \end{align} where $I_1(\theta)$ is the Fisher matrix for one single observation: \begin{align} I_1(\theta)_{jk} = \mathbb{E}\left[\left(\frac{\partial \log(f(X_1 ; \theta))}{\partial \theta_j}\right) \left(\frac{\partial \log(f(X_1 ; \theta))}{\partial \theta_k}\right)\right] \end{align} for any $j,k=1,\ldots, m$ and any $\theta \in \mathbb{R}^m$.

Since the observations are independent and have the same PDF, the log-likelihood is: \begin{align} \ell(\theta) = \sum_{i=1}^n \log(f(x_i;\theta)) \end{align} for any $\theta\in\mathbb{R}^m$.

Let $s$ be the score, defined as the gradient of the log-likelihood: \begin{align} s(\theta) = \frac{\partial \ell(\theta)}{\partial \theta} \end{align} for any $\theta\in\mathbb{R}^m$.

Indeed, for any $j,k=1,\ldots, m$, the equation $\mathbb{E}(s(\theta)) = 0$ implies: \begin{align*} I(\theta)_{jk} &= \textbf{Cov}(s(\theta)_j, s(\theta_k)) \\ &= \textbf{Cov}\left(\frac{\partial \ell(\theta)}{\partial \theta_j}, \frac{\partial \ell(\theta)}{\partial \theta_k}\right). \end{align*} The independence of the realizations implies: \begin{align*} I(\theta)_{jk} &= \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \sum_{i_1=1}^n \log(f(X_{i_1};\theta)), \frac{\partial}{\partial \theta_k} \sum_{i_2=1}^n \log(f(X_{i_2};\theta))\right) \\ &= \textbf{Cov}\left(\sum_{i_1=1}^n \frac{\partial}{\partial \theta_j} \log(f(X_{i_1};\theta)), \sum_{i_2=1}^n \frac{\partial}{\partial \theta_k} \log(f(X_{i_2};\theta))\right) \\ &= \sum_{i_1=1}^n \sum_{i_2=1}^n \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_{i_1};\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_{i_2};\theta))\right), \end{align*} for any $j,k=1,\ldots, m$, where the last equation uses the linearity properties of the covariance. However, the observations are independent, therefore, $$ \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_{i_1};\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_{i_2};\theta))\right) = 0 $$ if $i_1 \neq i_2$. Hence, \begin{align*} I(\theta)_{jk} &= \sum_{i=1}^n \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_i;\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_i;\theta))\right). \end{align*} Moreover, since all the observations have the same distribution, \begin{align*} &\textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_i;\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_i;\theta))\right) \\ &= \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_1;\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_1;\theta))\right) \\ &= I_1(\theta) \end{align*} for any $i=1,\ldots, n$. Hence, \begin{align*} I(\theta)_{jk} &= \sum_{i=1}^n I_1(\theta) \end{align*} which concludes the proof.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.