# For a Fisher Information matrix $I(\theta)$ of multiple variables, is it true that $I(\theta) = nI_1(\theta)$?

For a Fisher Information matrix $I(\theta)$ of multiple variables, is it true that $I(\theta) = nI_1(\theta)$? That is, if $\theta = (\theta_1, \ldots, \theta_k)$, will it be the case that the fisher information matrix of multiple parameters for an entire dataset will just be $n$ times the fisher information matrix for the first data point, assuming the data is iid?

Update: As a concrete example, consider a sequence of random variables $y_1, \ldots, y_n$ such that $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ where $\epsilon_i$ is assumed to be i.i.d. $N(0,\sigma^2)$, with $\sigma^2$ known. Additionally, assume that $n$ is even. I am trying to find the Fisher Information Matrix for $\beta = (\beta_1, \beta_2)$. I know that the log-likelihood for one observation is:

$$l(\beta_0, \beta_1) \propto -\frac{1}{2\sigma^2}(y-\beta_0-\beta_1x)^2$$

Hence, we have that the observed information matrix (before the expectation), is:

$\frac{\partial^2 l}{\partial \beta_0^2}= \frac{-1}{\sigma^2}$, $\frac{\partial^2 l}{\partial \beta_1^2}= \frac{-x^2}{\sigma^2}$, and $\frac{\partial^2 l}{\partial \beta_0 \partial \beta_1} = \frac{-x}{\sigma^2}$.

Thus the information matrix for a single observation is: $$-E\left(\frac{\partial^2 l}{\partial \beta^2}\right) =\frac{1}{\sigma^2}\left( \begin{array}{ccc} 1 & x \\ x & x^2 \end{array}\right)$$

and the information matrix for n pairs of observations $(x_i, y_i)$ is given by:

$$I(\beta_0, \beta_1) =\frac{1}{\sigma^2}\left( \begin{array}{ccc} n & \sum_{i=1}^{n}x_i \\ \sum_{i=1}^{n}x_i & \sum_{i=1}^{n}x_i^2 \end{array}\right)$$

Above, because the fisher information is additive, all we did to move from the single observation to the multiple observation case was just to add entry by entry. HOWEVER, I know that in general if $Y_1, \ldots, Y_n$ are iid, then $I(\theta) = nI_1(\theta)$. My question is, why is it NOT the case above we could have just multiplied each entry by $n$, and instead had to add?

• Is your vector of parameters really of the same dimension as your data set, $n$? – Christoph Hanck May 6 '16 at 7:01
• When you say multiple variables, do you really mean multiple (and i.i.d) observations? Or do you really that $\theta$ contains more than one variable (in which case consider the Fisher information for $N(\mu,\sigma^2)$, which has two variables in it). – Glen_b May 6 '16 at 7:05
• Thanks for the comments, I mean't to have the number of parameters be $k$, which is not equal to $n$. As for the multiple variables, I mean is that the observations are iid single observations, ie, $y_i \sim N(\mu, \sigma^2)$, where $y_i \in \mathbb{R}$, and that $\theta$ is the one which contains more than one variable, ie, $\theta = (\mu, \sigma^2)$. Thanks and sorry for the confusion! – user1398057 May 6 '16 at 8:34
• Although it could be made more clear by using a different symbol for the observed Fisher information (say J) as the Fisher information (I) as in the wikipedia page: en.wikipedia.org/wiki/Observed_information – Neil G May 7 '16 at 5:30
• Your model is not an iid model if the $x_i$ are to be treated as constants, because $E(y_i)=\beta_0+\beta_1x_i$. – Christoph Hanck May 9 '16 at 13:16

Since the wikipedia article https://en.wikipedia.org/wiki/Fisher_information do not contain a proof, I will write one here. Let $X_1, X_2, \dotsc, X_n$ be independent random variables with density function $f(x;\theta)$ (which might in addition depend on known covariates, so this covers more than the iid case). Then the loglikelihood function is $$\ell(\theta) = \sum_i \log f(X_i;\theta)$$ and the score function is $$s(\theta) = \frac{\partial \ell(\theta)}{\partial \theta}= \sum_i\frac{\partial}{\partial\theta}\log f(X;\theta)$$ The Fisher information matrix then can be written $$\DeclareMathOperator{\E}{\mathbb{E}} I(\theta) = \E\left[\sum_i \left( \frac{\partial}{\partial\theta}\log f(X_i;\theta) \right)\left( \frac{\partial}{\partial\theta}\log f(X_i;\theta) \right)^T \mid \theta\right]$$ and now the result follows by moving the summation sign outside the expectation operator, which shows that $I(\theta)=\sum_i I_i(\theta)$ where $I_i(\theta)$ is the Fisher information from variable $X_i$. In the iid case that becomes $I(\theta)=n I_1(\theta)$.

• This is at least incomplete, because covariance of different observations $X_i$ and $X_j$ must be taken into account in general. These covariances are zero, because the observations are independent, which is an important point of the proof. – Michael Baudin Dec 11 '20 at 9:29

The proof by kjetil b halvorsen is incorrect, you missed the cross-terms. The correct equation for the $$I(\theta)$$ should read something like this: $$I(\theta) = \mathbb{E}\left[\left(\sum\limits_i\frac{\partial}{\partial\theta}\log f(X_i;\theta)\right)\left(\sum\limits_j\frac{\partial}{\partial\theta}\log f(X_j;\theta)\right)^T | \theta\right]$$

• I think that your post would better be located as a comment of the original proof, not as a separate answer. – Michael Baudin Dec 12 '20 at 11:56

Let $$X$$ be a random variable with probability density function $$f(x;\theta)$$. Assume that the observations $$x_1,\ldots,x_n$$ are independent realizations of $$X$$. Let us prove that the Fisher matrix is: \begin{align} I(\theta) = n I_1(\theta) \end{align} where $$I_1(\theta)$$ is the Fisher matrix for one single observation: \begin{align} I_1(\theta)_{jk} = \mathbb{E}\left[\left(\frac{\partial \log(f(X_1 ; \theta))}{\partial \theta_j}\right) \left(\frac{\partial \log(f(X_1 ; \theta))}{\partial \theta_k}\right)\right] \end{align} for any $$j,k=1,\ldots, m$$ and any $$\theta \in \mathbb{R}^m$$.

Since the observations are independent and have the same PDF, the log-likelihood is: \begin{align} \ell(\theta) = \sum_{i=1}^n \log(f(x_i;\theta)) \end{align} for any $$\theta\in\mathbb{R}^m$$.

Let $$s$$ be the score, defined as the gradient of the log-likelihood: \begin{align} s(\theta) = \frac{\partial \ell(\theta)}{\partial \theta} \end{align} for any $$\theta\in\mathbb{R}^m$$.

Indeed, for any $$j,k=1,\ldots, m$$, the equation $$\mathbb{E}(s(\theta)) = 0$$ implies: \begin{align*} I(\theta)_{jk} &= \textbf{Cov}(s(\theta)_j, s(\theta_k)) \\ &= \textbf{Cov}\left(\frac{\partial \ell(\theta)}{\partial \theta_j}, \frac{\partial \ell(\theta)}{\partial \theta_k}\right). \end{align*} The independence of the realizations implies: \begin{align*} I(\theta)_{jk} &= \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \sum_{i_1=1}^n \log(f(X_{i_1};\theta)), \frac{\partial}{\partial \theta_k} \sum_{i_2=1}^n \log(f(X_{i_2};\theta))\right) \\ &= \textbf{Cov}\left(\sum_{i_1=1}^n \frac{\partial}{\partial \theta_j} \log(f(X_{i_1};\theta)), \sum_{i_2=1}^n \frac{\partial}{\partial \theta_k} \log(f(X_{i_2};\theta))\right) \\ &= \sum_{i_1=1}^n \sum_{i_2=1}^n \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_{i_1};\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_{i_2};\theta))\right), \end{align*} for any $$j,k=1,\ldots, m$$, where the last equation uses the linearity properties of the covariance. However, the observations are independent, therefore, $$\textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_{i_1};\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_{i_2};\theta))\right) = 0$$ if $$i_1 \neq i_2$$. Hence, \begin{align*} I(\theta)_{jk} &= \sum_{i=1}^n \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_i;\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_i;\theta))\right). \end{align*} Moreover, since all the observations have the same distribution, \begin{align*} &\textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_i;\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_i;\theta))\right) \\ &= \textbf{Cov}\left(\frac{\partial}{\partial \theta_j} \log(f(X_1;\theta)), \frac{\partial}{\partial \theta_k} \log(f(X_1;\theta))\right) \\ &= I_1(\theta) \end{align*} for any $$i=1,\ldots, n$$. Hence, \begin{align*} I(\theta)_{jk} &= \sum_{i=1}^n I_1(\theta) \end{align*} which concludes the proof.