Confused about notation in definition of Fisher Information matrix

Question

In the definition of the Fisher Information matrix: $$ \begin{align} I(\theta)_{ij} &= \mathbb{E}_{x \sim p(x \,;\, \theta)}\left[ \left(\frac{\partial}{\partial \theta_i} \log p(x \,;\, \theta) \right) \left(\frac{\partial}{\partial \theta_j} \log p(x \,;\, \theta) \right) \right] \end{align} $$ is the expectation on the RHS computed using the same value for "$\theta$" that is passed into $I(\theta)$ ?

Or is the density "$p(x \,;\, \cdot)$" in the expectation computed using the true but unknown parameter value of the parameter, call it $\theta^*$, so that: $$ \begin{align} I\left(\bar{\theta}\right)_{ij} &= {\large \int} \left( \frac{\partial}{\partial \theta_i} \log p(x \,;\, \theta)~\Biggr|_{ \theta=\bar{\theta}} \right) \left( \frac{\partial}{\partial \theta_j} \log p(x \,;\, \theta) ~\Biggr|_{ \theta=\bar{\theta}} \right) \, p(x \,;\, \theta^*) \, dx \end{align} $$

In this second definition, the Fisher information matrix would tell us how much information the true distribution (as specified by $\theta^*$) provides about the value of theta at location $\bar{\theta}$.

Alternatively, if the same value for theta (namely $\bar{\theta}$) is also used in the density "$p(x \; \cdot)$", then the meaning of the Fisher information matrix is something like "how much information does the density specified by $\bar{\theta}$ contain about itself?" And I'm not really sure how that quantity would be useful in practice.

Answer 1

It's the first one: all quantities are evaluated at the true value of $\theta$.

The reason this is the right definition is that $$\frac{\partial}{\partial\theta}\log p(x;\theta)$$ doesn't have mean zero except at the true value, which makes squaring a lot less useful. The information identity (that the variance of the first derivative is equal to the mean of the second derivative) is also only valid at the true value.

Well, strictly speaking, when I say all quantities are evaluated at the true value of $\theta$ what I really mean is evaluated at a $\theta$ we are currently pretending is the true value, whether it is or not. So, the Fisher scoring algorithm modifies the Newton-Raphson algorithm by replacing the actual second-derivative matrix (which might not be positive definite) with the inverse of the information matrix pretending that the current value is the truth (which is guaranteed to be positive semidefinite because it's a variance) .

Answer 2

The Fisher information gives the relation between the true value of $\theta$ & how much information about $\theta$ you'd expect to get from the data. To construct tests or confidence intervals you evaluate it at a hypothesized or estimated value of $\theta$. (It can be flat— regardless of the true value you'd expect to get the same amount of information from the data.)

A simple example: independent counts $x_1, \ldots, x_n$ from a Poisson distribution with mean $\theta$. The mass function for each count is $$ p(x_i;\theta) = \frac{\theta^{x_i}\mathrm{e}^{-\theta}}{x_i!}\,, $$ the score function $$ U(\theta)=\frac{\operatorname{d}\log p(x_1, \ldots, x_n; \theta)}{\operatorname{d}\theta} = \frac{\sum_{i=1}^n x_i - n\theta}{\theta}\,, $$ & the Fisher information $$ I(\theta) = \frac{n}{\theta}\,. $$ To perform Rao's score test for the null hypothesis $\theta=\theta_0$, the Fisher information is evaluated at $\theta_0$, & the test statistic is $$ \frac{U(\theta_0)}{\sqrt{I(\theta_0)}}=\frac{\sum_{i=1}^n x_i - n\theta_0}{\theta_0}\cdot\sqrt\frac{\theta_0}{n} = \left(\frac{\sum_{i=1}^n x_i}{n} -\theta_0\right)\cdot\sqrt\frac{n}{\theta_0}\,. $$

To perform the Wald test the Fisher information is evaluated at the maximum-likelihood estimate $\hat\theta=\frac{\sum_{i=1}^n x}{n}$, & the test statistic is

$$ (\hat\theta-\theta_0)\cdot\sqrt{I(\hat\theta)}= \left(\frac{\sum_{i=1}^n x_i}{n} -\theta_0\right)\cdot\sqrt{\frac{n}{\frac{\sum_{i=1}^n x_i}{n}}}\,. $$