9
$\begingroup$

The fisher information's connection with the negative expected hessian at $\theta_{MLE}$, provides insight in the following way: at the MLE, high curvature implies that an estimate of $\theta$ even slightly different from the true MLE would have resulted in a very different likelihood. $$ \mathbf{I}(\theta)=-\frac{\partial^{2}}{\partial\theta_{i}\partial\theta_{j}}l(\theta),~~~~ 1\leq i, j\leq p $$

This is good, as that means that we can be relatively sure about our estimate.

The other connection of Fisher information to variance of the score, when evaluated at the MLE is less clear to me. $$ I(\theta) = E[(\frac{\partial}{\partial\theta}l(\theta))^2]$$

The implication is; high Fisher information -> high variance of score function at the MLE.

Intuitively, this means that the score function is highly sensitive to the sampling of the data. i.e - we are likely to get a non-zero gradient of the likelihood, had we sampled a different data distribution. This seems to have a negative implication to me. Don't we want the score function = 0 to be highly robust to different sampling of the data?

A lower fisher information on the other hand, would indicate the score function has low variance at the MLE, and has mean zero. This implies that regardless of the sampling distribution, we will get a gradient of log likelihood to be zero (which is good!).

What am I missing?

$\endgroup$
2
9
$\begingroup$

Intuitively, this means that the score function is highly sensitive to the sampling of the data. i.e - we are likely to get a non-zero gradient of the likelihood, had we sampled a different data distribution. This seems to have a negative implication to me. Don't we want the score function = 0 to be highly robust to different sampling of the data?

That is not the correct intuition for the score function. Remember that the score function is a derivative with respect to the parameter, not the data. The Fisher information is defined as the variance of the score, but under simple regularity conditions it is also the negative of the expected value of the second derivative of the log-likelihood. So, if we write the log-likelihood as $\ell(\theta | \mathbf{X})$ and the score function as $s(\theta | \mathbf{X})$ (i.e., with explicit conditioning on data $\mathbf{X}$) then the Fisher information is:

$$\mathcal{I}(\theta) = -\mathbb{E} \Bigg( \frac{\partial^2 \ell}{\partial \theta^2} (\theta | \mathbf{X}) \Bigg) = -\mathbb{E} \Bigg( \frac{\partial s}{\partial \theta} (\theta | \mathbf{X}) \Bigg).$$

The thing to note here is that the derivatives are taken with respect to the parameter, not the data. So, we can see that a high (magnitude) value for the Fisher information means that the score function is, on average, highly sensitive to the parameter value, not the data. If the score function is highly sensitive to the parameter value, this means that the root of the equation (which is the MLE) is relatively insensitive to the parameter, and so the MLE has lower variance.

$\endgroup$
4
  • $\begingroup$ I understand the derivative itself is w.r.t to the parameter in the score function. My intuition stemmed from the variance of the score function, where the expectation is taken w.r.t to the data, and not the parameter I believe? $\endgroup$ Jul 18 at 15:45
  • $\begingroup$ Yes, that is true, but remember that the magnitude of the expectation is largely determined by the magnitude of the partial derivative with respect to the parameter. Expectation with respect to the data is essentially just a form of weighted averaging of these partial derivatives, so a high magnitude still suggests that the score function is sensitive to the parameter. $\endgroup$
    – Ben
    Jul 19 at 21:59
  • $\begingroup$ Also, to add further intuition to what Ben said. When the log-likelihood is concave, we have $ \frac{\partial^2l}{\partial \theta^2}(\theta|X) \leq 0$. So $-E\Big( \frac{\partial^2l}{\partial \theta^2}(\theta|X) \Big) = E\Big| \frac{\partial^2l}{\partial \theta^2}(\theta|X) \Big|$ also measures the "variance" of $ \frac{\partial^2l}{\partial \theta^2}(\theta|X) $ in some sense. $\endgroup$
    – user327671
    Jul 20 at 0:27
  • $\begingroup$ @LarsvanderLaan: Interesting point. Also, even if the log-likelihood is not concave, it should be locally concave at the MLE since the latter is a local maxima. So it is even more likely to be concave in a neighbourhood of the MLE than in general. $\endgroup$
    – Ben
    Jul 20 at 4:51
5
+100
$\begingroup$

Depending on which definition of the Fischer information you use, your intuition can mislead you. I will argue that one can make equally valid nonrigorous intuitive arguments that large Fischer information is bad or good. But, in the end, it is really mathematics that gives the correct answer.

Let $l(\theta|X) := \frac{d}{d\theta} \log p_{\theta}(X)$ denote the score function of some parametric density $p_{\theta}$. For simplicity, we assume $\theta$ is real-valued.

Definition 1 and its Intuition (High Fischer information is "bad"!)

One definition of Fischer information is $$I(\theta_0) = Var_{\theta_0} \Big[l(\theta_0|X) \Big].$$ Noting that $\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) \approx_d N(0, I(\theta_0)/n)$, this would mean that the empirical score equation at $\theta = \theta_0$ has larger variance as the Fischer information increases. Since the MLE $\theta_n$ is obtained by solving $\frac{1}{n}\sum_{i=1}^n l(\theta_n|X_i) = 0$, a smaller Fischer information, and therefore having the score $\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i)$ being less variable, would seem like a good thing. A small variance would mean that $\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) \approx 0$ with high probability and thus $\theta_0$ is an approximate MLE, right? This intuition is incorrect and it is based upon the flawed assumption that if $\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i)$ is close to $0=\frac{1}{n}\sum_{i=1}^n l(\theta_n|X_i)$ then $\theta_n$ is close to $\theta_0$. Note to argue that the two scores being close implies that $\theta_n$ and $\theta_0$ are close requires performing a taylor expansion where the derivative $\frac{1}{n}\sum_{i=1}^n \frac{dl}{d\theta}(\theta_0|X_i) \approx E_{\theta_0}\Big[\frac{dl}{d\theta}(\theta_0|X_i)\Big]$ will appear. The larger this derivative is the better a bound we can obtain on $|\theta_n - \theta_0|$. However, by the next definition, the variance of the score is equal to the expectation of its derivative, and therefore decreasing the Fischer information leads to a smaller variance of the score (which is a "good" thing) but at the same time must lead to a decrease of the derivative of the score (which is a "bad" thing). We will revisit this argument later in more detail.

Definition 2 and its Intuition (High Fischer information is "good"!)

The next equivalent definition of Fischer information is $$I(\theta_0) = -E_{\theta_0}\Big[\frac{dl}{d\theta}(\theta_0|X)\Big] = E_{\theta_0}\Big[\frac{d^2 \log p_{\theta}}{d\theta^2}(X)\big|_{\theta = \theta_0}\Big].$$

This definition intuitively suggests that when the Fischer information is large that the score $\theta \mapsto \frac{1}{n}\sum_{i=1}^n\frac{dl}{d\theta}(\theta|X_i)$ is highly sensitive to the value of $\theta$. In particular, it suggests that if $\theta$ has a score close to $0$ then it must be fairly close to $\theta_0$. Since the MLE attempts to solve $\frac{1}{n}\sum_{i=1}^n\frac{dl}{d\theta}(\theta_n|X_i) = 0$, this suggests the MLE will have a smaller variance as the Fischer information increases. However, this intuition implicitly assumes that $\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i)$ remains close to $0$ with high probability as the derivative of the score increases. However, as we will see, this is also not true. So this intuition is also flawed (although, this intuition by happenstance does give the correct answer).

Which intuition is correct?

So the above two intuitions for the different definitions of the Fischer information seem to be odds with one another. The truth is both these intuitions are not taking into account the full picture. This is the danger of nonrigorous intuitive arguments. The best way to solve this is to make the arguments in both intuitions rigorous.

I will derive that the asymptotic variance of the (scaled and centered) MLE $\sqrt{n}(\theta_n - \theta_0)$ is given by the following. See the proof afterward for why this formula of the variance of the MLE is quite natural/canonical (in comparison to $1/I(\theta_0)$). For the sake of the argument, pretend you did not know that $Var_{\theta_0}\big( l(\theta_0|X_i)\big) = E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X_i) \big]$, which my proof does not assume (nor did the intuitive arguments make use of).

The asymptotic variance of $\sqrt{n}\Big( \theta_0 - \theta_n\Big)$ is $$\sigma^2 = \frac{Var_{\theta_0}\big( l(\theta_0|X)\big)}{E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big]^2}.$$ We can now explain what went wrong/right with the two "intuitive" arguments. From the above, it is clear that $Var_{\theta_0}\big( l(\theta_0|X)\big)$ increasing (with all else held fixed) leads to a higher variance of the MLE ("BAD"). On the other hand, if we increase $E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big]$ (with all else held fixed), we will get a smaller variance of the MLE ("GOOD"). Thus, our intuitive arguments, which implicitly assume all else is held fixed, are not flawed perse. However, they are made in a world that doesn't exist: the imaginary world where $-E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big]$ and $Var_{\theta_0}\big( l(\theta_0|X)\big)$ are not connected (i.e. variation independent). But, the fact of the matter is, that these two objects are equal to one another and are equal to the Fischer information $I(\theta_0)$. So in fact, we have $$\sigma^2 = \frac{Var_{\theta_0}\big( l(\theta_0|X)\big)}{E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big]^2} = \frac{1}{I(\theta_0)},$$ which implies that larger Fischer information is indeed good and leads to a smaller variance of the MLE. But, none of the intuitive arguments made this the obvious result to me.

A quick semi-rigorous intuitive argument

Let $\theta_n$ be the MLE. We know $\frac{1}{n}\sum_{i=1}^n l(\theta_n|X_i) = 0$. We also know $\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) \approx_d N\Big(0, Var_{\theta_0}(l(\theta_0|X)/n) \Big).$ Thus, the two scores are usually around $$\frac{1}{n}\sum_{i=1}^n \big[l(\theta_0|X_i) - l(\theta_n|X_i)\big] \approx \sqrt{Var_{\theta_0}(l(\theta_0|X)) /n}$$ apart. We also know that $$ l(\theta_0|X) - l(\theta_n|X) \approx \frac{dl}{d\theta}(\theta_0|X)(\theta_0 - \theta_n).$$ So $\theta_n - \theta_0$ is usually around $$\theta_n - \theta_0 \approx \frac{\frac{1}{n}\sum_{i=1}^n \big[l(\theta_0|X_i) - l(\theta_n|X_i)\big]}{E_{\theta_0}\frac{dl}{d\theta}(\theta_0|X)}$$ apart. So, we get $$\theta_n - \theta_0 \approx \frac{\sqrt{Var_{\theta_0}(l(\theta_0|X))/n }\big]}{E_{\theta_0}\frac{dl}{d\theta}(\theta_0|X)},$$ where the RHS happens to equal $\frac{1}{\sqrt{I(\theta_0)}}$.

A mostly rigorous proof

I will be slightly nonrigorous here. Let $\theta_n$ be the MLE. We then have $$\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) = \frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) - \frac{1}{n}\sum_{i=1}^n l(\theta_n|X_i) ,$$ since $\frac{1}{n}\sum_{i=1}^n l(\theta_n|X_i) = 0$. By performing a taylor expansion of the RHS, we find $$ \frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) - \frac{1}{n}\sum_{i=1}^n l(\theta_n|X_i) = \frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) - l(\theta_n|X_i) = \frac{1}{n}\sum_{i=1}^n \frac{dl}{d\theta}(\theta_0|X_i)\Big( \theta_0 - \theta_n\Big) + R_n, $$ where the remainder $R_n = o_P(\theta_0 - \theta_n)$ satisfies $\frac{R_n}{|\theta_0 - \theta_n|} \rightarrow_n 0. $ Putting it together, we have so far shown $$\frac{1}{n}\sum_{i=1}^n l(\theta_0|X_i) = \Big( \theta_0 - \theta_n\Big) \frac{1}{n}\sum_{i=1}^n \frac{dl}{d\theta}(\theta_0|X_i)+ R_n,$$ which implies $$\frac{1}{\sqrt{n}}\sum_{i=1}^n l(\theta_0|X_i) = \sqrt{n}\Big( \theta_0 - \theta_n\Big) \frac{1}{n}\sum_{i=1}^n \frac{dl}{d\theta}(\theta_0|X_i)+ \sqrt{n}R_n.$$ Applying the law of large numbers to $\frac{1}{n}\sum_{i=1}^n \frac{dl}{d\theta}(\theta_0|X_i)$, we find $$\frac{1}{\sqrt{n}}\sum_{i=1}^n l(\theta_0|X_i) = \sqrt{n}\Big( \theta_0 - \theta_n\Big) E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X_i) \big] + \sqrt{n}\tilde{R}_n,$$ where $\sqrt{n}\tilde{R}_n = o_P(\sqrt{n}|\theta_n - \theta_0|) = o_P(1)$ if we assume $\theta_n$ is $\sqrt{n}$-consistent (as is usually the case). Now, since $ l(\theta_0|X)$ is mean-zero, we can apply the CLT to find $$N\Big(0, Var_{\theta_0}\big( l(\theta_0|X_i)\big) \Big) \approx_d \sqrt{n}\Big( \theta_0 - \theta_n\Big) E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X_i) \big]$$ which implies $$\sqrt{n}\Big( \theta_0 - \theta_n\Big) \approx_d N\Bigg(0, \frac{Var_{\theta_0}\big( l(\theta_0|X_i)\big)}{E_{\theta_0}\big[\frac{dl}{d\theta}(\theta_0|X_i) \big]^2} \Bigg).$$

We have just derivated the limiting distribution of $\sqrt{n}\Big( \theta_0 - \theta_n\Big)$ but notice the form of the RHS.

Model Mispecification

In a mispecified model, where the true data-generating distribution $p_0$ is not equal to $p_{\theta}$ for any $\theta$, the identity $E_{p_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big] = Var_{p_0}\big( l(\theta_0|X)\big)$ is no longer true where $\theta_0$ is such that $p_{\theta_0}$ is the best model approximation of the true density $p_0$. The MLE is still well-defined and is a consistent estimator for $\theta_0$ with asymptotic variance given by $$\sigma^2 = \frac{Var_{p_0}\big( l(\theta_0|X)\big)}{E_{p_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big]^2}.$$

In this case, we are free to change the true density $p_0$ as we see fit. So in principle, we may actually be able to choose $\theta_0$ and $p_0$ in a way such that $Var_{p_0}\big( l(\theta_0|X)\big)$ decreases while $E_{p_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big]$ increases. In this case, both intuitive arguments are then correct. We want a smaller (true) variance of the score $ l(\theta_0|X)$ and we want a larger expectation of the derivative $\frac{dl}{d\theta}(\theta_0|X)$. Note in this case, the Fischer information is defined to be $I(\theta_0) := -E_{p_0}\big[\frac{dl}{d\theta}(\theta_0|X) \big]$, in which case, a larger magnitude Fischer information is still good!

This example especially highlights how subtle the interpretation of the Fischer information really can be in the correctly specified case depending on the definition you use.

Here is a nonrigorous "intuitive" argument for the MLE has smaller variance when the variance of the score is larger

The following argument is super nonrigorous as to make it rigorous would require some heavy-duty smooth functional analysis.

Let $\theta$ be extended to a functional as $\theta(\log p_{\theta'}) := \theta'$. Assuming $\theta$ is a one-to-one parameterization of the statistical model. Then, $\theta$ is a well-defined invertible mapping.

Suppose that $$\frac{d}{d\theta} \log p_{\theta}(X) \Big |_{\theta = \theta_0}$$ is large in magnitude (e.g. high variance, which is a type of norm). This suggests (by the inverse function theorem) that $$\frac{d\theta(\log p_{\theta})}{d \log p_{\theta}}(X) \Big |_{\log p_{\theta} = \log p_{\theta_0}} = \frac{1}{\frac{d}{d\theta} \log p_{\theta}(X) \Big |_{\theta = \theta_0}}$$ is small in magnitude. In particular, $\theta$, viewed as a function of $\log p \in \left\{ \log p_{\theta}: \theta \in \Theta\right\}$, is a smooth/flat mapping at $\log p_{\theta_0}$. In particular, this suggests if $\log p_{\theta_0}$ is close to $\log p_{\theta_n}$ in some sense (e.g expectation) then $\theta_0$ is close to $\theta_n$. Noting that $\theta_n$, being an MLE, attempts to make $\frac{1}{n}\sum_{i=1}^n \log p_{\theta_n}(X_i) \approx E_{\theta_0} \log p_{\theta_n}(X) $ close to $\frac{1}{n}\sum_{i=1}^n \log p_{\theta_0}(X_i) \approx E_{\theta_0} \log p_{\theta_0}(X)$, this would suggest the MLE $\theta_n$ is close to $\theta_0$ when the variance/magnitude of the score is large. In other words, higher Fischer information makes the MLE be "closer" to $\theta_0$.

A similar argument works for the other definition (and is essentially already mentioned in the other answers).

$\endgroup$
4
$\begingroup$

I think it helps to consider a situation where the two quantities are different

Suppose $U(\theta)$ is an unbiased estimating function, so that $$E_{P_{\theta_0}}[U(\theta_0)]=0$$ and you solve $$U(\theta)=0$$ to estimate $\theta$.

The variance of $\hat\theta$ (assuming enough smoothness) is $H^{-1}JH^{-1}$, where the sensitivity matrix $H$ is $$H=\frac{\partial U}{\partial \theta}$$ and the variability matrix $J$ is $$J=\mathrm{var}[U].$$

We clearly want to have $H$ as large as possible and $J$ as small as possible, but we are limited by the ways they are connected. For example, if we multiply $U$ by a scalar $c$, we multiple $H$ by the same $c$ and $J$ by $c^2$, and there is no net effect.

For maximum likelihood estimators we have $H=J$ and they are both the Fisher information. This isn't telling us that we want $J$ to be large. We want $H$ to be large and $J$ to be small, but the best we can do in that direction is to have them be equal. If you have some other unbiased estimating function, you can scale it so that $H$ equals the Fisher information, and $J$ will then be larger than the Fisher information, giving a larger total variance for $\hat\theta$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.