4
$\begingroup$

I am currently studying the textbook In All Likelihood by Yudi Pawitan. Example 2.10 of chapter 2.5 Maximum and curvature of likelihood says the following:

Example 2.10: Based on $x$ from the binomial$(n, \theta)$ the log-likelihood function is $$\log L(\theta) = x \log \theta + (n - x) \log(1 - \theta).$$ We can first find the score function $$\begin{align} S(\theta) &\equiv \frac{\partial}{\partial{\theta}} \log L(\theta) \\ &= \frac{x}{\theta} - \frac{n - x}{1 - \theta}, \end{align}$$ giving the MLE $\hat{\theta} = x/n$ and $$\begin{align} I(\theta) &\equiv -\frac{\partial^2}{\partial{\theta}^2} \log L(\theta) \\ &= \frac{x}{\theta^2} + \frac{n - x}{(1 - \theta)^2}, \end{align}$$ so at the MLE we have the Fisher information $$I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}. \square$$

How is $I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$ calculated? I know that the MLE $\hat{\theta}$ is the solution of the score equation $S(\theta) \equiv \frac{\partial}{\partial{\theta}} \log L(\theta) = 0$, and I know that $I(\theta) \equiv - \frac{\partial^2}{\partial{\theta}^2} \log L(\theta)$ is the Fisher information, but I can't figure out precisely what $I(\hat{\theta})$ is, or exactly how $I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$ was calculated.

$\endgroup$
2

2 Answers 2

6
$\begingroup$

Hint: Substitute the value $\hat\theta=x/n$ in $I(\theta) .$ See what happens.

Substitute in place of $\theta$ to get $$\frac{n^2x}{x^2}+ \frac{n^2(n-x) }{(n-x) ^2}.$$ Simplify it.

$\endgroup$
1
  • $\begingroup$ Thanks, I think I just fixed it. With regards to your hint, there is no $\hat{\theta}$ in $I(\theta)$, so are you saying to substitute $x = \hat{\theta}n$? But, if we do that, then we don't have any way to eliminate the $\theta$ in $I(\theta)$. So, as you can see, I'm confused as to what we're supposed to do here. $\endgroup$ Commented Jan 19, 2023 at 14:49
5
$\begingroup$

You forgot the expectation operator in the definition of Fisher information: $I(\theta) = E_\theta\left[-\frac{\partial^2}{\partial\theta^2}\log L(\theta)\right]$ instead of what you posted. After correcting this and noting $E_\theta(x) = n\theta$ given $x \sim B(n, \theta)$, it turns out that \begin{align} & I(\theta) = E_\theta\left[-\frac{\partial^2}{\partial\theta^2}\log L(\theta)\right] \\ =& \frac{1}{\theta^2}E_\theta(x) + \frac{1}{(1 - \theta)^2}E_\theta(n - x) \\ =& \frac{n}{\theta} + \frac{n}{1 - \theta} \\ =& \frac{n}{\theta(1 - \theta)}, \end{align} matching the goal.


Through the discussion below, it seems that the reference you quoted fails to clearly present the concept of "Fisher information" (and other related sample-level concepts). I will try to clarify them based on Section 4.3 of Statistical Models by A. C. Davison.

Suppose $N$ i.i.d. observations $x_1, \ldots, x_N$ are drawn from the distribution $f(x; \theta)$, so that the log-likelihood is \begin{align} \ell(\theta) = \log L(\theta) = \sum_{i = 1}^N \log f(x_i; \theta). \end{align}

Fisher information (or expected information) is defined by \begin{align} I(\theta) = E\left[-\frac{\partial^2\ell(\theta)}{\partial\theta^2}\right]. \tag{1} \end{align}

While the observed information is defined as: \begin{align} J(\theta) = -\frac{\partial^2\ell(\theta)}{\partial\theta^2} = -\sum_{i = 1}^N \frac{\partial^2 \log f(x_i; \theta)}{\partial\theta^2}. \tag{2} \end{align}

Note that, if observations are i.i.d., Fisher information only contains parameter $\theta$ (i.e., no observations are included, but the sample size $N$ is still included).

For your example, you have a single observation $x \sim B(n, \theta)$. In this case, your post and my calculation above showed: \begin{align} & I(\theta) = \frac{n}{\theta(1 - \theta)}. \tag{3} \\ & J(\theta) = \frac{x}{\theta^2} + \frac{n - x}{(1 - \theta)^2}. \tag{4} \end{align}

To evaluate "Fisher information evaluated at MLE $\hat{\theta}$", simply plug in $\hat{\theta}$ into $(3)$ (just like evaluating a function's value at a given point) to get $I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$. Even though $J(\hat{\theta}) = I(\hat{\theta})$ in this case, that doesn't mean $J(\hat{\theta})$ is conceptually "Fisher information evaluated at MLE $\hat{\theta}$".

$\endgroup$
13
  • 1
    $\begingroup$ This is the way I would generally do and learnt. But the book defined observed $I(\theta) $ without the expectation. I need to digest that. $\endgroup$ Commented Jan 19, 2023 at 14:52
  • 1
    $\begingroup$ @ThePointer What your textbook put is $J(\theta)$ with one observation $x$ at hand, which in my opinion, is not "Fisher information". Although in this case, $J(\hat{\theta}) = I(\hat{\theta})$, in general, this is not the case (suppose you observed $x_1$ and $x_2$). $\endgroup$
    – Zhanxiong
    Commented Jan 19, 2023 at 15:06
  • 1
    $\begingroup$ I agree with you @Zhanxiong. Seeing initially made me nervous. I think your comment stating $J(\hat{\theta}) = I(\hat{\theta})$ is not the general case is useful for OP. May I urge you to add this in your post? $\endgroup$ Commented Jan 19, 2023 at 15:13
  • 1
    $\begingroup$ @User1865345 After some calculation, for binomial case, $I(\hat{\theta}) = J(\hat{\theta})$ is indeed true for all numbers of observations (I apologize for previous pre-matured comment). However, this doesn't change my general view on this concept (even he used $\mathcal{I}$ for Fisher information and $I$ for observed information, the problem as stated in my previous comment still stands). $\endgroup$
    – Zhanxiong
    Commented Jan 19, 2023 at 15:57
  • 1
    $\begingroup$ @User1865345 Done per your urge! $\endgroup$
    – Zhanxiong
    Commented Jan 19, 2023 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.