# Observed Fisher information for the binomial: How is $I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$ calculated?

I am currently studying the textbook In All Likelihood by Yudi Pawitan. Example 2.10 of chapter 2.5 Maximum and curvature of likelihood says the following:

Example 2.10: Based on $$x$$ from the binomial$$(n, \theta)$$ the log-likelihood function is $$\log L(\theta) = x \log \theta + (n - x) \log(1 - \theta).$$ We can first find the score function \begin{align} S(\theta) &\equiv \frac{\partial}{\partial{\theta}} \log L(\theta) \\ &= \frac{x}{\theta} - \frac{n - x}{1 - \theta}, \end{align} giving the MLE $$\hat{\theta} = x/n$$ and \begin{align} I(\theta) &\equiv -\frac{\partial^2}{\partial{\theta}^2} \log L(\theta) \\ &= \frac{x}{\theta^2} + \frac{n - x}{(1 - \theta)^2}, \end{align} so at the MLE we have the Fisher information $$I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}. \square$$

How is $$I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$$ calculated? I know that the MLE $$\hat{\theta}$$ is the solution of the score equation $$S(\theta) \equiv \frac{\partial}{\partial{\theta}} \log L(\theta) = 0$$, and I know that $$I(\theta) \equiv - \frac{\partial^2}{\partial{\theta}^2} \log L(\theta)$$ is the Fisher information, but I can't figure out precisely what $$I(\hat{\theta})$$ is, or exactly how $$I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$$ was calculated.

• Small hint: $I(\theta)=n\left\{\frac{x/n}{\theta^2}+\frac{(n-x)/n}{(1-\theta)^2}\right\}$. Commented Jan 19, 2023 at 15:08
• Commented Jan 19, 2023 at 15:19

Hint: Substitute the value $$\hat\theta=x/n$$ in $$I(\theta) .$$ See what happens.

Substitute in place of $$\theta$$ to get $$\frac{n^2x}{x^2}+ \frac{n^2(n-x) }{(n-x) ^2}.$$ Simplify it.

• Thanks, I think I just fixed it. With regards to your hint, there is no $\hat{\theta}$ in $I(\theta)$, so are you saying to substitute $x = \hat{\theta}n$? But, if we do that, then we don't have any way to eliminate the $\theta$ in $I(\theta)$. So, as you can see, I'm confused as to what we're supposed to do here. Commented Jan 19, 2023 at 14:49

You forgot the expectation operator in the definition of Fisher information: $$I(\theta) = E_\theta\left[-\frac{\partial^2}{\partial\theta^2}\log L(\theta)\right]$$ instead of what you posted. After correcting this and noting $$E_\theta(x) = n\theta$$ given $$x \sim B(n, \theta)$$, it turns out that \begin{align} & I(\theta) = E_\theta\left[-\frac{\partial^2}{\partial\theta^2}\log L(\theta)\right] \\ =& \frac{1}{\theta^2}E_\theta(x) + \frac{1}{(1 - \theta)^2}E_\theta(n - x) \\ =& \frac{n}{\theta} + \frac{n}{1 - \theta} \\ =& \frac{n}{\theta(1 - \theta)}, \end{align} matching the goal.

Through the discussion below, it seems that the reference you quoted fails to clearly present the concept of "Fisher information" (and other related sample-level concepts). I will try to clarify them based on Section 4.3 of Statistical Models by A. C. Davison.

Suppose $$N$$ i.i.d. observations $$x_1, \ldots, x_N$$ are drawn from the distribution $$f(x; \theta)$$, so that the log-likelihood is \begin{align} \ell(\theta) = \log L(\theta) = \sum_{i = 1}^N \log f(x_i; \theta). \end{align}

Fisher information (or expected information) is defined by \begin{align} I(\theta) = E\left[-\frac{\partial^2\ell(\theta)}{\partial\theta^2}\right]. \tag{1} \end{align}

While the observed information is defined as: \begin{align} J(\theta) = -\frac{\partial^2\ell(\theta)}{\partial\theta^2} = -\sum_{i = 1}^N \frac{\partial^2 \log f(x_i; \theta)}{\partial\theta^2}. \tag{2} \end{align}

Note that, if observations are i.i.d., Fisher information only contains parameter $$\theta$$ (i.e., no observations are included, but the sample size $$N$$ is still included).

For your example, you have a single observation $$x \sim B(n, \theta)$$. In this case, your post and my calculation above showed: \begin{align} & I(\theta) = \frac{n}{\theta(1 - \theta)}. \tag{3} \\ & J(\theta) = \frac{x}{\theta^2} + \frac{n - x}{(1 - \theta)^2}. \tag{4} \end{align}

To evaluate "Fisher information evaluated at MLE $$\hat{\theta}$$", simply plug in $$\hat{\theta}$$ into $$(3)$$ (just like evaluating a function's value at a given point) to get $$I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$$. Even though $$J(\hat{\theta}) = I(\hat{\theta})$$ in this case, that doesn't mean $$J(\hat{\theta})$$ is conceptually "Fisher information evaluated at MLE $$\hat{\theta}$$".

• This is the way I would generally do and learnt. But the book defined observed $I(\theta)$ without the expectation. I need to digest that. Commented Jan 19, 2023 at 14:52
• @ThePointer What your textbook put is $J(\theta)$ with one observation $x$ at hand, which in my opinion, is not "Fisher information". Although in this case, $J(\hat{\theta}) = I(\hat{\theta})$, in general, this is not the case (suppose you observed $x_1$ and $x_2$). Commented Jan 19, 2023 at 15:06
• I agree with you @Zhanxiong. Seeing initially made me nervous. I think your comment stating $J(\hat{\theta}) = I(\hat{\theta})$ is not the general case is useful for OP. May I urge you to add this in your post? Commented Jan 19, 2023 at 15:13
• @User1865345 After some calculation, for binomial case, $I(\hat{\theta}) = J(\hat{\theta})$ is indeed true for all numbers of observations (I apologize for previous pre-matured comment). However, this doesn't change my general view on this concept (even he used $\mathcal{I}$ for Fisher information and $I$ for observed information, the problem as stated in my previous comment still stands). Commented Jan 19, 2023 at 15:57
• @User1865345 Done per your urge! Commented Jan 19, 2023 at 15:59