Likelihood function and Fisher information of a coin experiment

Question

I'm trying to understand the notions of likelihood function and Fisher information with a simple example. Suppose I have a coin that gives 0 with probability $p$ and 1 with probability $(1-p)$. I've thrown the coin $n$ times and observed $m$ 0s.

My likelihood function is looking like $$L(p) = p^m(1-p)^{n-m}$$ I'm now trying to estimate p using maximum log-likelihood: $$\frac{d \ln L}{d p}(p)=0$$ That gives me $p=\frac{m}{n}$. Looks pretty good.

Now I want to evaluate the Fisher information that these $n$ measurements gave me about $p$. The formula on Wikipedia says that it is the average of $(\frac{d \ln L}{d p})^2$.

My question is: how should I take this average?

As far as I can understand the likelihood associatec with the given sample is a function of the parameter $p$. If $p$ is fixed, $L(p)$ is just a number with average equal to this number.

Answer 1

If you write down (somewhat paradoxically) the likelihood derivative (or score function) $$\dfrac{\text{d} \ln L}{\text{d} p}(p)$$as $$\dfrac{\text{d} \ln L}{\text{d} p}(p,m)=\dfrac{m}{p}-\dfrac{n-m}{1-p}$$ it becomes a function of both $p$ and of the random outcome $M=m$ of the coin throwing experiment, when $M\sim\mathcal{B}(n,q)$. And hence a random function as well.

Note: There are two parameters in the above, the first one is $p$ the argument of the likelihood function, namely any probability in $(0,1)$. And the second one is $q$, the true value of the parameter for the coin experiment, which is unknown and only informed through the result $M=m$ of the experiment, but which is necessary when computing expectations in $M$.

Let us consider the special case when $p=q$. You can thus consider the expectation of this quantity \begin{align} \mathbb{E}_p\left[\dfrac{\text{d} \ln L}{\text{d} p}(p,M)\right] &=\sum_{m=0}^n \dfrac{\text{d} \ln L}{\text{d} p}(p,m) \mathbb{P}_p(M=m)\\ &=\sum_{m=0}^n \dfrac{\text{d} \ln L}{\text{d} p}(p,m) {n \choose m} p^m (1-p)^{n-m}\\ &= \sum_{m=0}^n \left\{\dfrac{m}{p}-\dfrac{n-m}{1-p}\right\} {n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{1}{p} \sum_{m=0}^n m{ n \choose m} p^m (1-p)^{n-m} -\dfrac{1}{1-p} \sum_{m=0}^n (n-m){ n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{np}{p} - \dfrac{n(1-p)}{1-p}\\ &=0 \end{align} and its variance, which is \begin{align} \mathbb{E}_p\left[\left\{\dfrac{\text{d} \ln L}{\text{d} p}(p,M)\right\}^2\right] &=\sum_{m=0}^n \left\{\dfrac{\text{d} \ln L}{\text{d} p}(p,m)\right\}^2 \mathbb{P}_p(M=m)\\ &= \sum_{m=0}^n \left\{\dfrac{m}{p}-\dfrac{n-m}{1-p}\right\}^2 {n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{1}{p^2(1-p)^2}\sum_{m=0}^n \left\{(1-p)m-p(n-m)\right\}^2 {n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{\text{Var}((1-p)M-p(n-M))}{p^2(1-p)^2}\\ &=\dfrac{\text{Var}((1-p+p)M-pn)}{p^2(1-p)^2}\\ &=\dfrac{\text{Var}(M)}{p^2(1-p)^2}\\ &=\dfrac{np(1-p)}{p^2(1-p)^2}\\ &=\dfrac{n}{p(1-p)} \end{align}

Expectation of the second derivative gave me non-zero result. Strange as wiki said this two definitions are all the same "under certain regularity conditions".

If you consider the second derivative,

\begin{align} \mathbb{E}_p\left[\dfrac{\text{d}^2 \ln L}{\text{d} p^2}(p,M)\right] &=\mathbb{E}_p\left[\dfrac{\text{d}}{\text{d} p}\left\{\dfrac{M}{p}-\dfrac{n-M}{1-p}\right\}\right]\\ &=\mathbb{E}_p\left[-\dfrac{M}{p^2}-\dfrac{n-M}{(1-p)^2}\right]\\ &=-\dfrac{\mathbb{E}\left[M\right]}{p^2}-\dfrac{\mathbb{E}\left[n-M\right]}{(1-p)^2}\\ &=-\dfrac{np}{p^2}-\dfrac{n(1-p)}{(1-p)^2}\\ &=-\dfrac{n(p+1-p)}{p(1-p)}=-\dfrac{n}{p(1-p)} \end{align} which leads to the same value (modulo the minus sign).