I'm trying to understand the notions of likelihood function and Fisher information with a simple example. Suppose I have a coin that gives 0 with probability $p$ and 1 with probability $(1-p)$. I've thrown the coin $n$ times and observed $m$ 0s.

My likelihood function is looking like $$L(p) = p^m(1-p)^{n-m}$$ I'm now trying to estimate p using maximum log-likelihood: $$\frac{d \ln L}{d p}(p)=0$$ That gives me $p=\frac{m}{n}$. Looks pretty good.

Now I want to evaluate the Fisher information that these $n$ measurements gave me about $p$. The formula on Wikipedia says that it is the average of $(\frac{d \ln L}{d p})^2$.

My question is: how should I take this average?

As far as I can understand the likelihood associatec with the given sample is a function of the parameter $p$. If $p$ is fixed, $L(p)$ is just a number with average equal to this number.

  • 1
    $\begingroup$ The Wikipedia formula is an expectation and it is taken with respect to the random variable. Your random variable is $m$. $\endgroup$ – whuber Oct 12 '16 at 14:04
  • $\begingroup$ When it says average, it means the expected value. But we know the expected value of this experiment is $m = np$. Plus this into your expression for $I$ and voila. $\endgroup$ – ilanman Oct 12 '16 at 14:04
  • $\begingroup$ @ilanman If I place $np$ instead of $m$ I will have 0, which is not surprising as I I just squared 0. But I expect information to rise with the increasing of the sample size $\endgroup$ – Krivoi Oct 12 '16 at 14:13
  • 1
    $\begingroup$ You get 0 if you set ($\frac{d ln L}{d p})^2 = 0$. But who says you're allowed to do that when solving for Fisher's information? You're supposed to take the expectation of that expression, not set it to 0. If you do, you'll get a quadratic expression in $p$, which gives you the nice curvature you're expecting. Alternatively, take the expectation of the negative second derivative. It might be easier algebraically. $\endgroup$ – ilanman Oct 12 '16 at 14:16
  • $\begingroup$ @ilanman I've used maximum loglikelihood $\frac{d \ln L}{d p}=0$ to estimate $p$ for the given $m$, so $(\frac{d \ln L}{d p})^2$ for the same $m$ and $p$ should also be equal 0. Where am I going wrong? $\endgroup$ – Krivoi Oct 12 '16 at 14:26

If you write down (somewhat paradoxically) the likelihood derivative (or score function) $$\dfrac{\text{d} \ln L}{\text{d} p}(p)$$as $$\dfrac{\text{d} \ln L}{\text{d} p}(p,m)=\dfrac{m}{p}-\dfrac{n-m}{1-p}$$ it becomes a function of both $p$ and of the random outcome $M=m$ of the coin throwing experiment, when $M\sim\mathcal{B}(n,q)$. And hence a random function as well.

Note: There are two parameters in the above, the first one is $p$ the argument of the likelihood function, namely any probability in $(0,1)$. And the second one is $q$, the true value of the parameter for the coin experiment, which is unknown and only informed through the result $M=m$ of the experiment, but which is necessary when computing expectations in $M$.

Let us consider the special case when $p=q$. You can thus consider the expectation of this quantity \begin{align} \mathbb{E}_p\left[\dfrac{\text{d} \ln L}{\text{d} p}(p,M)\right] &=\sum_{m=0}^n \dfrac{\text{d} \ln L}{\text{d} p}(p,m) \mathbb{P}_p(M=m)\\ &=\sum_{m=0}^n \dfrac{\text{d} \ln L}{\text{d} p}(p,m) {n \choose m} p^m (1-p)^{n-m}\\ &= \sum_{m=0}^n \left\{\dfrac{m}{p}-\dfrac{n-m}{1-p}\right\} {n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{1}{p} \sum_{m=0}^n m{ n \choose m} p^m (1-p)^{n-m} -\dfrac{1}{1-p} \sum_{m=0}^n (n-m){ n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{np}{p} - \dfrac{n(1-p)}{1-p}\\ &=0 \end{align} and its variance, which is \begin{align} \mathbb{E}_p\left[\left\{\dfrac{\text{d} \ln L}{\text{d} p}(p,M)\right\}^2\right] &=\sum_{m=0}^n \left\{\dfrac{\text{d} \ln L}{\text{d} p}(p,m)\right\}^2 \mathbb{P}_p(M=m)\\ &= \sum_{m=0}^n \left\{\dfrac{m}{p}-\dfrac{n-m}{1-p}\right\}^2 {n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{1}{p^2(1-p)^2}\sum_{m=0}^n \left\{(1-p)m-p(n-m)\right\}^2 {n \choose m} p^m (1-p)^{n-m}\\ &=\dfrac{\text{Var}((1-p)M-p(n-M))}{p^2(1-p)^2}\\ &=\dfrac{\text{Var}((1-p+p)M-pn)}{p^2(1-p)^2}\\ &=\dfrac{\text{Var}(M)}{p^2(1-p)^2}\\ &=\dfrac{np(1-p)}{p^2(1-p)^2}\\ &=\dfrac{n}{p(1-p)} \end{align}

Expectation of the second derivative gave me non-zero result. Strange as wiki said this two definitions are all the same "under certain regularity conditions".

If you consider the second derivative,

\begin{align} \mathbb{E}_p\left[\dfrac{\text{d}^2 \ln L}{\text{d} p^2}(p,M)\right] &=\mathbb{E}_p\left[\dfrac{\text{d}}{\text{d} p}\left\{\dfrac{M}{p}-\dfrac{n-M}{1-p}\right\}\right]\\ &=\mathbb{E}_p\left[-\dfrac{M}{p^2}-\dfrac{n-M}{(1-p)^2}\right]\\ &=-\dfrac{\mathbb{E}\left[M\right]}{p^2}-\dfrac{\mathbb{E}\left[n-M\right]}{(1-p)^2}\\ &=-\dfrac{np}{p^2}-\dfrac{n(1-p)}{(1-p)^2}\\ &=-\dfrac{n(p+1-p)}{p(1-p)}=-\dfrac{n}{p(1-p)} \end{align} which leads to the same value (modulo the minus sign).

  • 1
    $\begingroup$ So in a general case when likelihood depends on $n$ experimental results, I should take expectation value, considering all of them as random variables but not as the numbers (results of experiments) $\endgroup$ – Krivoi Oct 12 '16 at 15:49
  • 1
    $\begingroup$ The information is constructed by considering the likelihood, which is by definition a function of the unknown parameter given the observed sample, as a function of the random sample and hence as a random object for a given value of the parameter. And then to take the expectation of this random object when the distribution is indexed by the same value of the parameter. What is often confusing for neophytes is the use of two types of parameters in those expressions, one as argument of $L$ and one as index of the expectation. $\endgroup$ – Xi'an Oct 12 '16 at 15:54
  • 1
    $\begingroup$ I'm very close to understanding that. I will consider some other examples. Thanks again! $\endgroup$ – Krivoi Oct 12 '16 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.