Suppose I have a coin toss experiment in which I want to calculate the maximum likelihood estimate of the coin parameter $p$ when tossing the coin $n$ times. After calculating the derivative of the binomial likelihood function $ L(p) = { n \choose x } p^x (1-p)^{n-x} $, I get the optimal value for $p$ to be $p^{*} = \frac{x}{n}$, with $x$ being the number of successes.

My questions now are:

  • How would I calculate the expected value/variance of this maximum likelihood estimate for $p$?
  • Do I need to calculate the expected value/variance for $L(p^{*})$?
  • If yes, how would I do that?
  • 1
    $\begingroup$ I'm guessing this is some kind of self-study (you should tag it as such). What exactly is it you want? Do inference on your parameter? $\endgroup$
    – pkofod
    Jan 26 '14 at 10:45
  • $\begingroup$ What do you mean inference on the parameter? I am quite unsure how i would calculate the expected value/variance for the quantity $p^{*}$. I mean, i know what mean/variance is and how to calculate it for simple examples, but don't understand how to apply it to $p^{*}$. $\endgroup$
    – Manu
    Jan 26 '14 at 10:50

First of all this is a self-study question, so I'm going to go too much into each and every little technical detail, but I'm not going on a derivation frenzy either. There are many ways to do this. I'll help you by using general properties of the maximum likelihood estimator.

Background information

In order to solve your problem I think you need to study maximum likelihood from the beginning. You are probably using some kind of text book, and the answer should really be there somewhere. I'll help you find out what to look for.

Maximum Likelihood is an estimation method which is basically what we call an M-estimator (think of the "M" as "maximize/minimize"). If the conditions required for using these methods are satisfied, we can show that the parameter estimates are consistent and asymptotically normally distributed, so we have:

$$ \sqrt{N}(\hat\theta-\theta_0)\overset{d}{\to}\text{Normal}(0,A_0^{-1}B_0A_0^{-1}), $$

where $A_0$ and $B_0$ are some matrices. When using maximum likelihood we can show that $A_0=B_0$, and thus we have a simple expression: $$ \sqrt{N}(\hat\theta-\theta_0)\overset{d}{\to}\text{Normal}(0,A_0^{-1}). $$ We have that $A_0\equiv -E(H(\theta_0))$ where $H$ denotes the hessian. This is what you need to estimate in order to get your variance.

Your specific problem

So how do we do it? Here let's call our parameter vector $\theta$ what you do: $p$. This is just a scalar, so our "score" is just the derivative and the "hessian" is just the second order derivative. Our likelihood function can be written as: $$ l(p)=(p)^x (1-p)^{n-x}, $$ which is what we want to maximize. You used the first derivative of this or the log likelihood to find your $p^*$. Instead of setting the first derivative equal to zero, we can differentiate again, to find the second order derivative $H(p)$. First we take logs: $$ ll(p)\equiv\log(l(p))=x\log(p)+(n-x)\log(1-p) $$ Then our 'score' is: $$ ll'(p)=\frac{x}{p}+\frac{n-x}{1-p}, $$ and our 'hessian': $$ H(p)=ll''(p)=-\frac{x}{p^2}-\frac{n-x}{(1-p)^2}. $$ Then our general theory from above just tells you to find $(-E(H(p)))^{-1}$. Now you just have to take the expectation of $H(p)$ (Hint: use $E(x/n)=p$), multiply by $-1$ and take the inverse. Then you'll have your variance of the estimator.

  • $\begingroup$ Is $Var(p) = \frac{p^2-1}{n} - \frac{1}{np}$ correct? $\endgroup$
    – Manu
    Jan 26 '14 at 12:45
  • $\begingroup$ @Manu : Not quite, but it looks like you just made a little error somewhere. Can you post some more steps? $\endgroup$
    – pkofod
    Jan 26 '14 at 12:47
  • $\begingroup$ $(-E(H(p)))^{-1} = E(-H(p))^{-1]} = (E(\frac{x}{p^2}) + E(\frac{n-x}{(1-p)^2}))^{-1} = ( p^{-2}np + (1-p)^{-2}(n -np) )^{-1}$. from there on i simplified by multiplying and taking the inverse. $\endgroup$
    – Manu
    Jan 26 '14 at 12:50
  • $\begingroup$ That is all correct, now just simplify. In the first part p cancels, and in the second part you can take n outside the parenthesis. $\endgroup$
    – pkofod
    Jan 26 '14 at 12:55
  • 2
    $\begingroup$ $(n/p+n/[1-p])^{-1}$ is what you have above. Just factor the $n$ out, put on a common denominator and then take the reciprocal. $\endgroup$
    – ekvall
    Jan 26 '14 at 20:24

To start you off, let's do the expected value:

If $x$ is the number of successes in $n$ throws, then $x/n$ is the proportion of successes in your sample. Consider $\mathbb{E}x$; for each throw, the probability of success is $p$ according to the assumptions, so when tossing the coin one time the expected "number of successes" is $p\times1+(1-p)\times 0=p$, right? Thus, if you throw the coin $n$ times, you would expect success $np$ times because the throws are independent. Then, since $np$ is the number of expected successes in $n$ throws, you get $$\mathbb{E}p^*=\mathbb{E}n^{-1}x=n^{-1}\mathbb{E}x=n^{-1}\times np=p$$

So the estimator is unbiased. Can you figure how to do the variance from here?

Edit: Let's do the variance, too. We use that $\text{Var}(p^*)=\mathbb{E}p{^*}^2-(\mathbb{E}p{^*})^2$. The second term we already have from the calculation of the expected value, so let's do the first: $$\mathbb{E}p{^*}^2=n^{-2}\mathbb{E}x^2$$To simplify some, we can express the number of successes in $n$ throws as follows: $$x=\sum_1^n\chi _i,$$ where $\chi_i$ takes the value 1 if throw $i$ was a success and 0 otherwise. Hence, $$\mathbb{E}x^2=\mathbb{E}(\sum_1^n\chi _i)^2=\mathbb{E}[\sum_1^n\chi _i^2+2\sum_{i<j}\chi _i\chi_j]=np+n(n-1)p^2,$$ and so putting things together you arrive at $$\text{Var}(p^*)=\frac{p(1-p)}{n}$$.

  • $\begingroup$ If you throw $n=3$ heads in a row, your $p_{MLE} =1.0$. However what exact value would Var($p^*$) take? $\endgroup$
    – piccolo
    Aug 13 '19 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.