3
$\begingroup$

I have a (biased) coin that has an unknown Head probability $p\in(0,1)$. To point estimate $p$, say that I'm going to use two approaches.

Approach 1. I can use the Bayesian inference technique. Starting from a Beta prior $p^0\sim Beta(a^0_H,a^0_T)$, I calculate the posterior from the observations. When I observed $n_H$ Heads and $n_T$ Tails in round $n_H+n_T$, The posterior will follow $Beta(a^0_H+n_H,a^0_T+n_T)$. As a point estimate for $p$, I can take the posterior mean, which is $\hat p_B=\frac{a^0_H+n_H}{a^0_H+n_H+a^0_T+n_T}$

Approach 2. I can use MLE. The MLE in this case is given by $\hat p_F=\frac{n_H}{n_H+n_T}$.

My question is what is the relationship between $\hat p_B$ and $\hat p_F$? Especially,

  1. It looks like when I have enough observations, the two estimates coincide. In this case, can I say $\hat p_B$ and $\hat p_T$ are asymptotically equal? or is there any other terminology that I can formally describe the asymptotic relationship between the two?
  2. When I only have a handful of observations, what can I say about the relationship? Can I say the two differ only by constant terms? or, again, is there any other formal description of the two under the small sample situation?
$\endgroup$
5
  • $\begingroup$ I guess that you mean $p^0\sim Beta(a_H^0,a_T^0)$. It looks like you are choosing the Beta parameters so that they sum up to 1. It would be a very strange and strong prior... $\endgroup$
    – Sergio
    Sep 8, 2020 at 14:39
  • $\begingroup$ @Sergio a Beta prior whose coefficients sum to unity would be a very weak prior. However, where do you get the impression the OP is assuming any such constraint? I haven't been able to find such an implication. $\endgroup$
    – whuber
    Sep 8, 2020 at 14:47
  • $\begingroup$ @whuber (a) if $p\sim Beta(0.5,0.5)$, then $p=0$ and $p=1$ have the maximum density. What am I missing? (b) Why $(a_H,a_T)$ instead of $(\alpha,\beta)$? $\endgroup$
    – Sergio
    Sep 8, 2020 at 16:11
  • 1
    $\begingroup$ @Sergio (b) is a matter of taste. (a) To see why Beta$(1/2,1/2)$ is a weak prior compared to a Beta distribution with larger parameters $\alpha$ and $\beta,$ compare the two posterior distributions obtained upon observing $H$ heads and $T$ tails. One of them is Beta$(H+1/2,T+1/2)$ and the other is Beta$(H+\alpha,T+\beta).$ As $\alpha$ and $\beta$ increase, the latter becomes more and more concentrated around the prior: that's what a strong prior does. $\endgroup$
    – whuber
    Sep 8, 2020 at 16:59
  • $\begingroup$ @whuber Thanks for your reply. $\endgroup$
    – Sergio
    Sep 8, 2020 at 17:57

1 Answer 1

4
$\begingroup$

The posterior point estimate is a weighted combination of the prior point estimate, $\frac{a^0_H}{a^0_H+a^0_T}$, and the maximum likelihood estimate $\frac{n_H}{n_H+n_T}$.

The weights are simply $\omega_{\text{Prior}} = \frac{n_{\text{Prior}}}{n_{\text{Prior}} + n_{\text{Lik}}}$ and $\omega_{\text{Lik}} = \frac{n_{\text{Lik}}}{n_{\text{Prior}} + n_{\text{Lik}}}$, where $n_{\text{Prior}} = a^0_H+a^0_T$ and $n_{\text{Lik}} = n_H+n_T$.

You can confirm this by substitution:

$$ \begin{align} \text{Posterior Mean} &= \frac{a^0_H}{a^0_H+a^0_T} \omega_{\text{Prior}} + \frac{n_H}{n_H+n_T} \omega_{\text{Lik}}\\ &= \frac{a^0_H}{a^0_H+a^0_T} \times \frac{a^0_H + a^0_T}{a^0_H+a^0_T+n_H+n_T} + \frac{n_H}{n_H+n_T} \times \frac{n_H+n_T}{a^0_H+a^0_T+n_H+n_T}\\ &= \frac{a^0_H}{a^0_H+a^0_T+n_H+n_T} + \frac{n_H}{a^0_H+a^0_T+n_H+n_T}\\ &= \frac{a^0_H+n_H}{a^0_H+n_H+a^0_T+n_T} \end{align} $$


  1. With lots of observations, $\omega_{\text{Lik}} \gg \omega_{\text{Prior}}$, and so the posterior mean is almost the same as the maximum likelihood estimate.
  2. With few observations, $\omega_{\text{Lik}} \ll \omega_{\text{Prior}}$, and so the posterior mean is almost the same as the prior mean.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.