# Coin flipping: Relationship between Bayesian and Frequentist's point estimates

I have a (biased) coin that has an unknown Head probability $$p\in(0,1)$$. To point estimate $$p$$, say that I'm going to use two approaches.

Approach 1. I can use the Bayesian inference technique. Starting from a Beta prior $$p^0\sim Beta(a^0_H,a^0_T)$$, I calculate the posterior from the observations. When I observed $$n_H$$ Heads and $$n_T$$ Tails in round $$n_H+n_T$$, The posterior will follow $$Beta(a^0_H+n_H,a^0_T+n_T)$$. As a point estimate for $$p$$, I can take the posterior mean, which is $$\hat p_B=\frac{a^0_H+n_H}{a^0_H+n_H+a^0_T+n_T}$$

Approach 2. I can use MLE. The MLE in this case is given by $$\hat p_F=\frac{n_H}{n_H+n_T}$$.

My question is what is the relationship between $$\hat p_B$$ and $$\hat p_F$$? Especially,

1. It looks like when I have enough observations, the two estimates coincide. In this case, can I say $$\hat p_B$$ and $$\hat p_T$$ are asymptotically equal? or is there any other terminology that I can formally describe the asymptotic relationship between the two?
2. When I only have a handful of observations, what can I say about the relationship? Can I say the two differ only by constant terms? or, again, is there any other formal description of the two under the small sample situation?
• I guess that you mean $p^0\sim Beta(a_H^0,a_T^0)$. It looks like you are choosing the Beta parameters so that they sum up to 1. It would be a very strange and strong prior... Commented Sep 8, 2020 at 14:39
• @Sergio a Beta prior whose coefficients sum to unity would be a very weak prior. However, where do you get the impression the OP is assuming any such constraint? I haven't been able to find such an implication.
– whuber
Commented Sep 8, 2020 at 14:47
• @whuber (a) if $p\sim Beta(0.5,0.5)$, then $p=0$ and $p=1$ have the maximum density. What am I missing? (b) Why $(a_H,a_T)$ instead of $(\alpha,\beta)$? Commented Sep 8, 2020 at 16:11
• @Sergio (b) is a matter of taste. (a) To see why Beta$(1/2,1/2)$ is a weak prior compared to a Beta distribution with larger parameters $\alpha$ and $\beta,$ compare the two posterior distributions obtained upon observing $H$ heads and $T$ tails. One of them is Beta$(H+1/2,T+1/2)$ and the other is Beta$(H+\alpha,T+\beta).$ As $\alpha$ and $\beta$ increase, the latter becomes more and more concentrated around the prior: that's what a strong prior does.
– whuber
Commented Sep 8, 2020 at 16:59
• @whuber Thanks for your reply. Commented Sep 8, 2020 at 17:57

The posterior point estimate is a weighted combination of the prior point estimate, $$\frac{a^0_H}{a^0_H+a^0_T}$$, and the maximum likelihood estimate $$\frac{n_H}{n_H+n_T}$$.
The weights are simply $$\omega_{\text{Prior}} = \frac{n_{\text{Prior}}}{n_{\text{Prior}} + n_{\text{Lik}}}$$ and $$\omega_{\text{Lik}} = \frac{n_{\text{Lik}}}{n_{\text{Prior}} + n_{\text{Lik}}}$$, where $$n_{\text{Prior}} = a^0_H+a^0_T$$ and $$n_{\text{Lik}} = n_H+n_T$$.
\begin{align} \text{Posterior Mean} &= \frac{a^0_H}{a^0_H+a^0_T} \omega_{\text{Prior}} + \frac{n_H}{n_H+n_T} \omega_{\text{Lik}}\\ &= \frac{a^0_H}{a^0_H+a^0_T} \times \frac{a^0_H + a^0_T}{a^0_H+a^0_T+n_H+n_T} + \frac{n_H}{n_H+n_T} \times \frac{n_H+n_T}{a^0_H+a^0_T+n_H+n_T}\\ &= \frac{a^0_H}{a^0_H+a^0_T+n_H+n_T} + \frac{n_H}{a^0_H+a^0_T+n_H+n_T}\\ &= \frac{a^0_H+n_H}{a^0_H+n_H+a^0_T+n_T} \end{align}
1. With lots of observations, $$\omega_{\text{Lik}} \gg \omega_{\text{Prior}}$$, and so the posterior mean is almost the same as the maximum likelihood estimate.
2. With few observations, $$\omega_{\text{Lik}} \ll \omega_{\text{Prior}}$$, and so the posterior mean is almost the same as the prior mean.