Bayesian vs. frequentist estimation

Question

I don't really understand the connection between bayesian to "normal" frequentist estimation.

Suppose we want to estimate the expected value of a population given a sample.

In frequentist statisics i'd calculate the sample mean $\overline{x}=\frac{1}{n}\sum_{i=1}^nx_i$ which is, as we know unbiased, consistent and efficent - and therefore the "best" estimator for the true mean we can use. This estimator has a standard deviation of $\text{SE}_\bar{x}\ = \frac{s}{\sqrt{n}}$.

In contrast in bayesian statistics i'd have some a-priori distribution, plug in data and calculate with the bayesian formula an a-posteriori distribution.

• So how does this a-posteriori distribution compare to the sample mean point estimate?
• Is the expected value of the a-posteriori distribution equal to the sample mean?
• Is the standard deviation of the a-posteriori distribution equal to the standard deviation of the sample mean?
• Which one of the two methods is the better one - or does the bayesian approach just give you more information (it gives you a distribution whereas the frequentist estimation only gives you point values)
• Is it possible to get point estimates like in the frequentist estimation from an a-posterio distibution (e.g. the expected value for an point estimation for the true mean)?

---

Edit:

Ok, I got the idea but what really bothers me is for example in https://en.wikipedia.org/wiki/German_tank_problem#Example the german tank problem.

We want to estimate the total number of tanks where an intelligence officer has spotted k = 4 tanks with serial numbers, 2, 6, 7, and 14.

The frequentist estimate to the tank count is $16.5$ whereas the bayesian is $19.5 \pm 10$ (although the frequentist answer is in the sd. of the bayesesian).

How can two different mathematical (scientific) approaches for the same question lead to two different answers? Intuitively there must be one "more" correct answer, or not?

Answer 1

The simple answer for all your questions is: in Bayesian model you include a priori information in your model besides the data,

$$ \text{posterior} \propto \text{likelihood} \times \text{prior} $$

so if you include in your model additional information, than your estimates (mean, standard deviation etc.) can possibly differ from maximum likelihood estimates. You can use noninformative priors, and in this case estimates should be the same as in maximum likelihood case. One example where Bayesian methods can outperform other methods are cases where we are dealing with small samples (see nice example here), even in cases such as predicting future mid-air collisions in times when non such collision happened yet - in such cases out-of-data information helps us to learn from insufficient data.

You ask also about posterior distribution vs. sample mean and about point estimates. In posterior distribution you have a whole distribution for your parameter of interest instead of a point value, you can take mean (or median, or possibly other statistics) of this distribution to get a point value. If you are interested only in point estimate, you can use maximum a posteriori methods and not bother with posterior distribution.

Finally, neither of the methods are "better", or "worse". They are just different. By reviewing multiple questions tagged as bayesian you can learn much about their pros and cons, for example: When are Bayesian methods preferable to Frequentist?, or Why are Bayesian methods widely considered particularly "convenient"?.

Answer 2

I've covered this topic on other threads listed below. I will address your specific questions here.

• So how does this a-posteriori distribution compare to the sample mean point estimate? Is the expected value of the a-posteriori distribution equal to the sample mean? Is the standard deviation of the a-posteriori distribution equal to the standard deviation of the sample mean?

Often the mean and standard deviation of the posterior distribution will be similar to the sample mean and its standard error, but not always. It depends on the selected prior distribution and the probability model for the data.

• Which one of the two methods is the better one - or does the Bayesian approach just give you more information (it gives you a distribution whereas the frequentist estimation only gives you point values)

The short answer to this question is that it depends on what you want to measure, the experiment or the experimenter. The frequentist defines probability in terms of a long-run frequency of the experiment. Inference on an unknown fixed parameter uses confidence levels and p-values based on the concept of sampling from repeated experiments. Historical data (external information) is incorporated into an analysis by pooling observations through the likelihood or performing a summary-level meta-analysis (Johnson 2021a). The frequentist can construct a confidence distribution or confidence curve that shows the plausibility of each hypothesis using p-values. This is analogous to the Bayesian posterior.

Bayesians define probability as the belief of the experimenter, so there are no limits to how the prior can be formed to influence the inference on a parameter. It also means these statements of belief are unfalsifiable. Rather than incorporating historical data (external information) through the likelihood, Bayesians will incorporate this through the prior. If the prior distribution is chosen in such a way that the posterior is dominated by the likelihood, Bayesian belief is more objectively viewed as a form of confidence based on frequency probability of the experiment (Johnson 2021b). This is an argument for using frequentist methods.

• Is it possible to get point estimates like in the frequentist estimation from an a-posterior distibution (e.g. the expected value for an point estimation for the true mean)?

Yes, Bayesians will use the mean, median, or mode of the posterior distribution as a point estimate for a parameter. In non-normal settings and when considering non-linear transformations of parameters, using the posterior mean can introduce bias in estimation (considering that the parameter is an unknown fixed quantity).