5
$\begingroup$

I don't really understand the connection between bayesian to "normal" frequentist estimation.

Suppose we want to estimate the expected value of a population given a sample.

In frequentist statisics i'd calculate the sample mean $\overline{x}=\frac{1}{n}\sum_{i=1}^nx_i$ which is, as we know unbiased, consistent and efficent - and therefore the "best" estimator for the true mean we can use. This estimator has a standard deviation of $\text{SE}_\bar{x}\ = \frac{s}{\sqrt{n}}$.

In contrast in bayesian statistics i'd have some a-priori distribution, plug in data and calculate with the bayesian formula an a-posteriori distribution.

  • So how does this a-posteriori distribution compare to the sample mean point estimate?
  • Is the expected value of the a-posteriori distribution equal to the sample mean?
  • Is the standard deviation of the a-posteriori distribution equal to the standard deviation of the sample mean?
  • Which one of the two methods is the better one - or does the bayesian approach just give you more information (it gives you a distribution whereas the frequentist estimation only gives you point values)
  • Is it possible to get point estimates like in the frequentist estimation from an a-posterio distibution (e.g. the expected value for an point estimation for the true mean)?

Edit:

Ok, I got the idea but what really bothers me is for example in https://en.wikipedia.org/wiki/German_tank_problem#Example the german tank problem.

We want to estimate the total number of tanks where an intelligence officer has spotted k = 4 tanks with serial numbers, 2, 6, 7, and 14.

The frequentist estimate to the tank count is $16.5$ whereas the bayesian is $19.5 \pm 10$ (although the frequentist answer is in the sd. of the bayesesian).

How can two different mathematical (scientific) approaches for the same question lead to two different answers? Intuitively there must be one "more" correct answer, or not?

Improve this question
$\endgroup$
  • $\begingroup$ I understand where you are going with this, but I think to make it clearer, what are you calculating the expected value of under the frequentist paradigm? You say the expected value of a population but what characteristic of the population? The mean doesn't always work well or isn't always appropriate unless you define what you are trying to estimate. $\endgroup$ – user95564 Nov 22 '15 at 0:43
  • 1
    $\begingroup$ "Which one of the two methods is the better one - or does the bayesian approach just give you more information (it gives you a distribution whereas the frequentist estimation only gives you point values)" -- Let's not start a civil war here.... $\endgroup$ – user95564 Nov 22 '15 at 0:44
  • $\begingroup$ See stats.stackexchange.com/questions/22/… $\endgroup$ – Tim♦ Nov 22 '15 at 10:01
  • $\begingroup$ "as we know unbiased, consistent and efficent" -- we do? I don't think I know this. What makes it efficient? $\endgroup$ – Glen_b Nov 22 '15 at 10:44
  • $\begingroup$ @Glen_b you're correct, it isn't efficient in every case, thanks $\endgroup$ – lukstei Nov 22 '15 at 10:54
8
$\begingroup$

The simple answer for all your questions is: in Bayesian model you include a priori information in your model besides the data,

$$ \text{posterior} \propto \text{likelihood} \times \text{prior} $$

so if you include in your model additional information, than your estimates (mean, standard deviation etc.) can possibly differ from maximum likelihood estimates. You can use noninformative priors, and in this case estimates should be the same as in maximum likelihood case. One example where Bayesian methods can outperform other methods are cases where we are dealing with small samples (see nice example here), even in cases such as predicting future mid-air collisions in times when non such collision happened yet - in such cases out-of-data information helps us to learn from insufficient data.

You ask also about posterior distribution vs. sample mean and about point estimates. In posterior distribution you have a whole distribution for your parameter of interest instead of a point value, you can take mean (or median, or possibly other statistics) of this distribution to get a point value. If you are interested only in point estimate, you can use maximum a posteriori methods and not bother with posterior distribution.

Finally, neither of the methods are "better", or "worse". They are just different. By reviewing multiple questions tagged as you can learn much about their pros and cons, for example: When are Bayesian methods preferable to Frequentist?, or Why are Bayesian methods widely considered particularly "convenient"?.

Improve this answer
$\endgroup$
  • $\begingroup$ thanks for your answer @Tim, sorry cannot upvote with only 8 karma $\endgroup$ – lukstei Nov 22 '15 at 11:09
  • $\begingroup$ w.r.t. better/worse. Shouldn't we define an error metric and assess the performance of a method? $\endgroup$ – Alex Kreimer Jan 26 '20 at 6:37
  • $\begingroup$ @AlexKreimer first of all, I meant "in general". Second, minimizing some error is not the only criteria for model choice. Third, you can measure error only on the data you have, it does not give you any guarantees on future performance on unseen data. $\endgroup$ – Tim♦ Jan 26 '20 at 8:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.