54
$\begingroup$

Note: I am aware of philosophical differences between Bayesian and frequentist statistics.

For example "what is the probability that the coin on the table is heads" doesn't make sense in frequentist statistics, since it has either already landed heads or tails -- there is nothing probabilistic about it. So the question has no answer in frequentist terms.

But such a difference is specifically not the kind of difference I'm asking about.

Rather, I would like to know how their predictions for well-formed questions actually differ in the real world, excluding any theoretical/philosophical differences such as the example I mentioned above.

So in other words:

What's an example of a question, answerable in both frequentist and Bayesian statistics, whose answer is different between the two?

(e.g. Perhaps one of them answers "1/2" to a particular question, and the other answers "2/3".)

Are there any such differences?

  • If so, what are some examples?

  • If not, then when does it actually ever make a difference whether I use Bayesian or frequentist statistics when solving a particular problem?
    Why would I avoid one in favor of the other?

$\endgroup$
  • 8
    $\begingroup$ John Kruschke just produced two videos where he compares Bayesian and standard statistical methods. He has many examples where the Bayesian method rejects but the standard method doesn't. Maybe not exactly what you were looking for, but anyway... youtu.be/YyohWpjl6KU and youtu.be/IhlSD-lIQ_Y . $\endgroup$ – Rasmus Bååth Nov 13 '12 at 10:16
  • 4
    $\begingroup$ The binomial distribution provides another example where frequentist (likelihood-based) inference and Bayesian inference differ in some cases. The profile likelihood of the parameter $N$ does not decay to $0$ as $N\rightarrow \infty$ (see) for some samples. This implies that some likelihood-confidence interval have infinite length. On the other hand, the marginal posterior distribution of $N$ always decays to $0$ as $N\rightarrow\infty$ given that it is integrable. $\endgroup$ – user10525 Nov 13 '12 at 11:31
  • $\begingroup$ @Procrastinator: Thanks, I'm looking at the slides mentioned right now. This seems a bit more intense than my mathematical background but hopefully I'll get something out of it. :) $\endgroup$ – Mehrdad Nov 13 '12 at 15:55
  • 2
    $\begingroup$ You might want to have a look at Stone's example. I explain it on my blog here: normaldeviate.wordpress.com/2012/12/08/… $\endgroup$ – Larry Wasserman Dec 24 '12 at 1:15
  • 1
    $\begingroup$ @mbq: Just wondering, why was this made community wiki? $\endgroup$ – Mehrdad Aug 10 '14 at 19:58
9
$\begingroup$

This example is taken from here. (I even think I got this link from SO, but cannot find it anymore.)

A coin has been tossed $n=14$ times, coming up heads $k=10$ times. If it is to be tossed twice more, would you bet on two heads? Assume you do not get to see the result of the first toss before the second toss (and also independently conditional on $\theta$), so that you cannot update your opinion on $\theta$ in between the two throws.

By independence, $$f(y_{f,1}=\text{heads},y_{f,2}=\text{heads}|\theta)=f(y_{f,1}=\text{heads})f(y_{f,2}=\text{heads}|\theta)=\theta^2.$$ Then, the predictive distribution given a $\text{Beta}(\alpha_0,\beta_0)$-prior, becomes \begin{eqnarray*} f(y_{f,1}=\text{heads},y_{f,2}=\text{heads}|y)&=&\int f(y_{f,1}=\text{heads},y_{f,2}=\text{heads}|\theta)\pi(\theta|y)d\theta\notag\\ &=&\frac{\Gamma\left(\alpha _{0}+\beta_{0}+n\right)}{\Gamma\left(\alpha_{0}+k\right)\Gamma\left(\beta_{0}+n-k\right)}\int \theta^2\theta ^{\alpha _{0}+k-1}\left( 1-\theta \right) ^{\beta _{0}+n-k-1}d\theta\notag\\ &=&\frac{\Gamma\left(\alpha_{0}+\beta_{0}+n\right)}{\Gamma\left(\alpha_{0}+k\right)\Gamma\left(\beta_{0}+n-k\right)}\frac{\Gamma\left(\alpha_{0}+k+2\right)\Gamma\left(\beta_{0}+n-k\right)}{\Gamma\left(\alpha_{0}+\beta_{0}+n+2\right)}\notag\\ &=&\frac{(\alpha_{0}+k)\cdot(\alpha_{0}+k+1)}{(\alpha_{0}+\beta_{0}+n)\cdot(\alpha_{0}+\beta_{0}+n+1)} \end{eqnarray*} For a uniform prior (a $\text{Beta}(1, 1)$-prior), this gives roughly .485. Hence, you would likely not bet. Based on the MLE 10/14, you would calculate a probability of two heads of $(10/14)^2\approx.51$, such that betting would make sense.

$\endgroup$
  • $\begingroup$ +1 exactly the kind of answer I was looking for, thanks. $\endgroup$ – Mehrdad Mar 26 '15 at 0:00
  • 5
    $\begingroup$ There was actually an update to the post referenced in the answer... Though he left the post up, "instead of using the uniform distribution as a prior, we can be even more agnostic. In this case, we can use the Beta(0,0) distribution as a prior. Such a distribution corresponds to the case where any mean of the distribution is equally likely. In this case, the two approaches, Bayesian and frequentist give the same results." !!! So we still need an example to answer this question! Hence +1 to the answer below as the true answer to this question. $\endgroup$ – user1745038 Aug 11 '15 at 5:07
10
$\begingroup$

See my question here, which mentions a paper by Edwin Jaynes that gives an example of a correctly constructed frequentist confidence interval, where there is sufficient information in the sample to know for certain that the true value of the statistic lies nowhere in the confidence interval (and thus the confidence interval is different from the Bayesian credible interval).

However, the reason for this is the difference in the definition of a confidence interval and a credible interval, which in turn is a direct consequence of the difference in frequentist and Bayesian definitions of probability. If you ask a Bayesian to produce a Bayesian confidence (rather than credible) interval, then I suspect that there will always be a prior for which the intervals will be the same, so the differences are down to choice of prior.

Whether frequentist or Bayesian methods are appropriate depends on the question you want to pose, and at the end of the day it is the difference in philosophies that decides the answer (provided that the computational and analytic effort required is not a consideration).

Being somewhat tongue in cheek, it could be argued that a long run frequency is a perfectly reasonable way of determining the relative plausibility of a proposition, in which case frequentist statistics is a slightly odd subset of subjective Bayesianism - so any question a frequentist can answer a subjectivist Bayesian can also answer in the same way, or in some other way should they choose different priors. ;o)

$\endgroup$
  • 4
    $\begingroup$ The use of "subjective Bayesian" is a bit of a self-sabotage (see). Modelling in general is full of subjectivism, the choice of a distribution for modelling a sample is also subjective. Even the choice of a goodness of fit test to check if a certain model is reasonable is subjective. $\endgroup$ – user10525 Nov 13 '12 at 11:46
  • 2
    $\begingroup$ I don't really agree with that, if someone considers "subjective" to be perjorative, that is their error. Sometimes when we mean probability, we really do mean subjective personal belief - I see no reason not to call it that if that is what is actually meant (choosing to accept only long run frequencies as the definition of probability is a purely subjective choice). $\endgroup$ – Dikran Marsupial Nov 13 '12 at 12:07
  • 1
    $\begingroup$ +1 thanks for the link, it's very enlightening. And also for the note on the difference between confidence and credible intervals as well. $\endgroup$ – Mehrdad Nov 13 '12 at 15:52
8
$\begingroup$

I believe this paper provides a more purposeful sense of the trade-offs in actual applications between the two. Part of this might be due to my preference for intervals rather than tests.

Gustafson, P. and Greenland, S. (2009). Interval Estimation for Messy Observational Data. Statistical Science 24: 328–342.

With regard to intervals, it may be worthwhile to keep in mind that frequentist confidence intervals require/demand uniform coverage (exactly or at least great than x% for each and every parameter value that does not have zero probability) and if they don't have that - they arn't really confidence intervals. (Some would go further and say that they must also rule out relevant subsets that change the coverage.)

Bayesian coverage is usually defined by relaxing that to "on average coverage" given the assumed prior turns out to be exactly correct. Gustafson and Greenland (2009) call these omnipotent priors and consider falliable ones to provide a better assessment.

$\endgroup$
  • 1
    $\begingroup$ +1 I never knew about this difference in restriction, thanks for pointing it out. $\endgroup$ – Mehrdad Nov 13 '12 at 15:55
3
$\begingroup$

I recommend looking at Exercise 3.15 of the freely-available textbook Information Theory, Inference and Learning Algorithms by MacKay.

When spun on edge 250 times, a Belgian one-euro coin came up heads 140 times and tails 110. 'It looks very suspicious to me', said Barry Blight, a statistics lecturer at the London School of Economics. `If the coin were unbiased the chance of getting a result as extreme as that would be less than 7%'. But do these data give evidence that the coin is biased rather than fair?

The example is worked out in detail on pp. 63-64 of the textbook. The conclusion is that the $p$-value is $0.07$, but the Bayesian approach gives varying levels of support for either hypothesis, depending on the prior. This ranges from a recommended answer of no evidence that the coin is biased (when a flat prior is used) to an answer of no more than $6:1$ against the null hypothesis of unbiasedness, in the case that an artificially extreme prior is used.

$\endgroup$
2
$\begingroup$

If someone were to pose a question that has both a frequentist and Bayesian answer, I suspect that someone else would be able to identify an ambiguity in the question, thus making it not "well formed".

In other words, if you need a frequentist answer, use frequentist methods. If you need a Bayesian answer, use Bayesian methods. If you don't know which you need, then you may not have defined the question unambiguously.

However, in the real world there are often several different ways to define a problem or ask a question. Sometimes it is not clear which of those ways is preferable. This is especially common when one's client is statistically naive. Other times one question is much more difficult to answer than another. In those cases one often goes with the easiest while trying to make sure his clients agree with precisely what question he is asking or what problem he is solving.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.