Simple example where Bayesian > Frequentist (Unambiguously)? [duplicate]

Question

I'm trying to understand the main appeal of Bayesian methods & whether they are indeed capable of offering more than regular frequentist methods. The thing is my impression is that by default the prior should be 'uninformative' (i.e. Objective Bayesian movement). And I've heard doing this gives you similar results to frequentist?

But I've also heard that Bayesian can be more sample efficient? Is this true? For example if you have a large prior sample (say a control group's data accumulated across several different research projects), but a small experimental group say <= 100 samples. Does bayesian handle inference in this situation better than frequentist somehow?

P.S. Specifics given in the body of my post are context. But question remains what is in the title.

Answer 1

I am holding a coin, of unknown bias, in my hand. I flip it once, and it comes down heads. A friend asks me what a fair bet would be for the next flip to be heads.

What's the frequentist point estimate of the probability that the next flip is heads?

(The answer is less obvious than it may appear, as "frequentist" does not specify the estimation procedure.)

As a Bayesian, this is straightforward. Because a vast amount of randomness in the result is due to my erratic coin-flipping procedure and the symmetry induced by having the coin either face up or face down when flipping it, I find it extremely unlikely that a coin flip will be biased by more than 1% (49%-51%). I put a Beta(10,000, 10,000) prior on the probability of a head, which gives me a prior mean of 50% and a prior standard deviation of 0.005. My posterior is therefore a Beta(10,001, 10,000) distribution, and the posterior mean is 10,001 / 20,001 or 0.500025.

Edit in response to comments:

Bayesian statistics is focused on "small" samples, as asymptotically in (most) situations Bayesian and MLE results become the same. However, in my past applied work - estimating degradation rates of solar panels based on a limited number of years of experimentation - even sample sizes of 100 (application specific) aren't as informative as you might think they would be, and prior information can really make a difference.

Also, why limit yourself to a small experimental group when you have sufficient prior information available to, indirectly, allow you to have a large experimental group? In a way, it's the experimental group you're really interested in, after all.

Answer 2

Just to give an example that hasn't been mentioned yet: in logistic regression, a regularizing term based on the Jeffreys prior was shown by Firth to produce unambiguously better estimates (in terms of reducing both bias and variance) than vanilla MLE. And as a cherry on top it also mitigates the problem of perfect separation (see Heinze and Schemper). There's already a great post explaining why it works, so I'm not going to repeat it here.

Ironically it is not commonly "advertised" as a Bayesian method, but rather a form of penalized maximum likelihood estimation.

Answer 3

The frequentist definition of a probability is based on a long run frequency, the Bayesian definition is based on plausibility or state of knowledge (whether subjective or objective). The Bayesian approach is going to be better, in the sense of being able to give a direct answer, if the question is not asking for a long run frequency (such as the probability that some proposition is true, which doesn't have a long run frequency, it is either true or it isn't).

Now frequentists can give indirect answers to questions about things like the truth of a proposition, usually by inventing some fictitious population which may have long run frequencies. However it is an indirect answer that will often be treated as a direct answer leading to misunderstandings such as the p-value fallacy (the p-value is not the probability that the null hypothesis is true, and indeed a frequentist fundamentally cannot attach a numeric probability to that anyway!).

However, Baysian analysis often requires the ability to perform numerical integration, which most of us find tricky, whereas the frequentist framework has proven well suited to a "statistics cookbook" approach and is often used by practitioners who don't really understand the underlying principles, but can use it because it is comparatively easy to implement. Hopefully packages such as Stan will make it easier to apply the Bayesian approach with time? This makes it difficult to say whether a Bayesian approach is unambiguously better than a frequentist approach, because there are multiple criteria that govern which framework to use.

Horses for courses, ideally choose the framework that answers your question directly... unless it is too tricky in which case reach for the cookbook ;o)

With regards to "informative" priors, in some cases we have knowledge from e.g. physics that mean that uninformative priors are objectively (in the everyday sense of the word) implausible. Our statistical analyses should include domain knowledge where appropriate. The XKCD cartoon about frequentist-v-Bayesian gives a good example of what happens if we try for purely data driven inference in situations where we do have useful domain knowledge!