How do we stop bayesian estimates from being overconfident?

Question

I posted this question today about strategies for regression with small sample sizes. I thought Bayesian regression might be a good choice here: Bayesian regression for correcting small sample sizes

One thing I was wondering about, I read that the advantage of Bayesian inference is that it can prevent the estimates from having very large variances in situations where we are not confident about our data or have small samples. In such cases, the estimates can be shrunk towards the prior compared to data, thus reducing the variance of the parameter estimates compared to the non-Bayesian approach.

I was just wondering, how can we prevent/how do we know that this reduction in variance from Bayesian is not a blatant underestimation of the variance? It seems to me that the only thing to do is choose a very large prior (e.g. non informative prior) to reflect our uncertainty, however it seems that doing this will unshrink the variance back to the non-Bayesian direction. Other than that, it sounds like there is no free lunch, and that Bayesian inference only has advantages if you are truly confident in your choice of priors?

Answer 1

You are correct with this statement:

Other than that, it sounds like there is no free lunch...

Bayes, like any other thing in statistics, is never going to be a silver bullet for any issue. Bayes has it's tradeoffs, chief among them being the difficulty in always having a Bayesian version of analysis (particularly for cutting-edge methods), computational time (which has reduced substantially in recent years but still a hill to climb), and as you say prior specification, which can be difficult.

...and that Bayesian inference only has advantages if you are truly confident in your choice of priors?

However, this is not entirely true. I think a lot of people learn Bayes as "it's the method that uses past research to tell you what the estimates should be." That of course is helpful. If we have past research that explicitly gives us parameter estimates that are plausible, we can use this directly. But often I think people don't discuss enough that Bayesian priors are also just useful for constraining your parameters to act like they should regardless of past information.

To give an example, suppose I am investigating the differences in native and non-native speakers of English on some dependent variable. The reference group (used for the intercept) is the non-native group and the comparison group (the slope) is the native group. I may have no idea what the mean differences are between the two, but it would be extremely odd if the native group performed worse than the non-native group on English tests. So one way of getting around this is forcing the prior for the native group to have a non-negative distribution, something like a truncated normal, in order to force the values to be positive. In this way, even if we do not have a strong prior from past research, we are still using our judgement in a way that could be considered superior to frequentist thinking because it's not just winging it with the parameter estimates.

Sometimes this will be model-contingent. Suppose we are instead running a regression where the predictors are z-score standardized before entering the model. In this case, it is likely impossible to have a large amount of variance around the parameter estimates and it tends to help speed up estimation given this treatment. This is because our predictor and outcome can only range from values on the standard normal from around $-3$ to $3$ and their association is normally only going to have a coefficient ranging from $-1$ to $1$. So explicitly telling your program of choice that this the distribution of coefficients is $\text{Normal}(0,10)$ would be a serious mis-step.

Having said that, it is true that very informative priors are always going to beat weakly informative priors in terms of their justification. The key will always be your specific problem you are dealing with. In samples that are large, frequentist and Bayesian estimates can be remarkably similar, where the data will dominate the prior and the prior will consequently become less useful. In a small sample context, one can be less trusting of frequentism even if we have only weakly informative priors. Here, even using some reason will be helpful, regardless of whether it is informed heavily by past research.

Answer 2

All analyses make assumptions which need to be justified

Moving between a frequentist estimation (or a flat prior) and Bayesian estimation is not really adding assumptions, it is changing your assumptions. It is quite obvious to note that an analysis will perform well when its assumptions are close to reality and poorly when they are not. So the question is: are there situations where a specific prior is more justified than a flat prior/frequentist approach?

We'll also note that for many cases, Bayesian analysis + prior gives the same uncertainty estimates as frequentist analysis + regularization, so you don't have to buy all philosophical commitments.

In a frequentist analysis of say a difference in means, you implicitly assume that before you see the data a difference of 0 is equally plausible as a difference of 1 million or a difference of $-10^{134}$. Bayesians would argue this is usually a ridiculous assumption. In practice you can often (though not always) deduce a pretty narrow prior just from very uncontroversial assumptions.

A good example is when your model works on the log scale. Let's say we are running a negative binomial regression with log link for the number of days spent in hospital with treatment as predictor. I.e. your coefficient of interest corresponds to fold-change in the mean number of days in hospital. More than 50x increase/decrease in duration of hospital stay is completely implausible across medical interventions, so let's start there. Putting 95% of prior weight within $[-\log(50), \log(50)]$ results in something like $N(0, 2)$, i.e. quite narrow prior on the coefficient.

So we can rephrase your statement it in the other direction. Let's assume we start with a Bayesian analysis and now we consider replacing it with a frequentist one:

how can we prevent/how do we know that this increase in variance from frequentist is not a blatant overestimation of the variance

Where I take that by "variance" you mean some measure of uncertainty about the estimate (e.g. the width of a credible/confidence interval). My argument is that often there is a good reason to suspect that the frequentist approach without regularization indeed gives overly wide uncertainty. Though for most practical problems and datasets the difference is pretty small.