Would you say this is a trade off between frequentist and Bayesian stats?

Question

I'm trying to review frequentist and Bayesian in parallel. Let's say we are doing the typical scenario of estimating the population mean.

In frequentist stats, if sample size is large enough, we can use CLT to say that the sampling distribution of the sample mean is approximately normal. Then we can make inferences about the population mean. Here it seems like we DO NOT have to make any assumption about the shape of the population distribution because of CLT and as long as sample size is large enough.

In bayesian stats, it seems like we HAVE to make an assumption about the shape of the population distribution so that we can use the likelihood function to generate a posterior distribution. Thus, we don't get the benefits of CLT when doing Bayesian analysis.

Would you say a tradeoff is, in frequentist, you get to use CLT, thus allowing you to tackle a wider range of problems - especially if you know the population is non-normal? For example, there's no burden of proof to show if the population follows a gamma or beta distribution since it doesn't matter.

In Bayesian, since you always are using the likelihood function, you must always make an assumption about the distribution of the population data - limiting your flexibility to modeling populations with known distribution functions. And, does this imply that there is some burden of proof to show that your data follows whatever distribution assumption you make? HOWEVER, you do get to inject prior views of what you think the parameter value should center around. You also have an estimate of the distribution of the parameter itself, for situations where you think the parameter might be random.

Answer 1

This is not the tradeoff between Bayesian and frequentist statistics. The likelihood function describes the probability (density) of the observations given particular parameter values.

$$\mathcal{L(\theta | x)} = f(x\vert\theta)$$

It is reversing the dependent and independent parameters in the function, but it remains the same function.

---

Likelihood vs probability

This reversal occurs because often the behavior of the observations as a function of the parameters is known, but in practice we do not know the parameters and we know the observations.

see the German tank problem for example

Common problems in probability theory refer to the probability of observations $x_1, x_2, ... , x_n$ given a certain model and given the parameters (let's call them $\theta$) involved. For instance the probabilities for specific situations in card games or dice games are often very straightforward.

However, in many practical situations we are dealing with an inverse situation (inferential statistics). That is: the observation $x_1, x_2, ... , x_k$ is given and now the model is unknown, or at least we do not know certain parameters $\theta$.

The central limit theorem, or any simplification of the probability of the observations $x$ as a function of the parameters $\theta$, $f(x \vert \theta)$, applies to Bayesian and frequentist statistics in the same way. Both methods use the function $f(x \vert \theta)$ as starting point and the simplifications based on the CLT are applied to that function. See for instance this article 'Bayesian synthetic likelihood' by Price, Drovandi, Lee and Nott as an example where the CLT is applied in Bayesian statistics.

---

The tradeoff

The tradeoff between Bayesian and frequentist statistics is

from Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals

What is different?

The confidence interval is restricted in the way that it draws the boundaries. The confidence interval places these boundaries by considering the conditional distribution $X_\theta$ and will cover $\alpha \%$ independent from what the true value of $\theta$ is (this independence is both the strength and weakness of the confidence interval).

The credible interval makes an improvement by including information about the marginal distribution of $\theta$ and in this way it will be able to make smaller intervals without giving up on the average coverage which is still $\alpha \%$. (But it becomes less reliable/fails when the additional assumption, about the prior, is not true)

The Bayesian and Frequentists methods condition their intervals on different scales. See for instance the differences in the conditional coverage for credible intervals (in the sense of highest posterior density interval) and confidence intervals

In the image below (from the example in this answer/question) the expression of conditional probability/chance of containing the parameter conditional on the true parameter $\theta$ (left image) and conditional on the observation $x$ (right image).

This relates to Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

The confidence interval is constructed in such a way that it has the same probability of containing the parameter, independent from the true parameter value.

The credible interval is constructed in such a way that it has the same probability of containing the parameter, independent from the observation.

The trade-off is that the credible (Bayesian) interval allows making predictions with smaller intervals (which is advantageous, by contrast, imagine making the prediction that the parameter value is between $-\infty$ and $\infty$). But... the credible interval depends on prior information.

Answer 2

While that is not a direct answer to your question, it might also be interesting to note that the posterior will also behave like a normal distribution in large samples, a result that used to be relevant before MCMC methods became widely available.

So, in the sense that asymptotics are always an approximation in the sense that we never have infinitely large samples in practice, the difference may not be that large as we obtain a normal shape in either case when sample size becomes large, and the issue is maybe rather how good that approximation is.

Paraphrasing the discussion in Greenberg, Introduction to Bayesian econometrics:

Writing the likelihood function of a simple random sample $y=(y_{1},\ldots ,y_{n})$ \begin{eqnarray*} L\left( \theta |y\right) &=&\prod_{i=1}^{n}f\left( y_{i}|\theta \right) \\ &=&\prod_{i=1}^{n}L\left( \theta |y_{i}\right) \end{eqnarray*}

Log-likelihood: \begin{eqnarray*} l\left( \theta |y\right) &=& \ln L(\theta|y)\\ &=&\sum_{i=1}^{n}\ln L\left( \theta |y_{i}\right) \\ &=&\sum_{i=1}^{n}l\left( \theta |y_{i}\right) \\ &=&n\bar{l}\left( \theta |y\right) , \end{eqnarray*} where $\bar{l}\left( \theta |y\right) $ is the average contribution to the log-likelihood. Hence, \begin{eqnarray*} \pi \left( \theta |y\right) &\propto &\pi \left( \theta \right) L(\theta|y) \\ &=&\pi \left( \theta \right) \exp \left( n\bar{l}(\theta|y)\right) \end{eqnarray*} Now consider a Taylor series approximation of $l(\theta|y)$ around the maximum likelihood estimator $\hat{\theta}$ \begin{eqnarray*} l(\theta|y) &\approx &l(\hat{\theta}|y) \\ &&+\ l^{\prime}(\hat{\theta}|y)(\theta -\hat{\theta}) \\ &&+\ \frac{1}{2}l^{\prime\prime}(\hat{\theta}|y)(\theta -\hat{\theta})^{2}\\ &=& l(\hat{\theta}|y)-\frac{n}{2v}(\theta -\hat{\theta})^{2} \end{eqnarray*} with $$ v=\left[ -\frac{1}{n}\sum_{i=1}^{n}l^{\prime \prime }\left( \hat{\theta}|y_{i}\right) \right] ^{-1} $$

For large $n$ approximatively \begin{eqnarray*} \pi(\theta|y) &\propto &\pi(\theta) \exp(l(\theta|y)) \\ &\approx&\pi \left( \theta \right) \exp \left( l(\hat{\theta}|y)- \frac{n}{2v}(\theta -\hat{\theta})^{2}\right) \\ &\propto &\pi \left( \theta \right) \exp \left( -\frac{1}{2\left( v/n\right) }(\theta -\hat{\theta})^{2}\right) \end{eqnarray*} Here, we have dropped terms that do not depend on $\theta$ (such as the fixed value of the ML estimate).

The exp-term is the (non-normalized) density of a normal distribution with expectation $\hat{\theta}$ and variance $v/n$. By "likelihood dominance" (by which I mean that the likelihood dominates the prior in large samples) we get kind of a Bayesian analogue to the asymptotic normality of an estimator.

Answer 3

I see issues with your reasoning before even getting to the Bayesian setup.

1. We absolutely do have to make assumptions when we use the central limit theorem! At the very least, we assume that the variance is finite. Perhaps we’re usually willing to make this assumption, but finite variance is not a given; it is an assumption.

2. We shouldn’t have to rely on the central limit theorem to test a mean. That allows us to use z-tests and t-tests, but there are many other tests.

3. Means aren’t always the values of interest, and if we want to test the variance, for instance, the central limit theorem is not as helpful since we aren’t testing the z-stat that the CLT says is asymptotically normal.