# Bayesian and frequentist connections regarding the central limit theorem

I have been wondering how the central limit theorem may be useful in Bayesian statistics with potentially misspecified model distribution. Suppose $$x$$ is a random variable that follows an unknown (and possibly non-normal) population distribution $$P_x$$. $$P_x$$ depends on two parameters, the mean $$\mu$$ and variance $$\sigma^2$$. We have observed i.i.d. samples $$x_{1:n}=(x_1,...,x_i,...,x_n)$$ from $$P_x$$. I am interested in the unknown parameter $$\mu$$.

Now consider the case that $$\sigma^2$$ is known. (1) A common frequentists’ point estimate would be the sample mean $$\hat{\mu}=\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i$$. According to the central limit theorem (CLT), $$\bar{x}\sim N(\mu,\sigma^2/n)$$ as $$n\to\infty$$. (2) But from an objective Bayesian perspective (using a flat prior on $$\mu$$), how can we obtain the posterior distribution $$\mu\mid(x_{1:n},\sigma^2)$$ when $$n\to\infty$$ without making any distributional assumptions? May be applying the Bernstein-von Mises theorem?

The Bernstein-von Mises theorem says something about the asymptotic behavior of the posterior; it does not inform your choice of likelihood.

There are approaches to unknown likelihoods in Bayesian statistics, most of which, however, are more complicate than your problem requires.

Because if all you want to model is a mean $$\mu$$ and a variance $$\sigma^2$$, you're effectively estimating a second-order approximation to a unimodal symmetric distribution about its mode. The distribution that is its own 2nd-order approximation is the Gaussian.

Bottomline: just model $$P_x$$ as a Gaussian, reminiscent of QMLE.

• Thanks. One thing remains unclear for me: Will the Gaussian approximation be better when $N\to\infty$? I mean, $\frac{1}{n}\log p(x_i;\mu,\sigma^2)$ may be more accurately approximated asymptotically? Sep 27, 2023 at 0:52
• The convergence of the likelihood function is more than just a Bayesian question. You'll probably find the answer here. Sep 27, 2023 at 3:12

Thanks for the help of @Durden. Here is my answer.

First, some relevant questions on this site: question 1 and question 2 are very relevant. @Sextus Empiricus ’s answer seems to utilize the idea of Bayesian synthetic likelihood:

So we could use the likelihood function: $$\mathcal{L}(\mu \vert \bar{X}) \approx \frac{1}{\sqrt{2 \pi \sigma^2/n}} \exp \left(\frac{(\bar X - \mu)^2}{2 \sigma^2/n} \right)$$

The summary statistic $$\bar{x}$$ is observed and we can reasonably assume it follows a normal distribution per CLT. Then the posterior $$\mu\mid x_{1:n}$$ (here $$\sigma^2$$ is omitted because it’s known and fixed) becomes $$\mu\mid\bar{x}$$. However, I have come up with a different explanation than replacing $$x_{1:n}$$ with $$\bar{x}$$.

Then, I provide an approach using Taylor expansion of log-likelihood. We konw in objective Bayes, the posterior distribution depends on the sum likelihood $$p(x_{1:n}\mid\mu)$$. For notation conveniency, let

$$l(\mu)=\log p(x\mid\mu)$$, the population log-likelihood.

$$l_i(\mu)=\log p(x_i\mid\mu)$$, the individual log-likelihood.

$$\bar{l}(\mu)=\frac{1}{n}\log p(x_{1:n}\mid\mu)=\frac{1}{n}\sum_{i=1}^{n}l_i(\mu)$$, the average log-likelihood. This corresponds to our target sum of log-likelihood.

$$E[l(\mu)]=\mathrm{plim}_{n\to\infty}\bar{l}(\mu)$$, the expected log-likelihood. The asymptotics can be used in this way.

Following this answer, the expected log-likelihood can be expanded in the parameter space (differing from the sample space) as a second-order Taylor series:

$$\bar{l}(\mu)=\bar{l}(\mu_0)+\bar{l}’(\mu_0)(\mu-\mu_0)+\frac{1}{2}\bar{l}’’(\mu_0)(\mu-\mu_0)^2+o^2.$$

Let $$\mu_0$$ be the maximum likelihood estimator (MLE) point, i.e. $$E[l’(\mu_0)]=0$$. Also, this answer said:

Then one goes to show that the remainder converges uniformly in probability to zero...

So asymptotically ($$n\to\infty$$) we have

$$E[l(\mu)]\approx E[l(\mu_0)]+\frac{1}{2}E[l’’(\mu_0)](\mu-\mu_0)^2,$$

which is essentially a quadratic approximation. This shows that asymptotically, the true average (also the expected) log-likelihood can be quadratically approximated. How do we do this quadratic approximation? A practical way is to model the individual log-likelihood $$l_i(\mu)$$ as Gaussian, whose highest order of Taylor expansion is 2. Then their sum and average will be just quadratic.

So my conclusion: In a large-sample setting, the problems of modelling the unkonwn population distribution as Gaussian will be alleviated, which is a different perspective compared to frequentists’ CLT.

However, I found that the Gaussian approximation error may not converge to zero at the second-order level even asymptotically, and the remaining error comes from the Fisher information differences of two models! The true asymptotic posterior and the Gaussian-approximted asymptotic postetior are both normal, but with different variances depending on their respective Fisher information matrices, as the Bernstein-von Mises theorem shows. This is a problem not specific to Bayesians. Frequentists too suffer from so, as the QMLE estimator does not reach the Cramer-Rao lower bound. So relying on asymptotic inference may be problematic!

The case of unknown $$\sigma^2$$ would be a bit more tricky and I leave this unfinished.