19
$\begingroup$

As someone who started by studying classical statistics where the Central Limit theorem is key to making inferences, and only later now am studying Bayesian statistics, I was late to realize that the Central Limit Theorem has a much smaller role to play in Bayesian statistics. Does the Central Limit Theorem play any role at all in Bayesian inference/statistics?

Later Addition: From Bayesian Data Analysis by Gelman et.al. 3rd edition - on the central limit in the Bayesian context. "This result is often used to justify approximating the posterior distribution with a normal distribution" (page 35). I went through a graduate course in Bayesian Statistics without encountering an example in which the posterior was approximated with a normal distribution. Under what circumstances is it useful to approximate the posterior with a normal distribution?

$\endgroup$
10
  • 3
    $\begingroup$ Sextus answer to this question seems relevant stats.stackexchange.com/questions/570503/… $\endgroup$
    – Manuel
    Jan 10, 2023 at 13:18
  • 5
    $\begingroup$ there are Bayesian versions of central limit theorems, but they play a fundamentally different role because Bayesians (in broad terms) don't need asymptotics to produce inference quantities; rather, they use simulation to get "exact" (i.e. up to numerical error) posterior quantities. There's no need to lean on asymptotics to justify a credible interval, as one would to justify a confidence interval based on the hessian of the likelihood, say. $\endgroup$ Jan 10, 2023 at 13:52
  • 1
    $\begingroup$ @JohnMadden, that's a legit point. In fact, a coherent answer can be made out of it, imo. $\endgroup$ Jan 10, 2023 at 14:14
  • 1
    $\begingroup$ @ColorStatistics I am still pondering a bit over this question. For instance, I am not a fan of dividing statisticians into Bayesian and frequentist statisticians, as if this is a property of the statistician rather than the technique. And regarding the central limit theorem, I believe that this is such a general principle that the question is like asking "what is the role of x in Bayesian statistics" where we can fill in for x something basic and general like 'integration', 'binomial distribution', 'cloud computation ', etc. $\endgroup$ Jan 10, 2023 at 17:50
  • 2
    $\begingroup$ Check the keyword Bernstein-von Mises as the Bayesian version of the CLT. There is a large and current literature on the topic. $\endgroup$
    – Xi'an
    Jan 11, 2023 at 10:14

3 Answers 3

17
$\begingroup$

The Frequentist needs asymptotics because the things they are interested in, like intervals which cover the true value 95% of the time or tests which have a false positive rate of less than 5% when the null hypothesis is true, typically do not exist. If the model is linear and the errors Gaussian, we can get exact confidence intervals, but rarely otherwise. However, we can build intervals which cover the truth asymptotically in very broad classes of models by exploiting a quadratic approximation of the likelihood.

The Bayesian does not have this problem. Given a prior and posterior, the 95% credible interval is a very well defined concept: any interval which contains 95% of the posterior mass. Likewise, Bayes factors can be defined in terms of posterior quantities. Life is easier in the linear/Gaussian case because these quantities will be available in closed form. But even in the general case, we can precisely define these quantities mathematically, and thus use the tools of numerical analysis to compute approximations. Most prominent would be Markov-chain Monte Carlo. The Bayesian, given infinite computing power, can thus get arbitrarily close to "correct" credible intervals/posterior means/etc for any sample size and any model.

[Of course, if the prior is not good, these quantities are utterly meaningless. Even if it is, they do not have any guarantee of relating to anything in the "real world" like a frequentist interval does; they are simply the results of "thinking rationally".]

You also ask about how Bayesians might avail themselves of the CLT. This comes in handy if the Bayesian doesn't have infinite computing power. MCMC is guaranteed to work eventually, but it might take too long on your computer. If the reason the posterior is expensive to evaluate is because you have a lot of data, we can deploy a normal approximation to the posterior. Various ways exist to choose the parameters of the approximating normal; perhaps the most popular is the Laplace Approximation, which uses a quadratic approximation of the posterior near its mode (this might remind you of frequentist asymptotics).

$\endgroup$
20
  • 2
    $\begingroup$ +1: I was not aware of the Laplace Approximation; thank you $\endgroup$ Jan 10, 2023 at 17:12
  • 3
    $\begingroup$ +1. Coherent indeed. $\endgroup$ Jan 10, 2023 at 17:13
  • 2
    $\begingroup$ @SextusEmpiricus unfortunately I'm struggling to pin down which part of my answer is in conflict with your comments. $\endgroup$ Jan 10, 2023 at 17:31
  • 2
    $\begingroup$ @ColorStatistics I agree; and FWIW, my grad Bayes class didn't mention Bayesian CLTs either :) $\endgroup$ Jan 10, 2023 at 17:32
  • 2
    $\begingroup$ Erm, MCMC is not Bayesian per se, i.e., the CLT used in MCMC is not for inference purposes but for Monte Carlo convergence control. $\endgroup$
    – Xi'an
    Jan 11, 2023 at 10:16
6
$\begingroup$

The limit theorem is 'central'

The central limit theorem (CLT) has a central role in all of statistics. That is why it is called central! It is not specific to frequentist or Bayesian statistics.

Note the early (and possibly first ever) use of the term 'central limit theorem' by George Pólya who used it in the article "Über den zentralen Grenzwertsatz der Wahrscheinlichkeitsrechnung und das Momentenproblem"

Das Auftreten der Gaußschen Wahrscheinlichkeitsdichte $e^{-x^2}$ bei wiederholten Versuchen, bei Messungsfehlern, die aus der Zusammensetsung von sehr vielen und sehr kleinen Elementarfehlern resultieren, bei Diffusionafurgängen usw. ist bekanntlich aus einem und demselben Grenzwertsatz zu erklären, der in der Wahrscheinlichkeitsrechnung ein zentralen Rolle spielt.

emphasis is mine.

The principle behind the limit is applied whenever we use a normal distribution

The CLT describes the tendency of sums of variables to approach a normal distribution and that is independent from how you would wish to analyse the variables, whether it is frequentist or Bayesian. Such sums occur anyware.

It is arguable that whenever a normal distribution is used, then it is indirectly an application of the central limit theorem. A normal distribution does not occur as an atomic distribution. There is nothing inherently normal distributed and when a normal distribution 'occurs' then it is always due to some process that sums several smaller variables (e.g like a Galton board where a ball is hitting multiple times a pin before ending up in a bin). And such sums can be approximated by a normal distribution.

The use of the normal distribution can have other motivations. For instance, it is the maximum entropy distribution for a given mean and variance. But in that case, it still indirectly relates to the CLT as we can see a maximum entropy distribution as arrising from many random operations that preserve some parameters (like in the case of the normal distribution, the mean and variance are preserved). When we add up many variables with a given mean and variance, then the resulting distribution is likely gonna be something with a high entropy, ie something close to the normal distribution.

The CLT is such a general principle that the question is like asking "what is the role of 'integration' in Bayesian statistics". Or fill in any other trivial process in place of CLT.

Practical application of CLT

It might be that in practice one observes a tendency for textbooks or statisticians/fields to often apply a particular technique, frequentist or Bayesian, and use relatively more or less often a normal approximation. But, that is in principle not related to those fields.

In practice a particular technique might be preferred. For instance when approximating intervals, then one can use a normal distribution as approximation, but that is not a neccesity. One can also use a Monte Carlo simulation to estimate the distribution or sometimes there is a formula for the exact distribution.

Possibly Bayesian approaches use the normal approximation less often because they are in a situation where they use Monte Carlo simulation/sampling already anyway (to find a solution for large intractable models).

It can be that in particular fields the models are too complex to apply a normal distribution approximation and that those fields also often apply Bayesian techniques. That doesn't make the role of the CLT is smaller for Bayesian techniques. At least not in principle.

There is a large amount of scientists that use nothing much more than simple things like ANOVA, chi-squared tests, ordinary least squares fits, or small variations of it. Those techniques happen to be frequentist and use a normal distribution approximation. Because of that it might seem like frequentist techniques often use the CLT but it doesn't rely on it in principle.


Related:

How would a bayesian estimate a mean from a large sample?

Would you say this is a trade off between frequentist and Bayesian stats?

$\endgroup$
26
  • 1
    $\begingroup$ Thank you for this perspective. If I can try to boil down your answer, you're saying that any time we use a Normal prior in Bayesian Inference we're standing on the shoulders of the CLT. Makes sense. On one hand, much of classical/frequentist inference would not exist were it not for the CLT; the CLT is directly making the inference possible. On the other hand, in Bayesian Inference, only in the particular case that we assume the prior to be Normal, does the CLT come into play; and it does so very indirectly, reminding us that the Normal distribution phenomenon is inherently linked to the CLT. $\endgroup$ Jan 10, 2023 at 18:35
  • 1
    $\begingroup$ @SextusEmpiricus but bootstrap's guarantees are asymptotic; I thought you were talking about model-based simulation of some sort. $\endgroup$ Jan 10, 2023 at 19:48
  • 2
    $\begingroup$ "The central limit theorem (CLT) is central to any field of statistics. That's why it is called central!" << No, it's called the "central limit" theorem because it literally says that the limit of a sum of independent variables will always be centered. $\endgroup$
    – Stef
    Jan 11, 2023 at 10:33
  • 1
    $\begingroup$ @Stef I agree that there can be many interpretations to the term 'central' but at least the inventor of the term George Pólya who used it in 1920 intended the term 'central' to refer to the importance and general application of the limit. $\endgroup$ Jan 11, 2023 at 11:47
  • 1
    $\begingroup$ @ChristianHennig I agree that the CLT does not hold exactly in nature (already infinity is never obtained, and the idea of the limit is an ideal theoretical concept) and nothing is truly normal distributed. But that is beyond my point. The idea is that normal approximations work and are ubiquitous wherever sums of variables arrise. As a consequence, whenever a Bayesian technique uses a normal distribution then in a way it is using the CLT or the idea behind it. $\endgroup$ Jan 11, 2023 at 16:18
4
$\begingroup$

Reproduced verbatim from the Wikipedia page:

In Bayesian inference, the Bernstein-von Mises theorem provides the basis for using Bayesian credible sets for confidence statements in parametric models. It states that under some conditions, a posterior distribution converges in the limit of infinite data to a multivariate normal distribution centered at the maximum likelihood estimator with covariance matrix given ${\displaystyle n^{-1}I(\theta _{0})^{-1}}$, where $\theta _{0}$ is the true population parameter and ${\displaystyle I(\theta _{0})}$ is the Fisher information matrix at the true population parameter value.

The Bernstein-von Mises theorem links Bayesian inference with frequentist inference. It assumes there is some true probabilistic process that generates the observations, as in frequentism, and then studies the quality of Bayesian methods of recovering that process, and making uncertainty statements about that process. In particular, it states that Bayesian credible sets of a certain credibility level $\alpha$ will asymptotically be confidence sets of confidence level $\alpha$, which allows for the interpretation of Bayesian credible sets.

With the reference

van der Vaart, A.W. (1998). "10.2 Bernstein–von Mises Theorem". Asymptotic Statistics. Cambridge University Press. ISBN 0-521-49603-9.

$\endgroup$
1
  • 3
    $\begingroup$ Thank you for putting the Bernstein-von Mises theorem on the map for me. My guess is that in BDA Andrew Gelman had this theorem in mind when he referred to approximating the posterior with a multivariate normal distribution. I find it quite fascinating. $\endgroup$ Jan 11, 2023 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.