Does Bayesian statistics bypass the need for the sampling distribution?

Question

Let's take the classic case where the population follows a normal distribution, observations are iid, and we want to estimate the mean of the population.

In Frequentist stats, we calculate the sample mean and sample variance from observed data. We know the sampling distribution of the sample mean follows a normal distribution with mean = population mean and variance = population variance/ sample size. Knowing that, we can test to see what is the % chance the population mean falls within some range, using the sampling distribution.

Seems like in Bayesian Stats, we don't need the sampling distribution of the sample mean to make inferences about the population mean. Let's say we assume the prior distribution of population mean is uniform. Using MLE, we determine the posterior distribution of the population mean. No where was the sampling distribution of the sample mean involved. To make inference about the population mean, we just directly calculate intervals from the posterior distribution. If the posterior distribution does not follow any known distribution, we just use our simulated distribution and do a frequency count between intervals?

Is my understanding correct?

Answer 1

We need to be precise about the terms "frequentist" and "Bayesian", because they are ambiguous. "Frequentism" can be understood as adhering to a specific interpretation of the meaning of probability, which doesn't necessarily imply that any specific methodology needs to be applied. In this sense one can be a frequentist without ever computing confidence intervals, and as a frequentist one can do Bayesian statistics (particularly if the prior has a frequentist interpretation). However more people use "frequentist" as referring to what is known as standard frequentist approaches to inference, estimation, tests and confidence regions. These rely crucially on the sampling distribution.

"Bayesian" on the other hand is often meant refer to a particular interpretation of the meaning of probability, usually understood as "epistemic" probabilities, although this is not the only possible meaning "Bayesian" can have. A frequentist probability would be defined by a data generating process in reality, whereas an epistemic probability refers to a state of knowledge of an individual (or science as a whole) about something rather than the real process that generates this "something". The concept of a "sampling distribution" is understood by frequentists as referring to a distribution of a statistic given that data are distributed according to the underlying real process. As there is no such thing in epistemic probability as an underlying real process defining probabilities, they don't have a sampling distribution in this sense. They don't "bypass" it, it is a concept that isn't meaningful for them. However a Bayesian still can think of a real process as a sampling process for choosing and processing their epistemic probabilities, in which case something can occur in the Bayesian computations that looks and acts like a sampling distribution.

Note: Following a remark by Sextus Empiricus I add that when writing about "frequentists" and "Bayesians" I don't intend to imply that anybody has to be either a frequentist or a Bayesian as a person. What I do think is that whenever we do data analysis involving probabilities, we should be clear what we think these probabilities mean, and this can be frequentist, or epistemic (various versions), or other. This shouldn't stop us from adopting a different interpretation in a different situation if it seems fit. So where I write "as a frequentist", I mean "as somebody who locally, in a given situation, takes a frequentist hat", etc.

Answer 2

Broadly speaking, Bayesian analyses satisfy the so-called likelihood principle, which means that all the information about parameters $\theta$ from an experiment that observed $X^\star$ is contained in the likelihood $$ L(\theta) \equiv p(X^\star | \theta), $$ which is crucially only evaluated at the observed $X^\star$.

Contrast this with the sampling distribution, $p(X|\theta)$ as a distribution in $X$. Crucially the data isn’t fixed to the observed value, and we instead consider this as a distribution in $X$.

Take for example the posterior, $$ p(\theta|X^\star) \propto p(X^\star | \theta) \pi(\theta). $$ It doesn’t depend on $p(X|\theta)$ anywhere other than at $X=X^\star$. So we would find the same posterior distribution for any sampling distribution $f$ so long as $f(X^\star|\theta) =p(X^\star|\theta)$. The posterior depends on the likelihood function, but not the entire sampling distribution.

Whilst the fundamental rules of Bayesian inference satisfy the likelihood principle, a few ideas violate it. For example, a few formal rules for constructing priors, e.g., so called reference priors and Jeffreys priors, use the likelihood function evaluated at all possible experimental outcomes (i.e., they use the sampling distribution). A few hybrid ideas, like the posterior and prior $p$-value, also violate it. I suppose ABC methods require the sampling distribution, but only as a means to ultimately approximate the likelihood at the observed data.

So, with a few exceptions, yes Bayesian statistics bypasses the need for the sampling distribution.

Answer 3

As Pohoua commented, your understanding is correct (but I would say not entirely*). Concepts like confidence intervals, p-values, and hypothesis tests are not calculated from the likelihood $f(\theta|x)$ with $x$ fixed, but instead with the pdf $f(x|\theta)$, where $\theta$ is fixed, which is a different slice of the joint distribution $f(x,\theta)$.Confidence intervals, p-value, and hypothesis tests, are different things than just the information from likelihood ratios.

So in that sense frequentist statistics 'needs'/'uses' the sampling distribution of the entire sample $f(x\vert \theta)$ (and as Tim Maks answer argues it does not need the sample distribution in many other ways). But in your example you speak about the sampling distribution of a statistic** as in a sample distribution of values like the sample mean and sample variance (an interpretation that you repeat in a question about the CLT). This more narrow sense of sampling distribution is not necessary/needed for frequentists statistics.

The sampling distribution (of a statistic) is not being used by frequentist statistics but it is the subject of many frequentist statistics.

Frequentist statistics is a lot about sampling distributions of an estimate/statistic, and in Bayesian statistics the sampling distribution hardly occurs. But, for several reasons, it would be wrong to say that Bayesian statistics is 'bypassing the use of the sampling distribution'.

A 'bypass' is not really the right word. Bayesian statistics is answering a different question than frequentist statistics (or at least takes a different point of view), and Bayesian statistics is no more bypassing the use of the sampling distribution than frequentist statistics is bypassing the use of the prior distribution. In a similar way a soccer/football player is not bypassing the use of a backhand and a tennis player is not bypassing the use of slidings, or a carpenter is not bypassing the use of paint and a painter is not bypassing the use of wood.

---

*Your understanding is incorrect in the sense that it relates to the role of the difference between population distribution and sample distribution of a statistic. This misunderstanding relates to something that you expressed in an earlier question, where you end up concluding that in a Bayesian analysis one can not use the CLT because we are not supposed to think of sample distributions when using a Bayesian analysis.

The likelihood function is not always so easy to compute and in that case one needs to use approximations instead of a direct analytical solution, like computational approximations by sampling. One may also use more analytical approximations, for instance like employing the CLT and a synthetic likelihood.

---

The bypass occurs at a different level.

A difference between Bayesian/frequentists statistics is that with a frequentist method you analyse the joint distribution $f(\boldsymbol{\theta},\mathbf{x})$ by considering the whole space of possible observations $x_1, x_2, \dots, x_n$, whereas with Bayesian methods you condition on the observation and only consider the values of the function $f(\boldsymbol{\theta},\mathbf{x})$ for a fixed single particular observation.

This difference makes that something like using a statistic (and the related sample distribution) is useful for a frequentist method because it greatly simplifies the computations and visualisation of the whole sample space for $\mathbf{x}$, by replacing it with the sample space for a statistic.

The Bayesian method does not bypass this sampling distribution. By this I do not mean that the Bayesian method needs the sampling distribution (it doesn't), but I mean that it is not a bypass.

What the Bayesian method is 'bypassing' is a need to make calculations with the joint distribution of parameters and observations $f(\boldsymbol{\theta},\mathbf{x})$ for values other than the actual observation, since the method conditions on the observation. And maybe the question is indirectly about that (but it is not so clear). The sampling distribution is in fact a shortcut (and not something cumbersome that is to be bypassed). With a frequentist method you can just as well work with the likelihood function and for instance do maximum likelihood estimation or confidence intervals. But the sample distribution of an estimate/statistic is the best language to do this.

Frequentist/Bayesian is not a dichotomy

There is no clear border what frequentist and Bayesian statistics means. One can do empirical Bayesian analysis or use Jeffreys prior in which case one is loosening the conditioning on the observation. And one can make an analysis that is frequentist-like but is not using an estimate/statistic and it's sample distribution.

Many people are just fitting curves with models by using some linear or non-linear fitting package and use something like an estimate of the inverse of the Fisher information matrix to express the variance/error of the estimate and there is no direct computation of the sample distribution.

Or one can do something else like using AIC/BIC to express goodness of fit, or use a Bayes factor or fiduciary or likelihood intervals.

When a sample distribution is used, then it is not really a tool that is something that can be 'bypassed'. The sample distribution is the goal itselve. And if you like you could apply it to a Bayesian estimate (although it makes less sense in such a setting).

In frequentist statistics you don't have to compute a sampling distribution of an estimate.

In frequentist statistics, or whatever it is, you don't have to calculate these statistics and their sample distribution. You can also work only with the likelihood function in order to make point or interval estimates.

The method in the example of the question, with the sampling distribution of the mean is derived from maximum likelihood estimation and effectively equivalent. You do not need a sample distribution of a statistic or estimate (but it does make the analysis simpler) to compute it.

For instance to make a maximum likelihood estimate for a population mean $\mu$ of a normal distributed population we use the likelihood function:

$$\mathcal{L}(\mu \vert x_1,x_2,\dots,x_n ,\sigma) = \prod_{1\leq i \leq n} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}\left(\frac{x_i-\mu}{\sigma}\right)^2} $$

and the $\mu$ that maximizes this function is the MLE estimate.

This is very similar to Bayesian maximum a posteriori estimate, which is just maximizing $$ f_{posterior}(\mu \vert x_1,x_2,\dots,x_n ) \propto \mathcal{L}(\mu \vert x_1,x_2,\dots,x_n ,\sigma) \cdot f_{prior}(\mu)$$

The only difference is that the likelihood function is multiplied with the prior probability.

Similarly for confidence intervals, one could use z or t-statistics, but effectively those statistics are shortcuts for the more difficult geometrical shape of the density distribution in all coordinates of the observation $\mathbf{x}$. We can derive p-values, statistical tests (and related confidence intervals) by only considering whether an observation is 'extreme' or not. And this can be defined by the likelihood function without considering a statistic/estimate and it's sample distribution (e.g. likelihood ratio test, if the likelihood is below a certain value than the value is not inside the confidence region). This view is also illustrated here where a test is not viewed by considering the sampling distribution of a statistic, but by considering the PDF of the whole data (in that case the data is two variables X and Y).

The sample distribution occurs particularly in the method of moments. We can use the moments of a sample to estimate the moments of a distribution and in that case we may wish to express the sample distribution of the moments of a sample. But the method of moments is different from maximum likelihood estimation (but maybe this is already not frequentist?), and we do not use this sample distribution in every type of analysis.

---

**This question is not entirely clear about what is meant with 'sampling distribution' (an ambiguity which causes two diverging type of answers). For this answer I interpret sampling distribution as the distribution of a statistic or the distribution of an estimate. And I interpret a statistic in the sense of R.A. Fisher "statistic may be defined as a function of the observations designed as an estimate of the parameters". In this answer I argue that you do not need such sampling distributions (e.g. you do not need to work as you describe, compute sample mean and sample variance. Instead, you can use the likelihood/probability function directly. But the sampling distribution, and related sufficient statistics, does make it easier.). I do not interpret sample distribution more generally as the distribution of observations/samples.

Answer 4

No, your understanding is not correct.

First, frequentist statistics does not allow us to "test to see what is the % chance the population mean falls within some range, using the sampling distribution". More precisely, frequentist statistics do not make probability statements on the population mean --- they only make probability statements on the estimates of the population mean. This is a well-known limitation of frequentist statisics that has caused much confusion and spawned many related questions on cross validated. (See, e.g., this thread.)

Secondly, in Bayesian stats, we do have the sampling distribution of the sample mean. We may not specifically refer to it, though. As others have mentioned though, Bayesian inference is a type of likelihood inference. Once you have defined your likelihood, you have, by deduction, the sampling distribution of the sample mean. Whether you use that distribution is another matter though.

In fact, there are variants of frequentist inference that bypass the likelihood, in that they only work with the moments of the sampling distribution and not the full distribution. See, e.g. the literature on method of moments. However, a "pure" Bayesian analysis will always involve the likelihood even if it is intractable, and therefore a sampling distribution is always implied.

Just for completeness, there are also variants of Bayesian inference that do not involve a full definition of the likelihood, and hence I specifically refered to "pure" Bayesian inference earlier.

An example for illustration \begin{align} X_i &\overset{iid}{\sim} N(\mu, 1) \tag{1} \\ \mu &\sim N(0, \sigma^2) \tag{2} \end{align} Here, equation (1) implies \begin{equation} \bar{X} = \sum_i^n X_i \sim N(\mu, 1/n) \tag{3} \end{equation} which is the sampling distribution of $\bar{X}$. Of course, in Bayesian inference, we typically don't care about (3), since our interest will usually be in \begin{equation} p(\mu|X) = \frac{p(X|\mu)p(\mu)}{p(X)} \end{equation} However, it happens in this case, since $\bar{X}$ is a sufficient statistic, that \begin{equation} p(\mu|X) = p(\mu|\bar{X}) = \frac{p(\bar{X}|\mu)p(\mu)}{p(\bar{X})} \end{equation} Thus, you can also use the sampling distribution (3) to derive your posterior distribution, if you like.