14
$\begingroup$

Let's take the classic case where the population follows a normal distribution, observations are iid, and we want to estimate the mean of the population.

In Frequentist stats, we calculate the sample mean and sample variance from observed data. We know the sampling distribution of the sample mean follows a normal distribution with mean = population mean and variance = population variance/ sample size. Knowing that, we can test to see what is the % chance the population mean falls within some range, using the sampling distribution.

Seems like in Bayesian Stats, we don't need the sampling distribution of the sample mean to make inferences about the population mean. Let's say we assume the prior distribution of population mean is uniform. Using MLE, we determine the posterior distribution of the population mean. No where was the sampling distribution of the sample mean involved. To make inference about the population mean, we just directly calculate intervals from the posterior distribution. If the posterior distribution does not follow any known distribution, we just use our simulated distribution and do a frequency count between intervals?

Is my understanding correct?

$\endgroup$
  • 4
    $\begingroup$ Yes, your understanding is correct, in Bayesian analysis you don't compute estimators of a parameter, but rather a posterior distribution of this parameter. Thus you don't compute any sampling distribution of estimators to get confidence intervals, the posterior is already enough to give credible intervals. $\endgroup$ – Pohoua May 22 at 8:35
  • $\begingroup$ Thank you for the help! $\endgroup$ – confused May 22 at 8:41
  • 3
    $\begingroup$ we have the formula $$p(\theta|x)\propto p(\theta)p(x|\theta)$$ it certainly shows the posterior depends on the sampling distribution! $\endgroup$ – probabilityislogic May 23 at 9:57
  • $\begingroup$ @probabilityislogic no it shows that the posterior depends on the likelihood function evaluated only at the observed data $\endgroup$ – innisfree May 23 at 10:41
  • 1
    $\begingroup$ @confused how do you define 'sampling distribution'? $\endgroup$ – Sextus Empiricus May 23 at 12:28
6
$\begingroup$

As Pohoua commented, your understanding is correct (but I would say not entirely*). Concepts like confidence intervals, p-values, and hypothesis tests are not calculated from the likelihood $f(\theta|x)$ with $x$ fixed, but instead with the pdf $f(x|\theta)$, where $\theta$ is fixed, which is a different slice of the joint distribution $f(x,\theta)$.Confidence intervals, p-value, and hypothesis tests, are different things than just the information from likelihood ratios.

So in that sense frequentist statistics 'needs'/'uses' the sampling distribution of the entire sample $f(x\vert \theta)$ (and as Tim Maks answer argues it does not need the sample distribution in many other ways). But in your example you speak about the sampling distribution of a statistic** as in a sample distribution of values like the sample mean and sample variance (an interpretation that you repeat in a question about the CLT). This more narrow sense of sampling distribution is not necessary/needed for frequentists statistics.

The sampling distribution (of a statistic) is not being used by frequentist statistics but it is the subject of many frequentist statistics.

Frequentist statistics is a lot about sampling distributions of an estimate/statistic, and in Bayesian statistics the sampling distribution hardly occurs. But, for several reasons, it would be wrong to say that Bayesian statistics is 'bypassing the use of the sampling distribution'.

A 'bypass' is not really the right word. Bayesian statistics is answering a different question than frequentist statistics (or at least takes a different point of view), and Bayesian statistics is no more bypassing the use of the sampling distribution than frequentist statistics is bypassing the use of the prior distribution. In a similar way a soccer/football player is not bypassing the use of a backhand and a tennis player is not bypassing the use of slidings, or a carpenter is not bypassing the use of paint and a painter is not bypassing the use of wood.


*Your understanding is incorrect in the sense that it relates to the role of the difference between population distribution and sample distribution of a statistic. This misunderstanding relates to something that you expressed in an earlier question, where you end up concluding that in a Bayesian analysis one can not use the CLT because we are not supposed to think of sample distributions when using a Bayesian analysis.

The likelihood function is not always so easy to compute and in that case one needs to use approximations instead of a direct analytical solution, like computational approximations by sampling. One may also use more analytical approximations, for instance like employing the CLT and a synthetic likelihood.


The bypass occurs at a different level.

A difference between Bayesian/frequentists statistics is that with a frequentist method you analyse the joint distribution $f(\boldsymbol{\theta},\mathbf{x})$ by considering the whole space of possible observations $x_1, x_2, \dots, x_n$, whereas with Bayesian methods you condition on the observation and only consider the values of the function $f(\boldsymbol{\theta},\mathbf{x})$ for a fixed single particular observation.

This difference makes that something like using a statistic (and the related sample distribution) is useful for a frequentist method because it greatly simplifies the computations and visualisation of the whole sample space for $\mathbf{x}$, by replacing it with the sample space for a statistic.

The Bayesian method does not bypass this sampling distribution. By this I do not mean that the Bayesian method needs the sampling distribution (it doesn't), but I mean that it is not a bypass.

What the Bayesian method is 'bypassing' is a need to make calculations with the joint distribution of parameters and observations $f(\boldsymbol{\theta},\mathbf{x})$ for values other than the actual observation, since the method conditions on the observation. And maybe the question is indirectly about that (but it is not so clear). The sampling distribution is in fact a shortcut (and not something cumbersome that is to be bypassed). With a frequentist method you can just as well work with the likelihood function and for instance do maximum likelihood estimation or confidence intervals. But the sample distribution of an estimate/statistic is the best language to do this.

Frequentist/Bayesian is not a dichotomy

There is no clear border what frequentist and Bayesian statistics means. One can do empirical Bayesian analysis or use Jeffreys prior in which case one is loosening the conditioning on the observation. And one can make an analysis that is frequentist-like but is not using an estimate/statistic and it's sample distribution.

Many people are just fitting curves with models by using some linear or non-linear fitting package and use something like an estimate of the inverse of the Fisher information matrix to express the variance/error of the estimate and there is no direct computation of the sample distribution.

Or one can do something else like using AIC/BIC to express goodness of fit, or use a Bayes factor or fiduciary or likelihood intervals.

When a sample distribution is used, then it is not really a tool that is something that can be 'bypassed'. The sample distribution is the goal itselve. And if you like you could apply it to a Bayesian estimate (although it makes less sense in such a setting).

In frequentist statistics you don't have to compute a sampling distribution of an estimate.

In frequentist statistics, or whatever it is, you don't have to calculate these statistics and their sample distribution. You can also work only with the likelihood function in order to make point or interval estimates.

The method in the example of the question, with the sampling distribution of the mean is derived from maximum likelihood estimation and effectively equivalent. You do not need a sample distribution of a statistic or estimate (but it does make the analysis simpler) to compute it.

For instance to make a maximum likelihood estimate for a population mean $\mu$ of a normal distributed population we use the likelihood function:

$$\mathcal{L}(\mu \vert x_1,x_2,\dots,x_n ,\sigma) = \prod_{1\leq i \leq n} \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}\left(\frac{x_i-\mu}{\sigma}\right)^2} $$

and the $\mu$ that maximizes this function is the MLE estimate.

This is very similar to Bayesian maximum a posteriori estimate, which is just maximizing $$ f_{posterior}(\mu \vert x_1,x_2,\dots,x_n ) \propto \mathcal{L}(\mu \vert x_1,x_2,\dots,x_n ,\sigma) \cdot f_{prior}(\mu)$$

The only difference is that the likelihood function is multiplied with the prior probability.

Similarly for confidence intervals, one could use z or t-statistics, but effectively those statistics are shortcuts for the more difficult geometrical shape of the density distribution in all coordinates of the observation $\mathbf{x}$. We can derive p-values, statistical tests (and related confidence intervals) by only considering whether an observation is 'extreme' or not. And this can be defined by the likelihood function without considering a statistic/estimate and it's sample distribution (e.g. likelihood ratio test, if the likelihood is below a certain value than the value is not inside the confidence region). This view is also illustrated here where a test is not viewed by considering the sampling distribution of a statistic, but by considering the PDF of the whole data (in that case the data is two variables X and Y).

example of mechanism

The sample distribution occurs particularly in the method of moments. We can use the moments of a sample to estimate the moments of a distribution and in that case we may wish to express the sample distribution of the moments of a sample. But the method of moments is different from maximum likelihood estimation (but maybe this is already not frequentist?), and we do not use this sample distribution in every type of analysis.


**This question is not entirely clear about what is meant with 'sampling distribution' (an ambiguity which causes two diverging type of answers). For this answer I interpret sampling distribution as the distribution of a statistic or the distribution of an estimate. And I interpret a statistic in the sense of R.A. Fisher "statistic may be defined as a function of the observations designed as an estimate of the parameters". In this answer I argue that you do not need such sampling distributions (e.g. you do not need to work as you describe, compute sample mean and sample variance. Instead, you can use the likelihood/probability function directly. But the sampling distribution, and related sufficient statistics, does make it easier.). I do not interpret sample distribution more generally as the distribution of observations/samples.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – whuber May 23 at 16:43
  • $\begingroup$ Future comments should be posted in the dedicated chat room linked above. Any additional comments posted here will be deleted. $\endgroup$ – gung - Reinstate Monica May 25 at 11:45
  • 1
    $\begingroup$ If there are concerns with an answer, shouldn't these be visible in the comments? Sorry for writing another comment but I have no idea how else to communicate about this..., would send a private message if I knew how to do this. $\endgroup$ – Lewian May 25 at 13:24
  • 1
    $\begingroup$ Not deleted, so I voice my concerns again here. All discussion in the chat, also Sextus Empiricus has replied to concerns there. 1) The definition of a statistic and sampling distribution implied here is not in line with the formal definition given in pretty much all (at least mathematically oriented) textbooks. 2) In standard frequentist inference, the parameter $\theta$ is not a random variable, and therefore there is no joint distribution $f(\theta,x)$. (Frequentists can treat $\theta$ as RV occasionally, but that's very exceptional.) $\endgroup$ – Lewian May 26 at 8:49
  • $\begingroup$ I am very clear about the definitions in which light you should view my contribution. It has as a consequence that $$\text{a statistic} \neq \text{raw data or raw observations}$$ discussion about it is not about the content and ideas of the answer, but about semantics and definitions used in the answer. $\endgroup$ – Sextus Empiricus May 26 at 9:10
5
$\begingroup$

We need to be precise about the terms "frequentist" and "Bayesian", because they are ambiguous. "Frequentism" can be understood as adhering to a specific interpretation of the meaning of probability, which doesn't necessarily imply that any specific methodology needs to be applied. In this sense one can be a frequentist without ever computing confidence intervals, and as a frequentist one can do Bayesian statistics (particularly if the prior has a frequentist interpretation). However more people use "frequentist" as referring to what is known as standard frequentist approaches to inference, estimation, tests and confidence regions. These rely crucially on the sampling distribution.

"Bayesian" on the other hand is often meant refer to a particular interpretation of the meaning of probability, usually understood as "epistemic" probabilities, although this is not the only possible meaning "Bayesian" can have. A frequentist probability would be defined by a data generating process in reality, whereas an epistemic probability refers to a state of knowledge of an individual (or science as a whole) about something rather than the real process that generates this "something". The concept of a "sampling distribution" is understood by frequentists as referring to a distribution of a statistic given that data are distributed according to the underlying real process. As there is no such thing in epistemic probability as an underlying real process defining probabilities, they don't have a sampling distribution in this sense. They don't "bypass" it, it is a concept that isn't meaningful for them. However a Bayesian still can think of a real process as a sampling process for choosing and processing their epistemic probabilities, in which case something can occur in the Bayesian computations that looks and acts like a sampling distribution.

Note: Following a remark by Sextus Empiricus I add that when writing about "frequentists" and "Bayesians" I don't intend to imply that anybody has to be either a frequentist or a Bayesian as a person. What I do think is that whenever we do data analysis involving probabilities, we should be clear what we think these probabilities mean, and this can be frequentist, or epistemic (various versions), or other. This shouldn't stop us from adopting a different interpretation in a different situation if it seems fit. So where I write "as a frequentist", I mean "as somebody who locally, in a given situation, takes a frequentist hat", etc.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Could the person who voted this down please explain what's wrong with it? $\endgroup$ – Lewian May 24 at 19:44
  • 3
    $\begingroup$ I wan't the downvote for this; I upvoted it because I think it is a useful clarification of terminology in the present context. Nevertheless, I can see why someone might downvote it if they thought that what you're saying is tangential to the question that was actually asked. Don't sweat the downvotes - we all get them from time to time. I've gone and upvoted a few of your other good answers to make you feel better. ;) $\endgroup$ – Ben May 25 at 1:51
  • 1
    $\begingroup$ OK thanks. No need to upvote me to make me feel better. Fair enough explanation; I thought I have addressed the question pretty directly, but I may have misunderstood what the thread opener really wanted. $\endgroup$ – Lewian May 25 at 9:08
4
$\begingroup$

Broadly speaking, Bayesian analyses satisfy the so-called likelihood principle, which means that all the information about parameters $\theta$ from an experiment that observed $X^\star$ is contained in the likelihood $$ L(\theta) \equiv p(X^\star | \theta), $$ which is crucially only evaluated at the observed $X^\star$.

Contrast this with the sampling distribution, $p(X|\theta)$ as a distribution in $X$. Crucially the data isn’t fixed to the observed value, and we instead consider this as a distribution in $X$.

Take for example the posterior, $$ p(\theta|X^\star) \propto p(X^\star | \theta) \pi(\theta). $$ It doesn’t depend on $p(X|\theta)$ anywhere other than at $X=X^\star$. So we would find the same posterior distribution for any sampling distribution $f$ so long as $f(X^\star|\theta) =p(X^\star|\theta)$. The posterior depends on the likelihood function, but not the entire sampling distribution.

Whilst the fundamental rules of Bayesian inference satisfy the likelihood principle, a few ideas violate it. For example, a few formal rules for constructing priors, e.g., so called reference priors and Jeffreys priors, use the likelihood function evaluated at all possible experimental outcomes (i.e., they use the sampling distribution). A few hybrid ideas, like the posterior and prior $p$-value, also violate it. I suppose ABC methods require the sampling distribution, but only as a means to ultimately approximate the likelihood at the observed data.

So, with a few exceptions, yes Bayesian statistics bypasses the need for the sampling distribution.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ This seems for a large part semantic. Bayesian and frequentist methods use the same joint distribution function for parameters and observations $f(\theta,x)$. But the difference is that in practice with Bayesian methods you only need to evaluate that function for the given observations $x$, and for many frequentist methods you use calculations that consider for a given observation $x$ also the values of the distribution at other potential outcomes. This concept is not what I associate with 'sampling distribution'. $\endgroup$ – Sextus Empiricus May 23 at 13:14
  • $\begingroup$ Absolutely not semantic; this is intimately related to the likelihood principle which has profound consequences. $\endgroup$ – innisfree May 23 at 13:18
  • $\begingroup$ yeah, but did you read the definition of "sampling distribution"? we should agree on that first $\endgroup$ – carlo May 23 at 13:33
  • $\begingroup$ Yes, I read nothing on the wiki I disagreed with $\endgroup$ – innisfree May 23 at 13:36
1
+200
$\begingroup$

No, your understanding is not correct.

First, frequentist statistics does not allow us to "test to see what is the % chance the population mean falls within some range, using the sampling distribution". More precisely, frequentist statistics do not make probability statements on the population mean --- they only make probability statements on the estimates of the population mean. This is a well-known limitation of frequentist statisics that has caused much confusion and spawned many related questions on cross validated. (See, e.g., this thread.)

Secondly, in Bayesian stats, we do have the sampling distribution of the sample mean. We may not specifically refer to it, though. As others have mentioned though, Bayesian inference is a type of likelihood inference. Once you have defined your likelihood, you have, by deduction, the sampling distribution of the sample mean. Whether you use that distribution is another matter though.

In fact, there are variants of frequentist inference that bypass the likelihood, in that they only work with the moments of the sampling distribution and not the full distribution. See, e.g. the literature on method of moments. However, a "pure" Bayesian analysis will always involve the likelihood even if it is intractable, and therefore a sampling distribution is always implied.

Just for completeness, there are also variants of Bayesian inference that do not involve a full definition of the likelihood, and hence I specifically refered to "pure" Bayesian inference earlier.

An example for illustration \begin{align} X_i &\overset{iid}{\sim} N(\mu, 1) \tag{1} \\ \mu &\sim N(0, \sigma^2) \tag{2} \end{align} Here, equation (1) implies \begin{equation} \bar{X} = \sum_i^n X_i \sim N(\mu, 1/n) \tag{3} \end{equation} which is the sampling distribution of $\bar{X}$. Of course, in Bayesian inference, we typically don't care about (3), since our interest will usually be in \begin{equation} p(\mu|X) = \frac{p(X|\mu)p(\mu)}{p(X)} \end{equation} However, it happens in this case, since $\bar{X}$ is a sufficient statistic, that \begin{equation} p(\mu|X) = p(\mu|\bar{X}) = \frac{p(\bar{X}|\mu)p(\mu)}{p(\bar{X})} \end{equation} Thus, you can also use the sampling distribution (3) to derive your posterior distribution, if you like.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ they only make probability statements on the estimates of the population mean. But when these estimates are an interval estimate instead of a point estimate then the interpretation is not so wrong. The statement "test to see what is the % chance the population mean falls within some range" could be seen as a probability statement about the 'range', for instance when that range is a confidence interval. $\endgroup$ – Sextus Empiricus May 25 at 7:31
  • 1
    $\begingroup$ Point taken but it does not appear to me that the OP has this interpretation in mind. $\endgroup$ – Tim Mak May 25 at 7:55
  • $\begingroup$ I agree that the way how it is phrased does indeed indicate your first interpretation. But I find it interesting, I have actually never before thought of the symmetry of these interpretations (one wrong, one right). $\endgroup$ – Sextus Empiricus May 25 at 8:19
  • 1
    $\begingroup$ Your argument is interesting. You seem to suggest that a Bayesian defines "likelihood" differently than a frequentist, as a Bayesian does not view it as a distribution on $X$ and only as a function on $\theta$. Is my understanding correct? $\endgroup$ – Tim Mak May 26 at 1:55
  • 1
    $\begingroup$ I personally have not come across situations where the researcher only defines $p(X|\theta)$ for $X=X^{\star}$ and not also for $X \neq X^{\star}$. Is this used in a particular method? In any case, I don't think the textbook definition of a Bayesian analysis views the likelihood in this way. $\endgroup$ – Tim Mak May 26 at 1:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.