6
$\begingroup$

The textbook definition of a statistic is any function of the data, $g(\mathbf{X})$. Much of frequentist inference is concerned with deriving sampling distributions for various statistics under some null model, so it makes sense that the term "statistic" has a precise definition and is used quite often.

The goal of Bayesian inference, on the other hand, is to derive the posterior distribution $P(\theta | \mathbf{X})$ of some parameter $\theta$ upon observing data $\mathbf{X}$. The discussions I've seen on Bayesian inference tend not to give a definition for the term "statistic."

I have two questions:

  1. Do Bayesians use the term "statistic" in the same way that frequentists do, or does Bayesian inference not depend as heavily on the concept of a statistic?
  2. Is it accurate to say that the posterior distribution $P(\theta|\mathbf{X})$ is itself a statistic? Even if it is technically correct, is it "morally" correct?
$\endgroup$
  • $\begingroup$ Reading "Deep Learning" by Goodfellow, Bengio, and Courville perhaps? ;) $\endgroup$ – DarkCygnus Dec 5 '17 at 23:04
  • 1
    $\begingroup$ Statistics are not ordinarily thought of as families of functions that depend on parameters. The posterior distribution does, though: it depends on the prior distribution. $\endgroup$ – whuber Dec 5 '17 at 23:17
  • $\begingroup$ @DarkCygnus no, but I would like to at some point. $\endgroup$ – tddevlin Dec 6 '17 at 1:37
6
$\begingroup$

No, posterior distribution is not a statistic if you define it strictly and agree that there is a distinction between statistic and estimator. First of all, it is an estimator of the posterior distribution (see What is the difference between an estimator and a statistic?). Second, as already noticed in the comment by @whuber, a statistic is a function of the data, while posterior distribution is a function of the data and the priors, so it depends not only on the data.

Posterior distribution is an estimate of the distribution of the parameter of the assumed model; statistic would be a function of the sample that characterizes it. The problem is not with the fact that it is a distribution (e.g. one could argue that kernel density estimator, as well as histogram, are statistics), but with your interest in the model, rather then the sample.

On another hand, the definition of statistic is very broad and many handbooks give only brief definitions ("function of the sample", or "numerical characteristic of a sample"), or like Leehman and Casella in Theory of Point Estimation say that "estimators are statistics" (p. 16). So when sticking to the definition that statistic is any function of the sample, the answer would be "yes".

  1. Do Bayesians use the term "statistic" in the same way that frequentists do, or does Bayesian inference not depend as heavily on the concept of a statistic?

In general, Bayesians use all of the statistical terms the same way as other statisticians. They only differ in how they interpret the probability, while sticking to the same definitions (Kolomogorov axioms etc.). If they didn't, they wouldn't be able to communicate with the rest of statistical community and would not be considered as a part of statistics.

$\endgroup$
  • 1
    $\begingroup$ I know your answer has already been marked as correct, but I would appreciate a definition of a statistic since you appear to have a definition in mind beyond "a function of the data." Is the "sample average plus 2" a statistic? It seems you would say no since it "goes beyond the data". $\endgroup$ – jaradniemi Dec 7 '17 at 2:58
  • $\begingroup$ @jaradniemi but this was discussed in the linked thread so there is probably no point in repeating the answers. $\endgroup$ – Tim Dec 7 '17 at 6:08
  • 1
    $\begingroup$ Except that question does not have an answer marked correct. Your answer to that question provides a rigorous definition of a statistic from Casella and Berger which indicates that a statistic needs to be a "real-valued or vector-valued function". Is that the strict definition you allude to here? Since the posterior is not vector-valued, it cannot be a statistic, but any countable collection of quantiles would be a statistic? $\endgroup$ – jaradniemi Dec 8 '17 at 15:26
  • 1
    $\begingroup$ @jaradniemi the question has two highly upvoted answers that I refer to. Lack of the "accepted" answer has nothing to do with answers being "correct" or not. As I said in my answer, while most of the authors give vague definitions of statistic, the actual usage of the term suggests that statistic is something different from estimator in the fact that statistic is used to describe the properties of the sample, while estimators estimate the quantities related to the model. What follows, statistic should depend on the data and only on the data if it is used to characterize the data. $\endgroup$ – Tim Dec 8 '17 at 15:36
  • $\begingroup$ ... Moreover, in my answer I explicitly say that the definitions are vague and other interpretations are possible. $\endgroup$ – Tim Dec 8 '17 at 15:38
5
$\begingroup$

Yes, a posterior is a statistic. As you point out a statistic is any function of the data and the posterior is a function of the data and thus a statistic. Yes, Bayesians use the notion of statistic but typically not describing the posterior as a statistic. We are certainly interested in sufficient statistics since a posterior, i.e. a distribution for a parameter given the data, is equivalent to the conditional distribution of the parameter given a sufficient statistic.

Yes it is technically correct and it is moral. You could even look at the sampling distribution of the posterior to understand properties of the posterior over different realizations of the data. Efron does this in this manuscript.

$\endgroup$
  • $\begingroup$ In connection with this issue, Don Fraser has argued in a 1962 paper that the likelihood function is a sufficient statistic. $\endgroup$ – Xi'an May 4 '18 at 8:55
2
$\begingroup$

I don't agree with the view that the posterior is not a statistic because it is an estimator (e.g., in the accepted answer to this question, and the accompanying answer to the related question on alleged differences between a statistic and an estimator). But for other reasons, set out in detail below, I would say that it is possible to form a "statistic" with this quantity, but the thing that is a statistic is not really the "posterior distribution".

To understand this, it is important to remember that a multivariate function can be viewed in many different ways, depending on which variables we are treating as arguments and which we are treating as fixed values. In particular, a multivariate function can be a distribution with respect to one argument variable, but not a distribution with respect to another argument variable. In the present case, the function $P(\theta | \boldsymbol{x})$ can be viewed in three different ways$^\dagger$:

  • As a function of both $\theta$ and $\boldsymbol{x}$, in which case it is a conditional distribution;

  • As a function of $\theta$ only (with $\boldsymbol{x}$ treated as fixed), in which case it is a single distribution; or

  • As a function of $\boldsymbol{x}$ only (with $\theta$ treated as fixed), in which case it is a statistic, but not a distribution.

From these three views, we see that it is possible to view this object as a "statistic" (holding one argument constant) or as the "posterior distribution" (holding the other argument constant), but it is not strictly correct to view it as both of these things at once. Hence, it is very dubious to claim that the "posterior distribution" (which is a distribution) is also a "statistic". A more detailed answer is given below, where I explore this issue as a mapping with a domain and codomain including distribution functions.


What is a "statistic": The standard definition of a statistic refers to a function of the data vector - i.e., a function whose domain is the support of the data vector. The mere fact that a statistic can function as an estimator of some model parameter does not preclude it from being a statistic. While it is arguable that an estimator is something more than a statistic (e.g., a statistic plus a specified parameter it is used to estimate), this does not imply that a statistic is not a statistic merely because it can be used as an estimator of something.

Now, some definitions of a "statistic" (in various textbooks) restrict the concept only to mappings that output a real number of real-vector (as opposed to a distribution or function), but this is usually a contextual definition, made to deal with standard real scalar or vector statistics in other contexts (e.g., when discussing the theory of statistical inference). In my view, there is no reason in principle that a function cannot be considered a "statistic" if it maps a data vector to some other output such as a distribution. Hence, if $\mathscr{X}$ is the support of our observable data vector in some inference problem, I would regard any mapping $f: \mathscr{X} \rightarrow \text{Codomain}$ to be a "statistic" on the specified codomain. With this broad definition of a "statistic" let us now consider the question at issue.


Is the posterior distribution a "statistic"? It is certainly true that the posterior is determined both by the data vector and the prior distribution. Letting $\mathscr{X}$ be the support of the data vector and letting $\Pi$ be the space of allowable distributions for the parameter $\theta$, we can consider Bayes' rule to be a mapping $P: \mathscr{X} \times \Pi \rightarrow \Pi$ which maps a data vector and prior to the posterior distribution (with the latter considered as a function of $\theta$). So here we have a mapping where the domain is not just the support of the data vector.

However, as with any function of two arguments, we can also treat one of these (the prior) as a fixed value and consider the mapping as a function only of the other argument (the data vector). That is, for any fixed prior distribution $\pi \in \Pi$ we can consider the corresponding mapping $P_\pi: \mathscr{X} \rightarrow \Pi$ which maps a data vector to a posterior distribution (the one that results from our fixed prior). So now we have a mapping where the domain is the support of the data vector, and hence, we have a "statistic" (in the broad sense defined above).

Although $P_\pi$ is a "statistic" in the sense specified above, it is a bit of a stretch to call it the "posterior distribution". Effectively, it is the mapping you get if you treat the posterior distribution as a function only of its conditioning variable $\boldsymbol{x}$ for a fixed prior. It is not really accurate to call this a "distribution" since it is not a distribution with respect to the argument variable $\boldsymbol{x}$.

I would argue that the mapping $P_\pi$ is a "statistic", insofar as it is a function whose domain is the support of the data vector. It is not really accurate to call this function the "posterior distribution"; rather, it is the mapping that maps any allowable data vector to the corresponding posterior distribution. The "posterior distribution" is the element $P_\pi(\boldsymbol{x})$ in the codomain, not the mapping itself.


$^\dagger$ The posterior is also implicitly dependent on an unstated prior distribution $\pi(\theta)$ and so we could also take an enlarged view of things, by looking at the multivariate function $P(\theta | \boldsymbol{x}, \pi)$, in which case there are even more interpretations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.