I read the following paragraph on Wikipedia recently:

Bayesian intervals treat their bounds as fixed and the estimated parameter as a random variable, whereas frequentist confidence intervals treat their bounds as random variables and the parameter as a fixed value.

However, I am not sure whether this is true. My interpretation of the credible interval was that it encapsulated our own uncertainty about the true value of the estimated parameter but that the estimated parameter itself did have some kind of 'true' value.

This is slightly different to saying that the estimated parameter is a 'random variable'. Am I wrong?

  • $\begingroup$ I would not defend every word choice, but the Wikipedia quote is essentially correct. Bayesian inference begins with a prior probability distribution on the parameter, taken to be a random variable. $\endgroup$ – BruceET Aug 24 '19 at 19:52
  • $\begingroup$ The sentence is confusing. In a Bayesian perspective, the parameter $\theta$ is treated as random, while the estimator of the parameter $\hat\theta(x)$ is not. What is the estimated parameter? $\endgroup$ – Xi'an Aug 24 '19 at 20:43
  • $\begingroup$ I agree that it is confusing. To take an example consider the simple beta-binomial model. My question is: how do we interpret the posterior beta distribution of the parameter 'p'? Are we saying that it reflects the fact that 'p' itself is literally a random variable or does it reflect our own uncertainty about what 'p' could be? $\endgroup$ – Johnny Breen Aug 24 '19 at 20:52
  • $\begingroup$ The question about how to interpret such kind of parameters and their probability distributions has two standard answers in Bayesian theory: 1. You use de Finetti's theorem to say that such parameter and its distribution are a short-hand way to speak about a joint predictive distribution; or 2. You interpret this parameter as a long-run frequency: you're effectively asking: "what's my degree of belief that the long-run frequency of the successes is within an interval $\mathrm{d} p$?". Both interpretations are justified by de Finetti's theorem. $\endgroup$ – pglpm Aug 25 '19 at 4:16
  • 1
    $\begingroup$ In the Bayesian paradigm, a random variable is a variable whose value we are uncertain about (and vice versa). $\endgroup$ – Ilmari Karonen Aug 25 '19 at 20:15

Consider the situation in which you have $n = 20$ observations of a binary (2-coutcome) process. Often the two possible outcomes on each trial are called Success and Failure.

Frequentist confidence interval. Suppose you observe $x = 15$ successes in the $n = 20$ trials. View the number $X$ of Successes as a random variable $X \sim \mathsf{Binom}(n=20; p),$ where the success probability $p$ is an unknown constant. The Wald 95% frequentist confidence interval is based on $\hat p = 15/20 = 0.75,$ an estimate of $p.$ Using a normal approximation, this CI is of the form $\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}$ or $(0.560, 0.940).$ [The somewhat improved Agresti-Coull style of 95% CI is $(0.526, 0.890).]$

A common interpretation is that the procedure that produces such an interval will produce lower and upper confidence limits that include the true value of $p$ in 95% of instances over the long run. [The advantage of the Agresti-Coull interval is that the long run proportion of such inclusions is nearer to 95% than for the Wald interval.]

Bayesian credible interval. The Bayesian approach begins by treating $p$ as a random variable. Prior to seeing data, if we have no experience with the kind binomial experiment being conducted or no personal opinion as to the distribution of $p,$ we may choose the 'flat' or 'noninformative' uniform distribution, saying $p \sim \mathsf{Unif}(0, 1) \equiv \mathsf{Beta}(1,1).$

Then, given 15 successes in 20 binomial trials, we find the posterior distribution of $p$ as the product of the prior distribution and the binomial likelihood function.

$$f(p|x) \propto p^{1-1}(1-p)^{1-1} \times p^{15}(1-p)^{5} \propto p^{16-1}(1-p)^{6-1},$$ where the symbol $\propto$ (read 'proportional to') indicates that we are omitting 'norming' constant factors of the distributions, which do not contain $p.$ Without the norming factor, a density function or PMF is called the 'kernel' of the distribution.

Here we recognize that the kernel of the posterior distribution is that of the distribution $\mathsf{Beta}(16, 6).$ Then a 95% Bayesian posterior interval or credible interval is found by cutting 2.5% from each tail of the posterior distribution. Here is the result from R: $(0.528,0.887).$ [For information about beta distributions, see Wikipedia.]

qbeta(c(.025,.975), 16, 6)
[1] 0.5283402 0.8871906

If we believed the prior to be reasonable and believe that the 20-trial binomial experiment was fairly conducted, then logically we must expect the Bayesian interval estimate to give useful information about the experiment at hand---with no reference to a hypothetical long-run future.

Notice that this Bayesian credible interval is numerically similar to the Agresti-Coull confidence interval. However, as you point out, the interpretations of the two types of interval estimates (frequentist and Bayesian) are not the same.

Informative prior. Before we saw the data, if we had reason to believe that $p \approx 2/3,$ then we might have chosen the distribution $\mathsf{Beta}(8,4)$ as the prior distribution. [This distribution has mean 2/3, standard deviation about 0.35, and puts about 95% of its probability in the interval $(0.39, 0.89).$]

qbeta(c(.025,.975), 8,4)
[1] 0.3902574 0.8907366

In that case, multiplying the prior by the likelihood gives the posterior kernel of $\mathsf{Beta}(23,7),$ so that the 95% Bayesian credible interval is $(0.603, 0.897).$ The posterior distribution is a melding of the information in the prior and the likelihood, which are in rough agreement, so the resulting Bayesian interval estimate is shorter than than the interval from the flat prior.

qbeta(c(.025,.975), 23,7)
[1] 0.6027531 0.8970164

Notes: (1) The beta prior and binomial likelihood function are 'conjugate`, that is, mathematically compatible in a way that allows us to find the posterior distribution without computation. Sometimes, there does not seem to be a prior distribution that is conjugate with the likelihood. Then it may be necessary to use numerical integration to find the posterior distribution.

(2) A Bayesian credible interval from an noninformative prior essentially depends on the likelihood function. Also, much of frequentist inference depends of the likelihood function. Thus is is not a surprise that a Bayesian credible interval from a flat prior may be numerically similar to a frequentist confidence interval based on the same likelihood.

| cite | improve this answer | |

Your interpretation is correct. In my opinion that particular passage in the Wikipedia article obfuscates a simple concept with opaque technical language. The initial passage is much clearer: "is an interval within which an unobserved parameter value falls with a particular subjective probability".

The technical term "random variable" is misleading, especially from a Bayesian point of view. It's still used just out of tradition; take a look at Shafer's intriguing historical study When to call a variable random about its origins. From a Bayesian point of view, "random" simply means "unknown" or "uncertain" (for whatever reason), and "variable" is a misnomer for "quantity" or "value". For example, when we try to assess our uncertainty about the speed of light $c$ from a measurement or experiment, we speak of $c$ as a "random variable"; but it's obviously not "random" (and what does "random" mean?), nor is it "variable" – in fact, it's a constant. It's just a physical constant whose exact value we're uncertain about. See § 16.4 (and other places) in Jaynes's book for an illuminating discussion of this topic.

The question "what does a Bayesian interval for a 'parameter' mean?" comes from the even more important question "what does this parameter mean?". There are two main points of view – not mutually exclusive – about the meaning of "parameters" in Bayesian theory. Both use de Finetti's theorem. Chapter 4 of Bernardo & Smith's Bayesian Theory has a beautifully deep discussion of the theorem; see also Dawid's summary Exchangeability and its ramifications.

The first point of view is that the parameter and its distribution are just mathematical objects that completely summarize an infinite set of joint belief distributions about the actually observable quantities $x_1, x_2, \dotsc$ (say, the outcomes of the tosses of a coin, or the presences of a genetic allele in individuals having a particular disease). So, in the binomial case, when we say "we have a 95% belief that the parameter value $p$ is within interval $I$", we mean "we have a belief between $b_1$% and $b_1'$% that $x_1=1$", "we have a belief between $b_2$% and $b_2'$% that $x_1=1$ and $x_2=1$", and all possible similar statements. The exact numerical relation between the $b_i$s and the interval $I$ is given by de Finetti's integral formula.

The second point of view is that such "parameters" are long-run observable quantities, so it does make sense to speak about our belief in their values. For example, the binomial parameter $p$ is the long-run frequency of observations of "successes" (tails for a coin, minor allele for the genetic case, and so on). So when we say "we have a 95% belief that the parameter value $p$ is within interval $I$" we mean "we have a 95% belief that the long-run relative frequency of successes is within interval $I$". The context here is that, if an oracle or jinn told us that the long-run relative frequency were, say, 0.643, then our belief that the next observation is a success would be, from symmetry reasons, 64.3%; for the next two observations, 41.3449%, and so on. ("From symmetry reasons" because we believe equally in all possible time sequences of successes and failures – this is the context of the theorem.) These long-run observations need not be infinite, but just large enough: in this case de Finetti's infinite-exchangeability formula can be considered as an approximation of a finite-exchangeability one (for example, the binomial distribution is an approximation for a hypergeometric one: "drawing without replacement"); see Diaconis & Freedman about such approximation. Often such parameters are related to long-run statistics (see again the cited chapter in Bernardo & Smith). In short, the "parameter" is a long-run frequency or other observable, empirical statistics.

I personally like the second point of view – which tries to find the empirical meaning of the parameter as a physical quantity – also because it helps me to assess my pre-data belief distribution about that specific physical quantity, in its specific context. See for example Diaconis & al's paper Dynamical bias in the coin toss for a beautiful study of the relation between long-run parameters and physical principles. Today, unfortunately, many "models" and parameters come just as black boxes: people use them just because other people use them. In Diaconis's words:

de Finetti's alarm at statisticians introducing reams of unobservable parameters has been repeatedly justified in the modern curve fitting exercises of today's big models. These seem to lose all contact with scientific reality focusing attention on details of large programs and fitting instead of observation and understanding of basic mechanism.

In frequentist theory the term "random variable" may have a different meaning though. I'm not an expert in this theory, so I won't try to define it there. I think there's some literature around that shows that frequentist confidence intervals and Bayesian intervals can be quite different; see for example Confidence intervals vs Bayesian intervals or https://www.ncbi.nlm.nih.gov/pubmed/6830080.

| cite | improve this answer | |
  • 1
    $\begingroup$ (+1) Jaynes has a lot to say that is important, but I think that the linked paper Confidence Intervals vs Bayesian Intervals is largely a polemic, and it may have been more relevant in the past when Bayesian methods were less accepted. $\endgroup$ – Michael Lew Aug 25 '19 at 0:29
  • $\begingroup$ @MichaelLew True; Jaynes himself writes in the intro to the reprint of that work: "We come now to the most polemical of all my articles. There are several reasons for this heated style...". But the examples he analyses in between polemics are interesting (for example, §III.(b), or III.(d) which I find very instructive), if one has some or a lot of patience to go through them... $\endgroup$ – pglpm Aug 25 '19 at 7:39

My interpretation of the credible interval was that it encapsulated our own uncertainty about the true value of the estimated parameter but that the estimated parameter itself did have some kind of 'true' value. This is slightly different to saying that the estimated parameter is a 'random variable'. Am I wrong?

Although you say that you interpret the credible interval as encapsulating our own uncertainty, the logic of your conclusion proceeds from the premise that a quantity with a true value is not a random variable. This is taking an aleatory view of probability (and subsequent "randomness") which conceives of randomness as a property that inheres in nature. Mathematically, a random variable is merely a quantity that corresponds to possible outcomes in a sample space, with a probability measure attached. Thus, your approach would only make sense if you take that probability measure as an inherent property of nature, giving the propensity for a metaphysically "random" event to occur. You then conclude that a parameter that has a true value must not be metaphysically "random" and therefore cannot be described by a (non-degenerate) probability measure.

That approach is at odds with the epistemic interpretation of probability that is generally used in Bayesian theory. Under the latter approach (which is the standard interpretation), the probability measure is interpreted only as a measure of degree-of-belief (under certain coherence requirements) of the analyst (or some other subject). Under the epistemic interpretation, "random variable" is synonymous with "unknown quantity", and thus, there is no problem with saying that a parameter has a true value, but is still a random variable with a (non-degenerate) probability measure. The quote you are looking at is using this epistemic approach to probability, but your conclusion appears to be using a premise that is at odds with this interpretation.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.