One of the main differences between Bayesians and frequentists is that they have a subjective interpretation to probability.

However, do Bayesians actually interpret subjectively the probabilities attached to an outcome GIVEN a set of parameters (i.e. for the likelihood), or is it just that they attach a subjective probability to the prior, and also to the posterior as a consequence? (but $p(x | \theta)$ is thought similarly to the way frequentists think about it.)

  • $\begingroup$ Bayesians don't necessarily view their interpretation of probability as subjective; there is also the school of objective Bayesian statistics. The key difference between frequentists and Bayesians is that frequentists interpret probabilities as long run frequencies. $\endgroup$ – Dikran Marsupial Nov 2 '13 at 9:49
  • $\begingroup$ I am talking about the subjective school, do they view the probabilities given by the likelihood as subjective as well? $\endgroup$ – mixtureModel Nov 2 '13 at 12:01
  • 4
    $\begingroup$ Indeed, there is nothing to stop Bayesians from interpreting a probability as a long run frequency, it is a perfectly good way of expressing (some forms of) a degree of plausibility. $\endgroup$ – Dikran Marsupial Nov 4 '13 at 7:46
  • 1
    $\begingroup$ Well, in a Bayesian framework probability is, per definition, "subjective". I put it in quotes because subjective here stand for personal or individual, rather than subjective as in biased or overly influenced by self interest. See: johndcook.com/blog/2008/02/26/what-a-probability-means $\endgroup$ – Rasmus Bååth Nov 12 '13 at 15:41

Roughly speaking, the Bayesian and the frequentist perspectives agree on calculation of the likelihood, but disagree on the interpretation. That's because the Bayesian interpretation of probability is, very roughly speaking, a superset of the frequentist one. From the frequentist perspective, the probability only makes sense if the likelihood is describing the probability of an event that will be repeated for a sufficiently large number of independent, identically distributed trials. From the Bayesian perspective, it's the quantified degree of belief. If there are to be a sufficiently large number of i.i.d. trials, then the Bayesians' belief, if rational and informed, should match the frequentist's probability.

The Bayesian might also be a little more explicit about the background assumptions. We're not really calculating $p(x|\theta)$, because that's not even a thing. We're really calculating $p(x|\theta,\Omega)$, where $\Omega$ represents all of the background assumptions.

That looks like a trivial change, because we hold $\Omega$ to be constant. I believe it's non-trivial It's subtle, and it's more about cognition than maths, but hear me out:

Because the Bayesian is routinely forced to make explicit, and quantify, the underlying assumptions, this becomes a way of thinking, or culture, or intuition, or whatever you'd like to call it. And this way of thinking is very valuable in decomposing $\Omega$, when the combination of observations and calculation leads to contradictions and extremely low posterior probabilities.

(at the risk of churning over old ground) This does highlight something that is at once a fundamental difference between the Bayesian and frequentist perspectives, and a commonality. As Sivia shows so elegantly in Data Analysis: A Bayesian Tutorial, the standard frequentist tools are exactly the analytic solutions you get from a Bayesian perspective with specific uninformative priors. Same tools, same answers, if the underlying assumptions are identical. The difference is that the Bayesian has the option of informative priors, and is compelled by the method to explicitly quantify their prior.


Interpretation of probability should be consistent across all probability statements: Regardless of the interpretation of probability that one is using in Bayesian analysis, that interpretation ought to be consistent across different kinds of probability statements. In terms of the fundamental meaning of what probability actually is, there is no reason to interpret sampling probabilities (relating to observable values conditional on unobserved parameters) differently to prior probabilities (relating to unobserved parameters). Thus, if a Bayesian gives an epistemic interpretation to probability, and treats the prior as an encapsulation of his beliefs about uncertainty in the unobservable parameters, then consistency requires that he should also give the postulated sampling distribution (and thus the likelihood function) an epistemic interpretation, reflecting his beliefs about the uncertainty in the sampling process.

"Subjectivism" is where it gets tricky: When you start talking about the "subjective" interpretation things get a little muddier, since that term can refer to two different things, broadly classified as "weak subjectivism" and "strong subjectivism".

Weak subjectivism: In this interpretation, the term "subjective" is used in its weaker sense, meaning merely that probability encapsulates the beliefs of a subject. (Some people prefer to use the term "epistemic" for this concept, since it does not actually require subjectivity in the stronger sense.) In this case both the likelihood and prior refer to beliefs of the subject, so both the likelihood and prior are "subjective" in this weak sense. Consistency requires that either both are weakly subjective, or neither are weakly subjective.

Strong subjectivism: In this interpretation, the term "subjective" is used in its stronger sense, meaning that weak subjectivism holds, and furthermore, the subject's belief lacks any outside "objective" justification (i.e., two or more different subjects could all hold different beliefs, and none would be more wrong than the others). In this case the "subjectivity" of the prior refers not only to the fact that it encapsulates the subject's belief, but also to the fact that it is not grounded in an objective justification, and that it may legitimately differ between subjects. It is possible for the prior to be "subjective" in this stronger sense, and the likelihood "objective", insofar as it is a belief based on an objective justification.

Example of the difference: Suppose you have a series of exchangeable binary values $X_1,X_2,X_3,...$ where belief in exchangeability is based on some objective reasoning referring to outside conditions. In this case, the belief in exchangeability logically leads (via de Finetti's representation theorem) to belief in the sampling distribution:

$$X_1,X_2,X_3,... \sim \text{IID Bern}(\theta) \quad \quad \quad \theta \equiv \lim_{n \rightarrow \infty} \frac{1}{n} \sum_{i=1}^n X_i.$$

Now, it might be the case that there is no objective reasoning that favours any particular prior for $\theta$ (e.g., you eschew appeal to "non-informative" priors), and so you simply decide to choose a prior that reflects your own personal beliefs about $\theta$. In this case both the likelihood and prior would be "subjective" in the weaker sense (more accurately described as "epistemic"), but only the prior would be "subjective" in the stronger sense.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.