Usually probability theory is taught with Kolgomorov's axioms. Do Bayesians also accept Kolmogorov's axioms?

  • 11
    $\begingroup$ Bayesian theory follows from the standard axioms of probability, hence from Kolmogorov axioms. $\endgroup$ – Xi'an Nov 30 '14 at 21:21
  • 3
    $\begingroup$ @Xi'an: That subjective degrees of belief can be represented by probability isn't so obvious - hence the question, & Ramsey & de Finetti's work. $\endgroup$ – Scortchi - Reinstate Monica Nov 30 '14 at 21:49
  • 2
    $\begingroup$ That's why I am an "objective" Bayesian and start with prior distributions defined according to the standards of probability theory... $\endgroup$ – Xi'an Nov 30 '14 at 21:53
  • 3
    $\begingroup$ I believe that Cox-Jaynes interpretation of probability provides a rigorous foundation for Bayesian probability. (see my answer). However, it would be nice to have Xi'an's opinion on that. $\endgroup$ – Summit Dec 1 '14 at 11:27
  • 2
    $\begingroup$ Here is a very recent arXiv paper addressing the issue. $\endgroup$ – Xi'an Dec 3 '14 at 9:17

In my opinion, Cox-Jaynes interpretation of probability provides a rigorous foundation for Bayesian probability:

  • Cox, Richard T. "Probability, frequency and reasonable expectation." American Journal of Physics 14.1 (1946): 1–13.
  • Jaynes, Edwin T. Probability theory: the logic of science. Cambridge University Press, 2003.
  • Beck, James L. "Bayesian system identification based on probability logic." Structural Control and Health Monitoring 17.7 (2010): 825–847.

The axioms of probability logic derived by Cox are:

  1. (P1): $\Pr[b|a]\ge0$ (by convention)
  2. (P2): $\Pr[\overline{b}|a]=1-\Pr[b|a]$ (negation function)
  3. (P3): $\Pr[b\cap c|a]=\Pr[c|b\cap a]\Pr[b|a]$ (conjunction function)

Axioms P1-P3 imply the following [Beck, 2010]:

  1. (P4): a) $\Pr[b|b\cap c] = 1$; b) $\Pr[\overline{b}|b\cap c] = 0$; c) $\Pr[b|c]\in[0,1]$
  2. (P5): a) $\Pr[a|c \cap (a \Rightarrow b)]\le\Pr[b|c\cap(a \Rightarrow b)]$, b) $\Pr[a|c\cap(a \Leftrightarrow b)] = \Pr[b|c\cap(a \Leftrightarrow b)]$, where $a \Rightarrow b$ means that $a$ is contained in $c$, and $a \Leftrightarrow b$ means that $a$ is equivalent to $b$.
  3. (P6): $\Pr[a \cup b|c] = \Pr[a|c]+\Pr[b|c]-\Pr[a\cap b|c]$
  4. (P7): Assuming that proposition $c$ states that one and only one of propositions $b_1,\ldots,b_N$ is true, then:
  • a) Marginalization Theorem: $\Pr[a|c]=\sum_{n=1}^N P[a \cap b_n|c]$
  • b) Total Probability Theorem: $\Pr[a|c] = \sum_{n=1}^N \Pr[a|b_n\cap c]\Pr[b_n|c]$
  • c) Bayes' Theorem: For $k=1,\ldots,N$: $\Pr[b_k|a\cap c] = \frac{\Pr[a|b_k\cap c]\Pr[b_k|c]}{\sum_{n=1}^N \Pr[a|b_n\cap c]\Pr[b_n|c]}$

They imply Kolmogorov's statement of logic, which can be viewed as a special case.

In my interpretation of a Bayesian viewpoint, everything is always (implicitly) conditioned on our believes and on our knowledge.

The following comparison is taken from Beck [2010]:

The Bayesian point of view

Probability is a measure of plausibility of a statement based on specified information.

  1. Probability distributions represent states of plausible knowledge about systems and phenomena, not inherent properties of them.
  2. Probability of a model is a measure of its plausibility relative to other models in a set.
  3. Pragmatically quantifies uncertainty due to missing information without any claim that this is due to nature's inherent randomness.

The Frequentist point of view

Probability is the relative frequency of occurrence of an inherently random event in the long run.

  1. Probability distributions are inherent properties of random phenomena.
  2. Limited scope, e.g., no meaning for the probability of a model.
  3. Inherent randomness is assumed, but cannot be proven.

How to derive Kolmogorov's axioms from the axioms above

In the following, section 2.2 of [Beck, 2010] is summarized:

In the following we use: probability measure $\Pr(A)$ on subset $A$ of a finite set $X$:

  1. [K1] $\Pr(A)\ge 0, \forall A \subset X$
  2. [K2]: $\Pr(X) = 1$
  3. [K3]: $\Pr(A\cup B)=\Pr(A)+\Pr(B), \forall A,B \subset X$ if $A$ and $B$ are disjoint.

In order to derive (K1–K3) from the axioms of probability theory, [Beck, 2010] introduced propositon $\pi$ that states $x\in X$ and specifies the probability model for $x$. [Beck, 2010] furthermore introduces $\Pr(A) = \Pr[x\in A|\pi]$.

  • P1 implies K1 with $b=\{x\in A\}$ and $c=\pi$
  • K2 follows from $\Pr[x\in X|\pi]=1$; P4(a), and $\pi$ states that $x\in X$.
  • K3 can be derived from P6: $A$ and $B$ are disjoint means that $x\in A$ and $x\in B$ are mutually exclusive. Therefore, K3: $\Pr(x\in A\cup B|\pi)=\Pr(x\in A|\pi)+\Pr(x\in B|\pi)$
  • 7
    $\begingroup$ From your K3 you can get to $\Pr(\cup_{i=1}^n A_i)=\sum_{i=1}^n\Pr(A_i)$ (finite additivity) but not to Kolmogorov's 3rd axiom, $\Pr(\cup_{i=1}^\infty A_i)=\sum_{i=1}^\infty\Pr(A_i)$ (countable additivity) when the $A$'s are elements of a $\sigma$-field, & not simply subsets of a finite set. $\endgroup$ – Scortchi - Reinstate Monica Dec 2 '14 at 10:04
  • 3
    $\begingroup$ @Scortchi K.R.Koch in his introduction to Bayesian Statistics cites Bernardo and Smith (1994), Bayesian Theory, p. 105, as a source that shows how to address countable infinity. I have not checked it, but as a reference it may as well be given here. $\endgroup$ – gwr Aug 23 '16 at 22:22

After the development of Probability Theory it was necessary to show that looser concepts answering to the name of "probability" measured up to the rigorously defined concept they had inspired. "Subjective" Bayesian probabilities were considered by Ramsey and de Finetti, who independently showed that a quantification of degree of belief subject to the constraints of comparability & coherence (your beliefs are coherent if no-one can make a Dutch book against you) has to be a probability.

Differences between axiomatizations are largely a matter of taste concerning what should be defined & what derived. But countable additivity is one of Kolmogorov's that isn't derivable from Cox's or Finetti's, & has been controversial. Some Bayesians (e.g. de Finetti & Savage) stop at finite additivity & so don't accept all of Kolmogorov's axioms. They can put uniform probability distributions over infinite intervals without impropriety. Others follow Villegas in also assuming monotone continuity, & get countable additivity from that.

Ramsey (1926), "Truth and probability", in Ramsey (1931), The Foundations of Mathematics and other Logical Essays

de Finetti (1931), "Sul significato soggettivo della probabilità", Fundamenta Mathematicæ, 17, pp 298 – 329

Villegas (1964), "On qualitative probability $\sigma$-algebras", Ann. Math. Statist., 35, 4.

  • 4
    $\begingroup$ Why should my answer only deal with 'objective Bayesian' probabilities? The seminal work of Cox (1946) explicitly addresses the issue of subjectivity! It is a very interesting - and easy to read paper. I do not think that it makes sense to distinguish between 'subjective' and 'objective' Bayesian probabilities: Everything is always implicitly conditioned to the person performing the analysis -> and in this regard 'subjective'. $\endgroup$ – Summit Dec 1 '14 at 13:58
  • $\begingroup$ concerning the derivation of the axioms stated Kolmogorov's from Cox's: I am satisfied by the way it is done in section 2.2 of Beck, James L. "Bayesian system identification based on probability logic." Structural Control and Health Monitoring 17.7 (2010): 825-847. $\endgroup$ – Summit Dec 1 '14 at 14:20
  • 1
    $\begingroup$ @Summit: (1) You're right; it's rather that Ramsey & de Finetti's dispositional view of probability puts them squarely in the "subjective" camp, whereas Cox's is more generally applicable. (2) Are you saying countable additivity can be deduced from Cox's postulates? $\endgroup$ – Scortchi - Reinstate Monica Dec 1 '14 at 17:03
  • $\begingroup$ I extended my answer, and look forward to your comments. $\endgroup$ – Summit Dec 1 '14 at 18:32
  • 1
    $\begingroup$ @Summit: Thanks - I hope to find time to make mine even half as thorough. I've pointed out the gap between where you can get to from Cox's theorem & the "full" Kolmogorov axioms & think it's especially germane to the question (though I'd forgotten about it entirely when I first answered). Jaynes had some interesting things to say about this BTW. $\endgroup$ – Scortchi - Reinstate Monica Dec 3 '14 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.