Part of a series of trying to understand Bayesian vs frequentist: 1 2 3 4 5 6 7

I think I get the difference of how Bayesians and frequentists approach choosing between hypotheses, but I'm not quite sure if or how that is supposed to explain to me how they view probability.

From what I understand, according to Wiki, a frequentist "defines" probability as follows:

Given probability space $(\Omega, \mathscr{F}, \mathbb{P})$, $\forall A \in \mathscr{F}$, $\mathbb{P}(A) \approx \frac{n_A}{n_t}$, where $n_t$ is the number of trials conducted and $n_A$ is the number of times A has occurred in those trials.

Furthermore, $\mathbb{P}(A) = \lim_{n_t \to \infty} \frac{n_A}{n_t}$.

Okay, so how do Bayesians define probability? The above seems to be one approach to computing probability of an event in addition to defining a probability.

Bayesians seems to assume a prior probability, conduct some trials and then update their probability, but that doesn't really seem to explain how they define what probability is.

Wiki says 'Bayesian probability is a quantity that we assign for the purpose of representing a state of knowledge, or a state of belief.'

What exactly does that mean? Is state synonymous to degree? For example, Walter's state of belief that a particular coin is fair is represented with the number 0.1 while Jesse's state of belief that the same coin is fair is represented with the number 0.2. Given new information, Walter's state of belief could become 0.96 while Jesse's state of belief could become 0.03. So, initially, Walter was less inclined to believe the coin is fair, but later on Jesse was more inclined to believe the coin is fair?

I'm hoping for something in terms of symbols like the frequentist one above.

Same Wiki page says 'The Bayesian interpretation of probability can be seen as an extension of propositional logic that enables reasoning with hypotheses, i.e., the propositions whose truth or falsity is uncertain.', it seems like Bayesian and frequentist probability are analogous to fuzzy and Boolean logic, respectively.

  • 12
    Both frequentists and Bayesians use the same definition of probability, namely the one based on Kolmogorov's axioms of probability, i.e. probability as a finite measure. The difference between Bayesians and frequentists is thus not related to the definition of probability, but to how they link this concept to data. – Michael M Sep 18 '15 at 11:58
  • 1
    @MichaelM Should I have said 'interpret' rather than 'define' ? – BCLC Sep 18 '15 at 11:59
  • 1
    @BCLC, I gave you kudos for your question, but after not being completely clear on the answer I found the first entry on the "Related" column very useful. – Antoni Parellada Sep 18 '15 at 18:17
  • @AntoniParellada Thanks. You mean this?. Seen it. Don't know what 'In the Bayesian approach, we interpret probability distributions as quantifying our uncertainty about the world. ' means hence this question. Or you mean this? How does help this answer my question? – BCLC Sep 18 '15 at 18:23
  • 1
    @MichaelM: But see Do Bayesians accept Kolmogorov axioms?. – Scortchi Oct 15 '15 at 21:42
up vote 6 down vote accepted

I believe that most 'frequentists' and 'Bayesians' would rigorously define probability in the same way: via Kolmogorov's axioms and measure theory, modulo some issues about finite vs countable additivity, depending on who you're talking to. So in terms of 'symbols' I reckon you'll likely find more or less the same definition across the board. Everyone agrees on how probabilities behave.

I would say the primary difference is in the interpretation of what probabilities are. My (tongue-in-cheek militant Bayesian) preferred interpretation is that probabilities are coherent representations of information about events.

'Coherent' here has a technical meaning: it means that if I represent my information about the world in terms of probabilities and then use those probabilities to size my bets on the occurrence or nonoccurrence of any given event, I am assured that I can not be made a sure loser by agents betting against me.

Note that this involves no notion of 'long-run relative frequency'; indeed, I can coherently represent my information about a one-off event - like the sun exploding tomorrow - via the language of probability. On the other hand, it seems more difficult (or arguably less natural) to talk about the event "the sun will explode tomorrow" in terms of long-run relative frequency.

For a deep dive on this question I'd refer you to the first chapter of Jay Kadane's excellent (and free) Principles of Uncertainty.

UPDATE: I wrote a relatively informal blog post that illustrates coherence.

  • 'If you are a member of the general public: The fact that you are seeing this page indicates that the website you just visited is either experiencing problems, or is undergoing routine maintenance.' --> Maybe later then. Thanks – BCLC Sep 21 '15 at 10:39
  • @BCLC Is that for the PoU link? It's working for me. – jtobin Sep 21 '15 at 22:14
  • 1
    (+1) Very clear exposition. @BCLC: See Savage (1954), The Foundations of Statistics for an axiomatic approach. The gist is that if you want to represent degree of belief & insist on comparability (for any statements $A$ & $B$ you can say you believe one more than the other, or both equally) & coherence (as explained above), it turns out you have to use a probability - i.e. a real number subjective to Kolmogorov's axioms (bar one, perhaps). – Scortchi Oct 16 '15 at 13:30
  • I'm sure there's some good explanation about how Bayesian probabilities do not allow for dutch books, but I don't immediately see the connection, so I have trouble saying this is a clear explanation of how Bayesians see probability. And further, if the information an agent has betting against you is asymmetric, then you must be able to be made a sure loser? Maybe that question implies what I'm not understanding... – Cliff AB Jul 18 '17 at 3:05

As already noted by others, there is no specific Bayesian definition of probability. There is only one way of defining probability, i.e. it's a real number assigned to some event by a probability measure, that follows the axioms of probability. If there were different definitions of probability, we wouldn't be able use it consistently, since different people would understand different things behind it.

While there is only one way we define it, there are multiple ways to interpret the probability. Probability is a mathematical concept, not related anyhow to real world (quoting de Finetti, "probability does not exist"). To apply it to real world we need to translate, or interpret, the mathematics into real world happenings. There are multiple different ways to interpret the probability, even different interpretations among Bayesians (check Interpretations of Probability in Stanford Encyclopedia of Philosophy for a review). The one that is most commonly associated with Bayesian statistics is subjectivist view, also known as personalistic probability.

In subjectivist view, probability is a degree of belief, or degree of confirmation. It measures how much someone considers something believable. It can be analyzed, or observed, most clearly in terms of betting behavior (de Finetti, 1937; see also Savage, 1976; Kemeny, 1955):

Let us suppose that an individual is obliged to evaluate the rate $p$ at which he would be ready to exchange the possession of an arbitrary sum $S$ (positive or negative) dependent on the occurrence of a given event $E$, for the possession of the sum $pS$; we will say by definition that this number $p$ is the measure of the degree of probability attributed by the individual considered to the event $E$, or, more simply, that $p$ is the probability of $E$ (according to the individual considered; this specification can be implicit if there is no ambiguity).

Betting is one of the situations where one needs to quantify how "likely" he believes something to be and the measure of such belief is clearly a probability. Translating such belief to numbers, least to measure of belief, i.e. probability.

Bruno de Finetti, one of the major figures among subjectivists, notices that the subjectivist view is coherent with axioms of probability and it needs to follow them:

If we acknowledge only, first that one uncertain event can only appear to us (a) equally probable, (b) more probable, or (c) less probable then another; second that an uncertain event always seems to us more probable then an impossible event and less probable then a necessary event; and finally, third that when we judge an event $E'$ more probable then event $E$, which is itself more probable then an event $E''$, then event $E'$ can only appear more probable then $E''$ (transitive property), it will suffice to add to there three evidently trivial axioms a fourth, itself of purely qualitative nature, in order to construct rigorously the whole theory of probability. The fourth axiom tells us that inequalities are preserved in logical sums: if $E$ is incompatible with $E_1$ and with $E_2$, then $E_1 \lor E$ will be more or less probable then $E_2 \lor E$, or they will be equally probable, according to wherever $E_1$ is more or less probable then $E_2$, or they are equally probable. More generally, it may be deduced from this that two inequalities, such as

$$ E_1 \text{ is more probable then } E_2,\\ E_1' \text{ is more probable then } E_2',$$

can be added to give

$$ E_1 \lor E_1' \text{ is more probable then } E_2 \lor E_2' $$

provided that the events added are incompatible with each other ($E_1$ with $E_1'$, $E_2$ with $E_2'$).

Similar points are made by multiple different authors, like Kemeny (1955), or Savage (1972), who like de Finetti draw connections between the axioms and subjectivist view of probability. They also show that such measure of belief needs to be consistent with the axioms of probability (so if it looks like a probability and quacks like a probability...). Moreover, Cox (1946) shows that probability can be thought as an extension of formal logic that goes beyond binary true and false, allowing for uncertainties.

As you can see, this has nothing to do with frequencies. Of course, if you observe that nicotine smokers die of cancer more often then non-smokers, rationally you would assume such death to be more believable for a smoker, so frequency interpretation does not contradict the subjectivist view. What makes such interpretation appealing is that it can be applied also to cases that have nothing to do with frequencies (e.g. the probability that Donald Trump wins the 2016 US presidential election, the probability that there are other intelligent lifeforms somewhere in the space besides us etc). When adopting subjectivist view you can consider such cases in probabilistic manner and build statistical models of such scenarios (see example of election forecasting by FiveThirtyEight, that is consistent with thinking about probability as measuring degree of belief based on the available evidence). This makes such interpretation very broad (some say, overly broad), so we can flexibly adapt probabilistic thinking to different problems. Yes, it is subjective, but de Finetti (1931) notices that as frequentist definition is based on multiple unrealistic assumptions, it does not make it more "rational" interpretation.


de Finetti, B. (1937/1980). La Prévision: Ses Lois Logiques, Ses Sources Subjectives. [Foresight. Its Logical Laws, Its Subjective Sources.] Annales de l'Institut Henri Poincaré, 7, 1-68.

Kemeny, J. (1955). Fair Bets and Inductive Probabilities. Journal of Symbolic Logic, 20, 263-273.

Savage, L.J. (1972). The foundations of statistics. Dover.

Cox, R.T. (1946). Probability, frequency and reasonable expectation. American journal of physics, 14(1), 1-13.

de Finetti, B. (1931/1989). 'Probabilism: A critical essay on the theory of probability and on the value of science'. Erkenntnis, 31, 169-223.

  • +1 Very nice answer (somehow I didn't see it before), and the references are good. – amoeba Apr 26 '17 at 10:11

I'll try to be incredible clear with my terminology. As you did, we'll focus on one coin, $X \sim Bernoulli(p)$, so $Pr(X=1) = p$.

Bayesians and frequentists both view $X$ as a random variable and they share the same views about the probability distribution $Pr(X)$. However, Bayesians also use probability distributions to model their uncertainty about a fixed parameter, in this case $p$.

If we now let $x_1, x_2, \dots \sim Bernoulli(p)$ and define $h_n = \sum_{i=1}^n x_i$, as you pointed out

$$ \lim_{n\rightarrow \infty} \frac{h_n}{n}= p. $$

This is relevant because $h_n/n$ is the MLE for $p$. However notice that for any positive numbers $a,b$ (in fact they don't even need to be positive):

$$ \lim_{n\rightarrow \infty} \frac{h_n+a}{n+a+b}= p. $$

One draw back of the estimator $h_n/n$ is that for small $n$ this might be crazy. The most extreme example of this is when $n = 1$, our estimate of $p$ will be $0$ or $1$. What if we set $a=b=5$ and use the second estimate. If we get a $1$ on the first flip our updated estimate is $6/11$, greater than $50\%$ but not as extreme as $1$.

This more restrained estimate can be easily derived by expressing our uncertainty about $p$ in the form of a prior (and eventually posterior) distribution. If you would like to look up this example in depth this is known as the Beta-Binomial. It involves putting a Beta prior on the parameter of a Binomial Distribution, and taking the expectation of the resulting posterior.

  • So Bayesians interpret probability as the limit of an MLE? Also, is that first limit statement due to $E(X) = P(X=1) = p$ and one of the laws of large numbers? – BCLC Sep 18 '15 at 16:15
  • This does not answer the question asked... This answer is more about using priors and Bayesian inference, not on how probability is defined. – Tim Sep 18 '15 at 16:22
  • @Tim I was thinking that that first limit statement is the answer? – BCLC Sep 18 '15 at 16:33
  • @Tim Maybe you should say 'how probability is interpreted'... – BCLC Sep 18 '15 at 16:34
  • I tried to explain and give an example of the fact that probability is defined the same way but used differently. – jlimahaverford Sep 18 '15 at 16:35

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.