# Mathematical basis for conditional probability

In order to get more fundamental in my understanding of probability I watched mathematicalmonk's lectures involving $\sigma$-algebras etc. - good. One of my main concerns was to better understand the basis for conditional probability: $P[A|B] = P[AB]/P[B]$. My question is simple

How do we know that this quotient is indeed a probability measure?
After all, things like the Odds'' $P[A] / P[\bar{A}]$ is not a probability measure.

-
well, how do we know any measure is a measure? –  TenaliRaman Jan 11 '13 at 14:12
The tough thing about conditional probability is that many times these quotients do not exist, because $P[B]=0$, but nevertheless the conditional probability $P[A|B]$ makes good sense. Many rigorous treatments lead back to a fundamental idea, the Radon-Nikodym theorem. –  whuber Jan 11 '13 at 14:30
+1 for mentioning the Radon-Nicodym theorem. –  Placidia Jan 11 '13 at 15:46
planetmath.org/… –  TenaliRaman Jan 11 '13 at 16:01
@whuber Of course I agree with you but I am under the impression that the OP is about the elementary case $\Pr(B)>0$ –  Stéphane Laurent Jan 11 '13 at 19:00

Assume that $P(B) > 0$ and define $P(A\mid B) = \frac{P(A\cap B)}{P(B)}$ for all events $A$ in the $\sigma$-algebra $\mathcal F$ over which $P(\cdot)$ is a probability measure. Note that we are assuming that $B \in \mathcal F$ (else $P(B)$ would not be defined). Then $P(\cdot\mid B)$ also is a probability measure on the same $\sigma$-algebra $\mathcal F$.

Axiom I: $P(A\mid B) \geq 0$ for all events $A \in \mathcal F$.

This should be obvious since $P(A\cap B) \geq 0$. Those who take Axiom I as $0 \leq P(A) \leq 1$ should note that since $(A\cap B) \subset B$, we have that $P(A\cap B) \leq P(B)$ and so $P(A\cap B) \leq 1$ as needed.

Axiom II: $P(\Omega\mid B) = 1$

This should also be obvious since $(\Omega \cap B) = B$ and so $P(\Omega\mid B) = 1$.

Axiom III: For any countable sequence of disjoint events $A_1, A_2, \ldots$, in $\mathcal F$ $$P\left(\left. \bigcup_{n=1}^\infty A_n \,\right\vert \,B\right) = \sum_{n=1}^\infty P(A_n \mid B)$$

This requires just a little bit of work. Begin by noting that $A_1\cap B, A_2\cap B, \ldots$ is a countable sequence of disjoint events. Then,

\begin{align*}P\left(\left. \bigcup_{n=1}^\infty A_n \,\right\vert \,B\right) &= \frac{P\left(\left(\bigcup_{n=1}^\infty A_n\right)\cap B\right)}{P(B)}\\ &= \frac{P\left(\bigcup_{n=1}^\infty (A_n\cap B)\right)}{P(B)}\\ &= \frac{\sum_{n=1}^\infty P(A_n\cap B)}{P(B)}\\ &= \sum_{n=1}^\infty P(A_n \mid B) \end{align*}

-
It looks so clear after you have seen it! Thanks. –  JonesTheAstronomer Jan 13 '13 at 16:23
@Dilip I think this answer may actually address my question? –  jsk May 15 at 17:29

The key motivation for the notion of probability is the long-term relative frequency phenomenon (LTRF). Please forget mathematics when reading the following sentence: let ${\cal E}$ be an experiment which is repeated infinitely many times and independently, in that no issue influences others, and let $A$ be an event related to the possible issues of ${\cal E}$; then the relative frequency of the occurences of $A$ converges to some number. The LTRF is considered as a "law of nature", not a mathematical statement; in particular, the notion of "independence" is not a mathematical one here; it intuitively means that no issue influences others. And as I said the LTRF is the key motivation for probability: $\Pr(A)$ is the mathematical modelling of the limit number mentioned in the LTRF.

The LTRF clearly motivates the basic additivity axiom of probability: $\Pr(A \cup B) = \Pr(A)+\Pr(B)$ for disjoint events $A$ and $B$. Indeed, considering the LTRF context and denoting by $N(A,n)$ the number of occurences of an event $A$ in the first $n$ independent performances of ${\cal E}$, one clearly has $N(A\cup B, n) = N(A,n)+N(B,n)$ and $\frac{N(A\cup B, n)}{n} \to \Pr(A\cup B)$ whereas $\frac{N(A, n)}{n} \to \Pr(A)$ and $\frac{N(B, n)}{n} \to \Pr(B)$. In fact the "basic" additivity holds from $N(\cdot, n)$. By "basic" I mean "finite". The theoretical $\sigma$-additivity is a purely mathematic notion.

The LTRF also motivates the notion of conditional probability $\Pr(B \mid A)$. A "by-product" of the conditional probability is the notion of independence (i.e. the mathematical modelling of the intuitive notion of independence mentioned in the LTRF): $A$ and $B$ will be said to be independent when $\Pr(B\mid A)=\Pr(B)$. Now the conditional probability is introduced as follows in the LTRF context: the conditional probability $\Pr(B \mid A)$ is the long-term proportion of experiments for which $B$ occurs among those experiments for which $A$ occurs. That is, $\Pr(B \mid A)$ is considered as the "LTRF limit" of $\frac{N(A\cap B, n)}{N(A,n)}$. Note that $\frac{N(A\cap B, n)}{N(A,n)}= \frac{N(A\cap B, n)/n}{N(A,n)/n}$, and by the LTRF rule it yields the mathematical definition $\Pr(B \mid A)=\frac{\Pr(A\cap B)}{\Pr(A)}$.

This fundation shows that the conditional probability $\Pr(\cdot \mid A)$ must be a probability. I know that does not answer your mathematical question, but I hope that helps you to familiarize yourself with the notion of conditional probability.

The answer to your question is not difficult: it is easy to see that $B \mapsto \frac{\Pr(A\cap B)}{\Pr(A)}$ satisfies the axioms of probability. In fact the $\sigma$-additivity works for $B \mapsto \Pr(A\cap B)$, and the other axioms obviously hold true.

## References

D. Williams. Weighing the Odds: A Course in Probability and Statistics. Cambridge University Press, 2001.

-
I use something similar to this in my teaching, though less incisive. I have two "problems" with this, however. (a) it is only an asymptotic limit and (b) it is unashamedly frequentist. But thanks. –  JonesTheAstronomer Jan 12 '13 at 9:12
@JonesTheAstronomere The LTRF reappears in probability theory as the law of large numbers. What is your problem ? –  Stéphane Laurent Jan 12 '13 at 9:47
I know what the Law of Large Numbers is (and was motivated by your very clear response to re-read it), but maybe I don't fully understand LLN! Does the LLN follow from the Kolmogorov axioms (or any other premise)? In my field of research much of the data analysis is Bayesian. Students grow up with the idea that probability is a measure that expresses something called a `degree of belief', rather than a frequency obtained by averaging over realizations. So I guess I was looking for an explanation that simply involved using the axioms, or even going to measure theory. –  JonesTheAstronomer Jan 12 '13 at 11:07
@JonesTheAstronomer Yes, LLN is a theorem in probability theory: it follows from the Kolmogorov axioms of the definition of a probability space. –  Stéphane Laurent Jan 12 '13 at 11:12
Thanks - I had not really appreciated that (not being a statistician) and so with that clue I found the derivation. It is what I had failed to understand from the discussion of Tenali and whuber. In effect you have all answered my question - thanks. –  JonesTheAstronomer Jan 12 '13 at 15:34

I am not a mathematician, but a probability measure has to sum to $1$ when added over all events of $A$ given $B$:

$$\displaystyle \sum_{A} \mathbb{P}(A \lvert B) = 1$$

and that does not hold for the odds, for example.

In other words, the axioms of probability must hold.

-
Correct! Of course one needs to show it correctly. Here you go - planetmath.org/… –  TenaliRaman Jan 11 '13 at 15:59
@Tenali Correct, yes--but it misses the point. One uses the sigma-algebra formalism to handle the non-elementary situation where sums have to be replaced by (Lebesgue) integrals. This answer does not apply in that setting; different tools have to be used, as I pointed out in a comment to the question. –  whuber Jan 11 '13 at 16:05
@whuber The question asked how do we know P[A/B] is a measure. And as long P(B) > 0, one has to just check the definition to see if it is a measure or not. I don't see why one needs to ascribe to Radon-Nikodym theorem for this? Radon-Nikodym Theorem, as far as I know, is fundamental if you want to see why conditional expectation exists. But I don't see why you need that power to show that the conditional probability is a measure. –  TenaliRaman Jan 11 '13 at 18:11
While the axioms of probability must indeed hold, I am not sure what interpretation to place on the word "all" in the statement "when added over all events of $A$ given $B$, $\sum_A P(A|B) = 1$". The sum of the probabilities (whether conditional or nor) of all $2^m$ events defined on a discrete sample space is $2^{m-1}$ since the events can be paired up into $2^{m-1}$ pairs $\{A, A^c\}$ of complementary probabilities. What would make a more appropriate statement is that the sum of the probabilities of all the events comprising a partition of the sample space $\Omega$ is $1$. –  Dilip Sarwate Jan 11 '13 at 18:15
@whuber or maybe I see now, since $\mathbb{E}[1_{A}] = P(A)$ so by showing that conditional expectation exists actually implies that it is a stronger result and hence shows conditional probability exists? –  TenaliRaman Jan 11 '13 at 18:16