11
$\begingroup$

Suppose that random variable $U$ follows a continuous Uniform distribution with parameters 0 and 10 (i.e. $U \sim \rm{U}(0,10)$ )

Now let's denote A the event that $U$ = 5 and B the event that $U$ is equal either to $5$ or 6. According to my understanding, both events have zero probability to occur.

Now, if we consider to compute $P(A|B)$ , we cannot use the conditional law $P\left( {A|B} \right) = \frac{{P\left( {A \cap B} \right)}}{{P\left( B \right)}}$, because $P(B)$ is equal to zero. However, my intuition tells me that $P(A|B) = 1/2$.

$\endgroup$
  • 2
    $\begingroup$ What would your intuition tell you if $U$ had non-uniform density $0.02u, u \in (0,10)$? $\endgroup$ – Dilip Sarwate Feb 6 '15 at 4:11
  • 1
    $\begingroup$ @DilipSarwate My intuition would tell me that the answer is a number slightly lower than 0.5 $\endgroup$ – Noob Feb 6 '15 at 19:52
9
$\begingroup$

"The concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible." A. Kolmogorov

For continuous random variables, $X$ and $Y$ say, conditional distributions are defined by the property that they recover the original probability measure, that is, for all measurable sets $A\in\mathcal{B}(\mathbf{X})$, $B\in\mathcal{B}(\mathbf{Y})$,$$\mathbb{P}(X\in A,Y\in B)=\int_B \text{d}P_Y(y) \int_B \text{d}P_{X|Y}(x|y)$$This implies that the conditional density is defined arbitrarily on sets of measure zero or, on other words, that the conditional density $p_{X|Y}(x|y)$ is defined almost everywhere. Since the set $\{5,6\}$ is of measure zero against the Lebesgue measure, this means that you can define both $p(5)$ and $p(6)$ in absolutely arbitrary manners and hence that the probability $$\mathbb{P}(U=5|U\in\{5,6\})$$can take any value.

This does not mean you cannot define a conditional density by the ratio formula $$f(y|x)=f(x,y)\big/f(x)$$as in the bivariate normal case but simply that the density is only defined almost everywhere for both $x$ and $y$.

"Many quite futile arguments have raged - between otherwise competent probabilists - over which of these results is 'correct'." E.T. Jaynes

The fact that the limiting argument (when $\epsilon$ goes to zero) in the above answer seems to give a natural and intuitive answer is related with Borel's paradox. The choice of the parametrisation in the limit matters, as shown by the following example I use in my undergrad classes.


Take the bivariate normal $$X,Y\stackrel{\text{i.i.d.}}{\sim}\mathcal{N}(0,1)$$ What is the conditional density of $X$ given that $X=Y$?


If one starts from the joint density $\varphi(x)\varphi(y)$, the "intuitive" answer is [proportional to] $\varphi(x)^2$. This can be obtained by considering the change of variable $$(x,t)=(x,y-x) \sim \varphi(x)\varphi(t+x)$$ where $T=Y-X$ has the density $\varphi(t/\sqrt{2})/\sqrt{2}$. Hence $$f(x|t)=\dfrac{\varphi(x)\varphi(t+x)}{\varphi(t/\sqrt{2})/\sqrt{2}}$$ and $$f(x|t=0)=\dfrac{\varphi(x)\varphi(x)}{\varphi(0/\sqrt{2})/\sqrt{2}}=\varphi(x)^2\sqrt{2}$$ However, if one considers instead the change of variable $$(x,r)=(x,y/x) \sim \varphi(x)\varphi(rx)|x|$$ the marginal density of $R=Y/X$ is the Cauchy density $\psi(r)=1/\pi\{1+r^2\}$ and the conditional density of $X$ given $R$ is $$f(x|r)=\varphi(x)\varphi(rx)|x| \times \pi \{1+r^2\}$$ Therefore, $$f(x|r=1)= \pi\varphi(x)^2|x|/2\,.$$ And here lies the "paradox": the events $R=1$ and $T=0$ are the same as $X=Y$, but they lead to different conditional densities on $X$.

$\endgroup$
  • 2
    $\begingroup$ This is just plain wrong. If you take a rigorous course in probability theory you will see that conditioning on events of measure zero is possible, and practical. Consider a bitivariate Gaussian. Everyone knows you can condition on the first variable taking the value zero, though this event has zero probability. See wikipedia. en.wikipedia.org/wiki/… $\endgroup$ – Yair Daon Feb 6 '15 at 14:47
5
$\begingroup$

Here's a controversial answer:

Xi'an is right that you can't condition on events with zero probability. However, Yair is also right that once you decide on a limiting process, you can evaluate a probability. The problem is there are many limiting processes that arrive at the desired condition.

I think the principle of indifference can sometimes resolve such choices. It argues that the result should not be affected by an arbitrary interchange of labels. in your case, say, flipping the interval so that it's uniform on $(1, 11)$ and the points 5 and 6 have been switched. Flipping changes an answer $p$ to $1-p$. So if you had chosen a different limiting process for one than the other, then you have by an arbitrary change of labels (in this case, changing positive infinity for negative infinity) gotten a different result. That should not happen according to the principle of indifference. Therefore, the answer is 0.5 as you guessed.

Note that many statisticians do not accept the principle of indifference. I like it because it reflects my intuitions. Although I'm not always sure how to apply it, maybe in 50 years it will be more mainstream?

$\endgroup$
  • $\begingroup$ Thank you for a thoughtful post. I, for one, seriously doubt the "principle of indifference" will ever be mainstream, because it is not workable. Your argument falls apart when the underlying values are re-expressed. The uniform distribution on $[0,10]$ might thereby become, say, a Cauchy distribution, $5$ could become $0$, and $6$ become $\sqrt{1-\frac{2}{\sqrt{5}}}$. Your "principle of indifference" now produces a completely different answer. (I used the probability transforms to work out this example.) $\endgroup$ – whuber Mar 27 '15 at 21:37
  • $\begingroup$ @whuber: The flipping argument wouldn't work for a Cauchy distribution, unless you flipped around its mode. $\endgroup$ – Neil G Mar 27 '15 at 21:39
  • $\begingroup$ Sure it does: there are plenty of ways to transform one continuous distribution to another that swap two values. Actually, your "flipping" didn't even preserve the original distribution. (It changed its support altogether.) So it would appear that all you're doing is replacing one distribution by another one. There doesn't seem to be any principle operating here at all. $\endgroup$ – whuber Mar 27 '15 at 21:45
  • $\begingroup$ @whuber: it replaced one distribution with another whereby the uniform regions around the 5 and 6 were unchanged — in the same way I think that zooming out tries to leave densities unchanged in the original circles in the Bertrand paradox. $\endgroup$ – Neil G Mar 27 '15 at 21:50
  • 1
    $\begingroup$ @whuber: You're right. I really liked Potato's answer to one of my questions. I personally think that if there is discrepancy between theory and intuition, that we should seek new, more complete theories. Maybe the "principle of indifference" isn't quite right, or isn't generally workable, but I have a natural desire for probability theory to answer questions for which we have an intuitive understanding. Maybe Lebesgue had the same kind of angst about Riemann integration when he created his integral? $\endgroup$ – Neil G Mar 28 '15 at 0:52
1
$\begingroup$

Yes we can! You can condition on events of zero probability! The math gets complicated - you need some measure theory but you can do it. In simple cases like this I would seek intuition by defining $A = [5 - \frac{\epsilon}{2} , 5 + \frac{\epsilon}{2}]$ and $B = [5 - \frac{\epsilon}{4} , 5 + \frac{\epsilon}{4}] \cup [6 - \frac{\epsilon}{4} , 6 + \frac{\epsilon}{4}]$. Do everything now as you did before and take $\epsilon \to 0$.

Let me stress again (and again) that the above method is used for intuition. Conditioning on events of zero probability is done very often without much thought. The best example I can think of is if $(X_1, X_2) \sim \mathcal{N}(0, \Sigma)$ is a bivariate gaussian. One often considers the density of $X_1$ given (say) $X_2 = 0$, which is an event of measure zero. This is well grounded in theory, but not at all trivial. Regarding @Xi'an's quote of Kolmogorov - I can only quote Varadhan: "One of our goals is to seek a definition that makes sense when $P(\xi = a) = 0$" (Probability Theory, Courant lecture notes, page 74).

So, yes, you can give meaning to conditioning on events of measure zero.

$\endgroup$
  • 4
    $\begingroup$ Suppose $U\sim \text{U}[0,10]$: that is, both $0$ and $10$ are possible. How would you deal with the situation when $A=\{0\}$ and $B=\{0,6\}$? Would $P(A|B)=1/2$ (which "intuitively" is the right answer because all numbers in $[0,10]$ have the same densities) or perhaps $1/3$ (which a simple change of $5$ to $0$ in your formula would give) or even $0$? $\endgroup$ – whuber Feb 6 '15 at 17:41
  • 2
    $\begingroup$ @YairDaon Thank you for you answer! If I understood well, you mean to do the following: for small $\varepsilon $, we have: $P\left( {A|B} \right) = \frac{{P\left( {A \cap B} \right)}}{{P\left( B \right)}} = \frac{{\int\limits_{5 - \frac{\varepsilon }{4}}^{5 + \frac{\varepsilon }{4}} {f\left( u \right)du} }}{{\int\limits_{5 - \frac{\varepsilon }{4}}^{5 + \frac{\varepsilon }{4}} {f\left( u \right)du} + \int\limits_{6 - \frac{\varepsilon }{4}}^{6 + \frac{\varepsilon }{4}} {f\left( u \right)du} }} = \frac{{\frac{\varepsilon }{2}}}{{\frac{\varepsilon }{2} + \frac{\varepsilon }{2}}} = 0.5$ $\endgroup$ – Noob Feb 6 '15 at 20:08
  • 3
    $\begingroup$ @YairDaon But I think that the result is not invariant if originally we had defined A as $\left[ {5 - \frac{\varepsilon }{8},5 + \frac{\varepsilon }{8}} \right]$ (and B the same as before). In such a case the result would be ${\frac{1}{8}}$ $\endgroup$ – Noob Feb 6 '15 at 20:10
  • 4
    $\begingroup$ It is excellent for the intuition by showing there is no unique answer: that is the basis for Kolmogorov's statement quoted by @Xi'an. The fact you had to change your procedure to make things come out as you thought they should ought to alert you to the problems with this approach. $\endgroup$ – whuber Feb 7 '15 at 0:14
  • 3
    $\begingroup$ The density of $X_2$ given $X_1$ is well-defined, contrary to the density of $X_2$ given $X_1=0$. $\endgroup$ – Xi'an Feb 7 '15 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.