Say that $Y$ is a continuous random variable, and $X$ is a discrete one. $$ \Pr(X=x|Y=y) = \frac{\Pr(X=x)\Pr(Y=y|X=x)}{\Pr(Y=y)} $$

As we know, $\Pr(Y=y) = 0$ because $Y$ is a continuous random variable. And based on this, I am tempted to conclude that the probability $\Pr(X=x|Y=y)$ is undefined.

However, Wikipedia claims here that it is actually defined as follows: $$ \Pr(X=x|Y=y) = \frac{\Pr(X=x) f_{Y|X=x}(y)}{f_Y(y)} $$

Question: Any idea how did Wikipedia manage to get that probability defined?

My attempt

Here is my attempt in order to get that Wikipedia outcome in terms of limits: $$\begin{split}\require{cancel} \Pr(X=x|Y=y) &= \frac{\Pr(X=x)\Pr(Y=y|X=x)}{\Pr(Y=y)}\\ &= \lim_{d \rightarrow 0}\frac{\Pr(X=x) \big(d \times f_{Y|X=x}(y)\big)}{\big(d \times f_Y(y)\big)}\\ &= \lim_{d \rightarrow 0}\frac{\Pr(X=x) \big(\cancel{d} \times f_{Y|X=x}(y)\big)}{\big(\cancel{d} \times f_Y(y)\big)}\\ &= \frac{\Pr(X=x) f_{Y|X=x}(y)}{f_Y(y)}\\ \end{split}$$

Now, $\Pr(X=x|Y=y)$ seems to be defined to be $\frac{\Pr(X=x) f_{Y|X=x}(y)}{f_Y(y)}$, which matches that Wikipedia claim.

Is that how Wikipedia did it?

But I am still feeling that I am abusing calculus here. So I think that $\Pr(X=x|Y=y)$ is undefined, but in the limit as we get as close as possible to define $\Pr(Y=y)$ and $\Pr(Y=y|X=x)$, but not eyactly, then $\Pr(X=x|Y=y)$ is defined.

But I am largely unsure about many things, including the limits trick that I did there, I feel that maybe I am not even fully understanding the meaning of what I did.

  • 1
    $\begingroup$ Indeed, Pr(X=x)=0 but density of X in x f(x) may not be equal to 0. Shouldn't you use a label 'self-study'?? $\endgroup$ – Lil'Lobster Dec 12 '16 at 12:14
  • 2
    $\begingroup$ @Lil As far as I know, the 'self-study' tag is when solving homework. I'm not doing that. $\endgroup$ – caveman Dec 12 '16 at 12:16
  • 1
    $\begingroup$ The Wikipedia page actually refers to the derivation: en.wikipedia.org/wiki/Bayes'_theorem#Derivation $\endgroup$ – Ytsen de Boer Dec 12 '16 at 12:41
  • 3
    $\begingroup$ I am afraid your derivation has no mathematical justification as $\mathbb{P}(Y=y)=0$ for all $y\in\mathcal{Y}$ when $Y$ is continuous. $\endgroup$ – Xi'an Dec 12 '16 at 13:11

The conditional probability distribution $\mathbb{P}(X=x|Y=y)$, $x\in\mathcal{X}$, $y\in\mathcal{Y}$, is formally defined as a solution of the equation$$\mathbb{P}(X=x,Y\in A)=\int_{A}\mathbb{P}(X=x|Y=y)f_Y(y)\text{d}y\quad\forall A\in\sigma(\mathcal{Y})$$where $\sigma(\mathcal{Y})$ denotes the $\sigma$-algebra associated with the distribution of $Y$. One of those solutions is provided by Bayes' (1763) formula as indicated in Wikipedia:$$\mathbb{P}(X=x|Y=y) = \dfrac{\mathbb{P}(X=x) f_{Y|X=x}(y)}{f_Y(y)}\qquad\forall x\in\mathcal{X},\ y\in\mathcal{Y}$$although versions that are arbitrarily defined on a measure-zero set in $\sigma(\mathcal{Y})$ are also valid.

The concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible. For we can obtain a probability distribution for [the latitude] on the meridian circle only if we regard this circle as an element of the decomposition of the entire spherical surface onto meridian circles with the given poles — Andrei Kolmogorov

As shown by the Borel-Kolmogorov paradox, given a specific value $y_0$ potentially taken $Y$, the conditional probability distribution $\mathbb{P}(X=x|Y=y_0)$ has no precise meaning, not only because the event $\{\omega;\,Y(\omega)=y_0\}$ is of measure zero, but also because this event can be interpreted as measurable against an infinite range of $\sigma$-algebras.

Note: Here is an even more formal introduction, take from a review of probability theory on Terry Tao's blog:

Definition 9 (Disintegration) Let $Y$ be a random variable with range $R$. A disintegration $(R', (\mu_y)_{y \in R'})$ of the underlying sample space $\Omega$ with respect to $Y$ is a subset $R'$ of $R$ of full measure in $\mu_Y$ (thus $Y \in R'$ almost surely), together with assignment of a probability measure ${\bf P}(|Y=y)$ on the subspace $\Omega_y := \{ \omega \in \Omega: Y(\omega)=y\}$ of $\Omega$ for each $y \in R$, which is measurable in the sense that the map $y \mapsto {\bf P}(F|Y=y)$ is measurable for every event $F$, and such that $$ \displaystyle {\bf P}(F) = {\bf E} {\bf P}(F|Y) $$ for all such events, where ${\bf P}(F|Y)$ is the (almost surely defined) random variable defined to equal ${\bf P}(F|Y=y)$ whenever $Y=y$.

Given such a disintegration, we can then condition to the event $Y=y$ for any $y \in R'$ by replacing $\Omega$ with the subspace $\Omega_y$ (with the induced $\sigma$-algebra), but replacing the underlying probability measure ${\bf P}$ with ${\bf P}(|Y=y)$. We can thus condition (unconditional) events $F$ and random variables $X$ to this event to create conditioned events $(F|Y=y)$ and random variables $(X|Y=y)$ on the conditioned space, giving rise to conditional probabilities ${\bf P}(F|Y=y)$ (which is consistent with the existing notation for this expression) and conditional expectations ${\bf E}(X|Y=y)$ (assuming absolute integrability in this conditioned space). We then set ${\bf E}(X|Y)$ to be the (almost surely defined) random variable defined to equal ${\bf E}(X|Y=y)$ whenever $Y=y$.

| cite | improve this answer | |
  • 1
    $\begingroup$ Already +1'd, but... maybe it's nitpicking, but wouldn't it be more accurate to refer to Bayes theorem as a formula by Bayes / Laplace ..? $\endgroup$ – Tim Dec 12 '16 at 13:35
  • 2
    $\begingroup$ @Tim: thank you, but I do not want to sound overly chauvinistic! And it is a fact that Bayes' formula for $X$ discrete (Binomial) and $Y$ continuous (Beta) appears in Bayes (1763) paper. Of course, Laplace set the result in much broader generality. $\endgroup$ – Xi'an Dec 12 '16 at 14:04

I'll give a sketch of how the pieces can fit together when $Y$ is continuous and $X$ is discrete.

The mixed joint density:

$$ f_{XY}(x,y) $$

Marginal density and probability:

$$ f_Y(y) = \sum_{x \in X} f_{XY}(x, y) $$

$$ P(X = x) = \int f_{XY}(x, y) \;dy$$

Conditional density and probability:

$$ f_{Y\mid X}(y \mid X = x) = \frac{f_{XY}(x, y)}{P(X=x)} $$

$$ P(X=x \mid Y = y) = \frac{f_{XY}(x, y)}{f_Y(y)} $$

Bayes Rule:

$$ f_{Y\mid X}(y \mid X = x) = \frac{P(X=x \mid Y = y) f_Y(y)}{P(X=x)} $$

$$ P(X=x \mid Y = y) = \frac{f_{Y\mid X}(y \mid X = x)P(X=x)}{f_Y(y)}$$

Of course, the modern, rigorous way to deal with probability is through measure theory. For a precicse definition, see Xi'an's answer.

| cite | improve this answer | |

Note that the Wikipedia article actually uses the following definition: $$f_X(x|Y=y) = \frac{P(Y=y|X=x)f_X(x)}{p(Y=y)} $$ That is, it treats the outcome as a density, not a probability as you have it. So I'd say you're right that $P(X=x|Y=y)$ is undefined when $X$ is continuous and $Y$ discrete, which is why we instead consider only probability densities over $X$ in that case.

Edit: Due to a confusion about notation (see comments) the above actually refers to the opposite situation to what caveman was asking about.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.