How is $\Pr(X=x|Y=y)$ defined when $Y$ is continous and $X$ discrete?

Question

Say that $Y$ is a continuous random variable, and $X$ is a discrete one. $$ \Pr(X=x|Y=y) = \frac{\Pr(X=x)\Pr(Y=y|X=x)}{\Pr(Y=y)} $$

As we know, $\Pr(Y=y) = 0$ because $Y$ is a continuous random variable. And based on this, I am tempted to conclude that the probability $\Pr(X=x|Y=y)$ is undefined.

However, Wikipedia claims here that it is actually defined as follows: $$ \Pr(X=x|Y=y) = \frac{\Pr(X=x) f_{Y|X=x}(y)}{f_Y(y)} $$

Question: Any idea how did Wikipedia manage to get that probability defined?

---

My attempt

Here is my attempt in order to get that Wikipedia outcome in terms of limits: $$\begin{split}\require{cancel} \Pr(X=x|Y=y) &= \frac{\Pr(X=x)\Pr(Y=y|X=x)}{\Pr(Y=y)}\\ &= \lim_{d \rightarrow 0}\frac{\Pr(X=x) \big(d \times f_{Y|X=x}(y)\big)}{\big(d \times f_Y(y)\big)}\\ &= \lim_{d \rightarrow 0}\frac{\Pr(X=x) \big(\cancel{d} \times f_{Y|X=x}(y)\big)}{\big(\cancel{d} \times f_Y(y)\big)}\\ &= \frac{\Pr(X=x) f_{Y|X=x}(y)}{f_Y(y)}\\ \end{split}$$

Now, $\Pr(X=x|Y=y)$ seems to be defined to be $\frac{\Pr(X=x) f_{Y|X=x}(y)}{f_Y(y)}$, which matches that Wikipedia claim.

Is that how Wikipedia did it?

But I am still feeling that I am abusing calculus here. So I think that $\Pr(X=x|Y=y)$ is undefined, but in the limit as we get as close as possible to define $\Pr(Y=y)$ and $\Pr(Y=y|X=x)$, but not eyactly, then $\Pr(X=x|Y=y)$ is defined.

But I am largely unsure about many things, including the limits trick that I did there, I feel that maybe I am not even fully understanding the meaning of what I did.

Answer 1

The conditional probability distribution $\mathbb{P}(X=x|Y=y)$, $x\in\mathcal{X}$, $y\in\mathcal{Y}$, is formally defined as a solution of the equation$$\mathbb{P}(X=x,Y\in A)=\int_{A}\mathbb{P}(X=x|Y=y)f_Y(y)\text{d}y\quad\forall A\in\sigma(\mathcal{Y})$$where $\sigma(\mathcal{Y})$ denotes the $\sigma$-algebra associated with the distribution of $Y$. One of those solutions is provided by Bayes' (1763) formula as indicated in Wikipedia:$$\mathbb{P}(X=x|Y=y) = \dfrac{\mathbb{P}(X=x) f_{Y|X=x}(y)}{f_Y(y)}\qquad\forall x\in\mathcal{X},\ y\in\mathcal{Y}$$although versions that are arbitrarily defined on a measure-zero set in $\sigma(\mathcal{Y})$ are also valid.

The concept of a conditional probability with regard to an isolated hypothesis whose probability equals 0 is inadmissible. For we can obtain a probability distribution for [the latitude] on the meridian circle only if we regard this circle as an element of the decomposition of the entire spherical surface onto meridian circles with the given poles — Andrei Kolmogorov

As shown by the Borel-Kolmogorov paradox, given a specific value $y_0$ potentially taken $Y$, the conditional probability distribution $\mathbb{P}(X=x|Y=y_0)$ has no precise meaning, not only because the event $\{\omega;\,Y(\omega)=y_0\}$ is of measure zero, but also because this event can be interpreted as measurable against an infinite range of $\sigma$-algebras.

Note: Here is an even more formal introduction, take from a review of probability theory on Terry Tao's blog:

Definition 9 (Disintegration) Let $Y$ be a random variable with range $R$. A disintegration $(R', (\mu_y)_{y \in R'})$ of the underlying sample space $\Omega$ with respect to $Y$ is a subset $R'$ of $R$ of full measure in $\mu_Y$ (thus $Y \in R'$ almost surely), together with assignment of a probability measure ${\bf P}(|Y=y)$ on the subspace $\Omega_y := \{ \omega \in \Omega: Y(\omega)=y\}$ of $\Omega$ for each $y \in R$, which is measurable in the sense that the map $y \mapsto {\bf P}(F|Y=y)$ is measurable for every event $F$, and such that $$ \displaystyle {\bf P}(F) = {\bf E} {\bf P}(F|Y) $$ for all such events, where ${\bf P}(F|Y)$ is the (almost surely defined) random variable defined to equal ${\bf P}(F|Y=y)$ whenever $Y=y$.

Given such a disintegration, we can then condition to the event $Y=y$ for any $y \in R'$ by replacing $\Omega$ with the subspace $\Omega_y$ (with the induced $\sigma$-algebra), but replacing the underlying probability measure ${\bf P}$ with ${\bf P}(|Y=y)$. We can thus condition (unconditional) events $F$ and random variables $X$ to this event to create conditioned events $(F|Y=y)$ and random variables $(X|Y=y)$ on the conditioned space, giving rise to conditional probabilities ${\bf P}(F|Y=y)$ (which is consistent with the existing notation for this expression) and conditional expectations ${\bf E}(X|Y=y)$ (assuming absolute integrability in this conditioned space). We then set ${\bf E}(X|Y)$ to be the (almost surely defined) random variable defined to equal ${\bf E}(X|Y=y)$ whenever $Y=y$.

Answer 2

I'll give a sketch of how the pieces can fit together when $Y$ is continuous and $X$ is discrete.

The mixed joint density:

$$ f_{XY}(x,y) $$

Marginal density and probability:

$$ f_Y(y) = \sum_{x \in X} f_{XY}(x, y) $$

$$ P(X = x) = \int f_{XY}(x, y) \;dy$$

Conditional density and probability:

$$ f_{Y\mid X}(y \mid X = x) = \frac{f_{XY}(x, y)}{P(X=x)} $$

$$ P(X=x \mid Y = y) = \frac{f_{XY}(x, y)}{f_Y(y)} $$

Bayes Rule:

$$ f_{Y\mid X}(y \mid X = x) = \frac{P(X=x \mid Y = y) f_Y(y)}{P(X=x)} $$

$$ P(X=x \mid Y = y) = \frac{f_{Y\mid X}(y \mid X = x)P(X=x)}{f_Y(y)}$$

Of course, the modern, rigorous way to deal with probability is through measure theory. For a precicse definition, see Xi'an's answer.

Answer 3

Note that the Wikipedia article actually uses the following definition: $$f_X(x|Y=y) = \frac{P(Y=y|X=x)f_X(x)}{p(Y=y)} $$ That is, it treats the outcome as a density, not a probability as you have it. So I'd say you're right that $P(X=x|Y=y)$ is undefined when $X$ is continuous and $Y$ discrete, which is why we instead consider only probability densities over $X$ in that case.

Edit: Due to a confusion about notation (see comments) the above actually refers to the opposite situation to what caveman was asking about.