Formula of conditional probability when we have discrete and continuous random variables

Question

Consider two random variables $Y,V$ where $Y$ is discrete and $V$ is continuous. Is it true that $$ \Pr(Y=y)=\int \Pr(Y=y|V=v) f(v) dv, $$ where $f$ is the probability density function of $V$? I am confused because I'm mixing probability masses with densities.

Note: Imagine that I don't have the joint distribution of $(Y,V)$. I can only use the distribution of $Y$ conditional on $V$ and the distribution of $V$.

Answer 1

Suppose that $Y$ and $V$ are defined on a common probability space $(\Omega, \mathscr{F}, P)$, and $V$ has density $f$ with respect to Lebesgue measure, then your conjecture is true. In fact, whether $Y$ is discrete or not is unessential for the proof, as the formula you listed is a special case of a more general relation $(3)$ to be shown below.

For any $A \in \mathscr{F}$, by the definition of conditional probability, it holds that \begin{align} P(A) = \int_\Omega P(A | V)dP, \tag{1} \end{align} where "$P(A|V)$" is a shorthand for the standard sub-$\sigma$-field-based conditional probability notation $P(A|\sigma(V))$. Also as a part of the definition of conditional probability, $P(A|V)$ is $\sigma(V)$-measurable, implying that there exists a Borel function $g: \mathbb{R} \to \mathbb{R}$ such that $P(A|V) = g(V)$. The change of variable theorem then gives \begin{align} \int_\Omega P(A|V)dP = \int_\Omega g(V(\omega))P(d\omega) = \int_\mathbb{R}g(v)f(v)dv. \tag{2} \end{align} By convention, the function $g(v)$ is usually denoted by $P(A|V = v)$. Technically speaking, the notation makes sense because for any $v \in \mathbb{R}$, the random variable $P(A | V)$ is almost surely a constant on the set $\{\omega: V(\omega) = v\}$. In other words, on this $\sigma(V)$-set, $P(A|V)_\omega$ almost surely equals to a constant that depends only on $v$, which is therefore reasonable to be denoted by $g(v)$ (for more discussions on how to justify the "$P(A|V = v)$" notation rigorously under the measure-theoretic probability, refer to Exercises 33.4, 33.5, 33.6 in Probability and Measure by Patrick Billingsley).

Combining $(1)$ and $(2)$ gives \begin{align} P(A) = \int_\mathbb{R}P(A|V = v)f(v)dv. \tag{3} \end{align}

Now take $A = \{Y = y\} = \{\omega: Y(\omega) = y\} \in \mathscr{F}$ in $(3)$ finishes the proof of your conjecture.

Answer 2

Let me offer a (long) intuitive explanation without entering into measure-theoretic arguments.

The main problem is thus how to make sense of the conditional probability $P(Y = y| V=x)$ when $V$ is an (absolutely) continuous random variable and for which $P(V=x)=0$. First of all, rest assured that such a probability exists and makes sense. Here is a practical example.

Example. Suppose that a real number $V$ is selected at random, with density $f$. If $V$ takes value $x$, a coin with probability $g(x)$ is tossed $(0 \leq g(x) \leq 1)$. It is thus natural to assert that the conditional probability of obtaining a head, given $V=x$, is $g(x)$. But $V$ is absolutely continuous, so $\{ V=x \}$ has probability 0, thus the conditional probabilities do exist but they need to be defined. In the explanation below I'll use this example to make my argument more concrete.

An intuitive but less rigorous way to define the overall probability of obtaining a head is by the following infinitesimal argument. The probability that $V$ will fall into the interval $(x,x+dx]$ is approximately $f(x)dx$. Given that $V$ falls into this interval, the probability of a head is roughly $g(x)$. So from the law of total probability, we expect that the probability of a head will be $\sum_x g(x) f(x) dx$, which approximates $\int_{-\infty}^{\infty}g(x)f(x)dx$. Thus the probability in question is nothing but a weighted average of conditional probabilities, the weights being assigned in accordance with the density $f$.

Now let's look a bit more closely at what's happening here. We have two random variables $Y$ and $V$ [$Y=y$, $V$ = (say) the number of heads obtained]. We are specifying the density of $V$ and for each $x$ and each Borel set $B$, we are specifying a quantity $P_x(B)$ that is to be interpreted intuitively as the conditional probability that $V\in B$ given that $Y=x$; a longer notation is $P_x(B) = P\{V\in B | Y = x\}$.

We are trying to conclude that the probabilities of tall events involving $Y$ and $V$ are now determined. Suppose that $C$ is a two-dimensional Borel set. What is a reasonable figure for $P\{(Y, V)\in C\}$ ? Intuitively, the probability that $Y$ falls into $(x,x+dx]$ is $f_Y(x)dx$. Given that this happens, that is, roughly given $Y=x$, the only way $(Y,V)$ can lie in $C$ is if $Y$ belongs to the "section" $C_x = \{y: (x,y)\in C\}$ (as in the left figure). But this happens with probability $P_x(C_x)$. Thus we expect that the total probability that $(Y,V)$ will belong to $C$ is

$$ \int_{-\infty}^{\infty} P_x(C_x)f_Y(x)dx. $$

In particular, if $C = A\times B = \{(x,y): x\in A, y\in B\}$ as in the Figure on the right, $C_x = \emptyset$ if $x\not \in A$, $C_x = B$ if $x\in A$. Thus

$$ P\{(Y,V)\in C\} = P(Y\in A, V \in B) = \int_A P_x(B)f_Y(x)dx. $$

Now this reasoning may be formalized by letting the sample space be $\Omega = \mathbb{R}^2$, the Borel subsets be $\mathcal{F}$ $Y(x,v) = y$, $V(x,v) = v$ and letting $f_Y$ be the density function on $\mathbb{R}$. Suppose that for each real $x$ we are given a probability measure $P_x$ on Borel subsets of $\mathbb{R}$ and assume also that $P_x(B)$ is a piecewise continuous function of $x$, for each fixed $B$. Then it turns out that there is a unique probability measure $P$ on $\mathcal F$ such that for all Borel subsets $A, B$ of $\mathbb{R}$

$$ P(A\times B) = \int_A P_x(B) f_Y(x)dx.\tag{*} $$

The requirement (*), which can be regarded as a continuous version of the law of total probability, determines $P$ uniquely. In fact, if $C\in \mathcal{F}$, $P(C)$ is given by

$$ P(C) = \int_{-\infty}^{\infty} P_x(C_x)f_Y(x)dx.\tag{**} $$

Now if $Y(x,y) = x, V(x,y) = y$, then $$ P(A\times B) = P(Y\in A, V\in B) $$ and

$$P(C) = P\{(Y,V)\in C\}.$$

Furthermore, the distribution function of $Y$ is given by

$$ F_Y(x_0) = P(Y\leq x_0) = P\{X \in A, V \in B\} = \int_{A}P_x(B)f_Y(x)dx = \int_{-\infty}^{x_0} f_Y(x)dx, $$ where $A = (-\infty, x_0]$ and $B = (-\infty, \infty)$. Furthermore,

$$ P(V \in B) = P\{Y\in A, V\in B\} $$ where $A = (-\infty, \infty)$, hence

$$ P(Y\in B) = \int_{\infty}^{\infty} P_x(B)f_Y(x)dx. $$

So to summarize: If we start out with a density for $Y$ and a set of probabilities $P_x(B)$ that we interpret as $P(Y\in B| V=x)$, the probabilities of events of the form $\{(Y, V)\in C\}$ are determined in a natural way if you believe that there should be a continuous version of the law of total probability; $P\{(Y, V)\in C\}$ is given by (**), which reduces to (*), in the special case when $C = A\times B$.

Example (Cont'd). If $Y$ has density $f_Y$, and a coin with probability of heads $g(x)$ is tossed whenever $Y = x$ (suppose a head corresponds to $V=1$), then the probability of obtaining a head is

$$ P(V = 1) = \int_{-\infty}^{\infty} P(V=1 | Y=x) f_Y(x)dx = \int_{-\infty}^{\infty} g(x) f_Y(x)dx, $$ in agreement with the previous intuitive argument.

Answer 3

Let $(\Omega,\mathcal F,P)$ be a probability space and $V$ a continuous random variable having density $f$ defined on the space and $A$ some element of $\mathcal F$.

Then actually the question is: what is the function prescribed by:$$v\mapsto P(A|V=v)$$?????...

Because events like $\{V=v\}$ have probability $0$ we cannot use the usual definition stating that $PA\mid C)=P(A\cap C)/P(C)$.

Further we want this to be a function that satisfies:$$P(A)=\int P(A\mid V=v)f(v)dv\tag1$$

Here the Radon-Nikodym derivative comes in.

Let $\nu$ denote the measure on $\sigma(V)$ prescribed by:$$B\mapsto P(A\cap B)$$Then - if $Q$ is the restriction of $P$ on $\sigma(V)$ - measure $\nu$ is absolutely continuous with respect to $Q$ (i.e. $Q(B)=0$ implies that $\nu(B)=0$).

Then according to the theorem of Radon-Nikodym a $\sigma(V)$-measurable random variable $Z$ exists with:$$\nu(B)=\int_BZdQ=\mathbb E1_BZ\text{ for every }B\in\sigma(V)$$The fact that $Z$ is $\sigma(V)$-measurable implies that a Borel measurable function $g$ exists such that $Z=g(V)$ which makes us - in special case $B=\Omega$ - arrive at:$$P(A)=\nu(\Omega)=\mathbb Eg(V)=\int g(v)f(v)dv$$So this $g$ is a suitable candidate for satisfaction of $(1)$ and we take the liberty to define it is as $P(A|V=v)$.

Well defined?... Well, not exactly because in general there are more candidates and we just picked one out. Fortunately though it can be proved that two candidates at most differ on a $\sigma(V)$-measurable $P$-null set.

This of course works for every event $A$ so also for events $\{Y=y\}$ where $Y$ is some other random variable on the space. Doing so we find:$$P(Y=y)=\int P(Y=y\mid V=v)f(v)dv$$