Does Bayes theorem apply to joint distributions of discrete and continuous random variables?

Question

Bayes theorem is defined for both discrete variables in terms of probabilities and continuous variables in terms of densities. If random variables $X,Z$ are jointly distributed, with $f_X(x)$ continuous density of $X$ and $p_Z(z)$ the discrete probability mass at $Z=z$, does Bayes theorem hold in the sense that $$p_{Z|X}(z) = \frac{f_{X|Z}(x)p_Z(z)}{f_X(x)}?$$ If not, is there an analogue for such discrete-continuous mixtures?

Answer 1

Yes.

A joint distribution $f_{X,Z}(x, z)$ of continuous variable $X \sim f_X$, and discrete variable $Z \sim p_Z$, is defined as any non-negative function of $x$ and $z$ that satisfies

$$ \int f_{X,Z}(x, z) dx = p_Z(z), $$

$$ \sum_z f_{X,Z}(x, z) = f_X(x). $$

For a given distribution $f_{X,Z}$, the conditional distributions are defined:

$$ p_{Z \mid X}(z) \equiv \frac{f_{X,Z}(x, z)}{f_X(x)}, $$ and

$$ f_{X \mid Z}(x) \equiv \frac{f_{X,Z}(x, z)}{p_Z(z)}. $$

Note that both expressions satisfy the proper unity condition when you apply the sum or integral from earlier.

The mixed form of Bayes theorem can be obtained simply by rearranging the above formulas for the conditional distribution. Rearranging the second equation for $f_{X,Z}(x, z)$ and substituting the result into the first equation, you get,

$$ p_{Z \mid X}(z) = \frac{f_{X \mid Z}(x) p_Z(z)}{f_X(x)}. $$

Answer 2

[This is just rough intuition, avoiding measure theory]

For continuous random variables ${ X }$ and ${ Y ,}$ their joint density ${ f _{X, Y} }$ (if it exists) is a map ${ f _{ {\color{purple}{X}}, {\color{blue}{Y}} } : {\color{purple}{\mathbb{R}}} \times {\color{blue}{\mathbb{R}}} \to \mathbb{R} _{\geq 0} }$ such that probability of any event ${ (X,Y) \in [a,b] \times [c,d] }$ is the volume under graph of ${ f _{X,Y} }$ and over ${ [a,b] \times [c,d] }.$

${ \mathbb{P}(X \in [a,b], Y \in [c,d]) }$ ${ = \int \int _{[a,b] \times [c,d]} f _{X,Y} (x,y) \, dx \, dy }$

Intuitively, ${ \mathbb{P}(X \in [x, x + \Delta x], Y \in [y, y + \Delta y]) }$ ${ \approx f _{X,Y} (x, y) \Delta x \Delta y }$ for small ${ \Delta x, \Delta y \gt 0 }$ (assuming continuity of ${ f _{X,Y} }$ at ${ (x,y) }$).

Similarly for a discrete random variable ${ X }$ with range ${ \lbrace x _1, x _2, \ldots \rbrace }$ and a continuous random variable ${ Y },$ their joint density ${ f _{X, Y} }$ (if it exists) is a map ${ f _{ {\color{purple}{X}}, {\color{blue}{Y}} } : {\color{purple}{\lbrace x _1, x _2, \ldots \rbrace}} \times {\color{blue}{\mathbb{R}}} \to \mathbb{R} _{\geq 0} }$ such that probability of any event ${ (X, Y) \in \lbrace x _i \rbrace \times [c,d] }$ is the area under graph of ${ f _{X,Y} }$ and over ${ \lbrace x _i \rbrace \times [c,d] }.$

${ \mathbb{P}(X = x _i, Y \in [c,d]) }$ ${ = \int _{[c,d]} f _{X, Y} (x _i, y) \, dy }$

Intuitively, ${ \mathbb{P}(X = x _i, Y \in [y, y + \Delta y]) }$ ${ \approx f _{X, Y} (x _i, y) \Delta y }$ for small ${ \Delta y \gt 0 }$ (assuming continuity of ${ f _{X,Y} (x _i, \cdot) }$ at ${ y }$).

Eg: From this, in the ${ X }$ discrete and ${ Y }$ continuous case:
[Marginals] PMF ${ \mathbb{P}(X = x _i) }$ ${ = \int f _{X,Y} (x _i, y) dy .}$ Also ${ \mathbb{P}(Y \in [y, y + \Delta y]) }$ ${ = \sum _i \mathbb{P}(X = x _i, Y \in [y, y + \Delta y]) }$ ${ \approx \sum _{i} f _{X,Y} (x _i, y) \Delta y}$ suggesting ${ f _Y (y) = \sum _{i} f _{X,Y} (x _i, y) .}$
[Conditionals] Intuitively, similar to how ${ f _Y (y) \Delta y \approx \mathbb{P}(Y \in [y, y + \Delta y]) },$ conditional density ${ f _{Y \vert X } ( \cdot \vert x _i) }$ is such that ${ f _{Y \vert X} (y {\color{red}{\vert x _i)}} \Delta y }$ ${ \approx \mathbb{P}(Y \in [y, y + \Delta y] {\color{red}{\vert X = x _i)}} }.$ So ${ f _{Y \vert X} (y \vert x _i) \Delta y }$ ${ \approx \frac{f _{X,Y} (x _i, y) \Delta y}{\int f _{X,Y} (x _i, y) dy} }$ suggesting ${ f _{Y \vert X} (y \vert x _i) }$ ${ = \frac{f _{X,Y} (x _i, y)}{\int f _{X,Y} (x _i, y) dy} .}$
Intuitively, conditional PMF ${ p _{X \vert Y} (\cdot \vert y) }$ is such that ${ p _{X \vert Y} (x _i \vert y) }$ is limit of ${ \mathbb{P}(X = x _i \vert Y \in [y, y + \Delta y]) }$ as ${ \Delta y \to 0 ^{+} }.$ But ${ \mathbb{P}(X = x _i \vert Y \in [y, y + \Delta y]) }$ ${ \approx \frac{f _{X,Y} (x _i, y) \Delta y }{ \sum _i f _{X,Y} (x _i, y) \Delta y} }$ suggesting ${ p _{X \vert Y} (x _i \vert y) }$ ${ = \frac{ f _{X,Y} (x _i, y)}{\sum _i f _{X,Y} (x _i, y) }. }$

To summarise, marginals are ${ p _{X} (x _i) = \int f _{X,Y} (x _i, y) \, dy }$ and ${ f _Y (y) = \sum _i f _{X,Y} (x _i, y) },$ and conditionals are ${ f _{Y \vert X} (y \vert x _i) = \frac{f _{X,Y}(x _i, y)}{p _{X} (x _i)} }$ and ${ p _{X \vert Y} (x _i \vert y) = \frac{f _{X,Y} (x _i, y)}{f _Y (y)} .}$

Especially ${ X, Y }$ are independent implies ${ f _{Y \vert X} (y \vert x _i) = f _Y (y) }$ i.e. ${ f _{X,Y} (x _i, y) = p _{X} (x _i) f _Y (y) },$ and vice versa. We also have Bayes rule ${ f _{Y \vert X} (y \vert x _i) }$ ${ = \frac{p _{X \vert Y} (x _i \vert y) f _Y (y) }{p _X (x _i)} }.$