9
$\begingroup$

Bayes theorem is defined for both discrete variables in terms of probabilities and continuous variables in terms of densities. If random variables $X,Z$ are jointly distributed, with $f_X(x)$ continuous density of $X$ and $p_Z(z)$ the discrete probability mass at $Z=z$, does Bayes theorem hold in the sense that $$p_{Z|X}(z) = \frac{f_{X|Z}(x)p_Z(z)}{f_X(x)}?$$ If not, is there an analogue for such discrete-continuous mixtures?

$\endgroup$
1
  • 6
    $\begingroup$ There is no distinction between discrete and continuous when you move to a measure-theoretic view of integration, which makes questions like this impossible to ask (not a criticism of your question, most practitioners don't know this!) $\endgroup$
    – jbowman
    Feb 5, 2020 at 17:56

2 Answers 2

14
$\begingroup$

Yes.

A joint distribution $f_{X,Z}(x, z)$ of continuous variable $X \sim f_X$, and discrete variable $Z \sim p_Z$, is defined as any non-negative function of $x$ and $z$ that satisfies

$$ \int f_{X,Z}(x, z) dx = p_Z(z), $$

$$ \sum_z f_{X,Z}(x, z) = f_X(x). $$

For a given distribution $f_{X,Z}$, the conditional distributions are defined:

$$ p_{Z \mid X}(z) \equiv \frac{f_{X,Z}(x, z)}{f_X(x)}, $$ and

$$ f_{X \mid Z}(x) \equiv \frac{f_{X,Z}(x, z)}{p_Z(z)}. $$

Note that both expressions satisfy the proper unity condition when you apply the sum or integral from earlier.

The mixed form of Bayes theorem can be obtained simply by rearranging the above formulas for the conditional distribution. Rearranging the second equation for $f_{X,Z}(x, z)$ and substituting the result into the first equation, you get,

$$ p_{Z \mid X}(z) = \frac{f_{X \mid Z}(x) p_Z(z)}{f_X(x)}. $$

$\endgroup$
1
  • $\begingroup$ Thanks for explaining this without measure theory! $\endgroup$
    – tomka
    Feb 5, 2020 at 19:47
3
$\begingroup$

[This is just rough intuition, avoiding measure theory]

For continuous random variables ${ X }$ and ${ Y ,}$ their joint density ${ f _{X, Y} }$ (if it exists) is a map ${ f _{ {\color{purple}{X}}, {\color{blue}{Y}} } : {\color{purple}{\mathbb{R}}} \times {\color{blue}{\mathbb{R}}} \to \mathbb{R} _{\geq 0} }$ such that probability of any event ${ (X,Y) \in [a,b] \times [c,d] }$ is the volume under graph of ${ f _{X,Y} }$ and over ${ [a,b] \times [c,d] }.$

${ \mathbb{P}(X \in [a,b], Y \in [c,d]) }$ ${ = \int \int _{[a,b] \times [c,d]} f _{X,Y} (x,y) \, dx \, dy }$

Intuitively, ${ \mathbb{P}(X \in [x, x + \Delta x], Y \in [y, y + \Delta y]) }$ ${ \approx f _{X,Y} (x, y) \Delta x \Delta y }$ for small ${ \Delta x, \Delta y \gt 0 }$ (assuming continuity of ${ f _{X,Y} }$ at ${ (x,y) }$).

Similarly for a discrete random variable ${ X }$ with range ${ \lbrace x _1, x _2, \ldots \rbrace }$ and a continuous random variable ${ Y },$ their joint density ${ f _{X, Y} }$ (if it exists) is a map ${ f _{ {\color{purple}{X}}, {\color{blue}{Y}} } : {\color{purple}{\lbrace x _1, x _2, \ldots \rbrace}} \times {\color{blue}{\mathbb{R}}} \to \mathbb{R} _{\geq 0} }$ such that probability of any event ${ (X, Y) \in \lbrace x _i \rbrace \times [c,d] }$ is the area under graph of ${ f _{X,Y} }$ and over ${ \lbrace x _i \rbrace \times [c,d] }.$

${ \mathbb{P}(X = x _i, Y \in [c,d]) }$ ${ = \int _{[c,d]} f _{X, Y} (x _i, y) \, dy }$

Intuitively, ${ \mathbb{P}(X = x _i, Y \in [y, y + \Delta y]) }$ ${ \approx f _{X, Y} (x _i, y) \Delta y }$ for small ${ \Delta y \gt 0 }$ (assuming continuity of ${ f _{X,Y} (x _i, \cdot) }$ at ${ y }$).

Eg: From this, in the ${ X }$ discrete and ${ Y }$ continuous case:
[Marginals] PMF ${ \mathbb{P}(X = x _i) }$ ${ = \int f _{X,Y} (x _i, y) dy .}$ Also ${ \mathbb{P}(Y \in [y, y + \Delta y]) }$ ${ = \sum _i \mathbb{P}(X = x _i, Y \in [y, y + \Delta y]) }$ ${ \approx \sum _{i} f _{X,Y} (x _i, y) \Delta y}$ suggesting ${ f _Y (y) = \sum _{i} f _{X,Y} (x _i, y) .}$
[Conditionals] Intuitively, similar to how ${ f _Y (y) \Delta y \approx \mathbb{P}(Y \in [y, y + \Delta y]) },$ conditional density ${ f _{Y \vert X } ( \cdot \vert x _i) }$ is such that ${ f _{Y \vert X} (y {\color{red}{\vert x _i)}} \Delta y }$ ${ \approx \mathbb{P}(Y \in [y, y + \Delta y] {\color{red}{\vert X = x _i)}} }.$ So ${ f _{Y \vert X} (y \vert x _i) \Delta y }$ ${ \approx \frac{f _{X,Y} (x _i, y) \Delta y}{\int f _{X,Y} (x _i, y) dy} }$ suggesting ${ f _{Y \vert X} (y \vert x _i) }$ ${ = \frac{f _{X,Y} (x _i, y)}{\int f _{X,Y} (x _i, y) dy} .}$
Intuitively, conditional PMF ${ p _{X \vert Y} (\cdot \vert y) }$ is such that ${ p _{X \vert Y} (x _i \vert y) }$ is limit of ${ \mathbb{P}(X = x _i \vert Y \in [y, y + \Delta y]) }$ as ${ \Delta y \to 0 ^{+} }.$ But ${ \mathbb{P}(X = x _i \vert Y \in [y, y + \Delta y]) }$ ${ \approx \frac{f _{X,Y} (x _i, y) \Delta y }{ \sum _i f _{X,Y} (x _i, y) \Delta y} }$ suggesting ${ p _{X \vert Y} (x _i \vert y) }$ ${ = \frac{ f _{X,Y} (x _i, y)}{\sum _i f _{X,Y} (x _i, y) }. }$

To summarise, marginals are ${ p _{X} (x _i) = \int f _{X,Y} (x _i, y) \, dy }$ and ${ f _Y (y) = \sum _i f _{X,Y} (x _i, y) },$ and conditionals are ${ f _{Y \vert X} (y \vert x _i) = \frac{f _{X,Y}(x _i, y)}{p _{X} (x _i)} }$ and ${ p _{X \vert Y} (x _i \vert y) = \frac{f _{X,Y} (x _i, y)}{f _Y (y)} .}$

Especially ${ X, Y }$ are independent implies ${ f _{Y \vert X} (y \vert x _i) = f _Y (y) }$ i.e. ${ f _{X,Y} (x _i, y) = p _{X} (x _i) f _Y (y) },$ and vice versa. We also have Bayes rule ${ f _{Y \vert X} (y \vert x _i) }$ ${ = \frac{p _{X \vert Y} (x _i \vert y) f _Y (y) }{p _X (x _i)} }.$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.