An algebra of the joint and conditional 'operators'

Question

In treatments of Bayesian methods you typically see terms combining joint distributions with conditional distributions such as $P(A,B|C)$ and $P(A|B,C)$. Expressions arise such as the chain rule $$ P(A,C|B) = P(A|B,C) P(C|B). $$

I find I'm often caught out by such manipulations. For example, moving the $C$ from the LHS of the conditional sign to the RHS we need to multiply by $P(C|B)$, not by $P(C)$.

To tune my intuition, what I would like to see is an algebra of the joint and conditional 'operators' (comma and mid) but try as I might I haven't been able to find such an algebra or derive it myself from first principles. Does such an algebra exist?

Answer 1

All the algebra you need is the chain rule.

$$ P(X, Y) = P(X | Y) P(Y) = P(Y | X) P(X) $$ Here $X$ and $Y$ may mean random events that are composed of other random events. The same is true for densities. Also, Bayes theorem is a corollary of this formula:

$$ P(X|Y) = \frac{P(Y|X) P(X)}{P(Y)} = \frac{P(X, Y)}{P(Y)} $$

From the chain rule you can derive $$ P(A, C | B) = \frac{P(A, \overbrace{C, B}^{Y})}{P(B)} = \frac{P(A | B,C) P(B, C)}{P(B)} = \frac{P(A | B, C) P(C|B) P(B)}{P(B)} = P(A|B,C) P(C|B) $$

You can treat this as the Conditioning Operator Rule: if you have 2 sets of random events $A$ and $B$ and you want to "split" them, you should do it accounting for possible dependence: one term's condition box should contain another (But not both). Also, you keep your conditions: in general case you can't drop it (unless you have independence). $$ P(A, B | C) = P(A | B, C) P(B | C) $$ (this can be viewed as a generalization of the first formula. Indeed, if you set $C = \Omega$ (the set of all outcomes), you can derive it)