# Law of total probability and conditioning on multiple events

Pearl et al. "Causal Inference in Statistics: A Primer" (2016) p. 56-57 includes the following equations (I have omitted a subscript $$_m$$ to $$P$$ as it plays no role in my question): \begin{align} &P(Y=y|X=x)= \\ & \sum_{z} P(Y=y|X=x,Z=z)P(Z=z|X=x) \tag{3.3}=\\ & \sum_{z} P(Y=y|X=x,Z=z)P(Z=z). \tag{3.4} \end{align}

The first equality is said to be obtained using the law of total probability; the second is said to use independence between $$Z$$ and $$X$$. Let me also add $$P(Y=y|X=x) =\\ \sum_{z} P(Y=y|X=x,Z=z)P(Z=z) \tag{3.5}$$ just to have a reference for this particular equality.

I am a little uncomfortable with the equations $$(3.3)$$ and $$(3.4)$$. Here is how I approached the problem myself. If conditioning on $$X=x$$ were omitted, we would have $$P(Y=y) = \sum_{z} P(Y=y|Z=z)P(Z=z)$$ and that would be a straightforward application of the law of total probability. But we have the conditioning on $$X=x$$. First, let us treat $$Y=y|X=x$$ as an event $$A\color{blue}{^*}$$ and then $$P(A) = \sum_{z} P(A|Z=z)P(Z=z)$$ thus directly producing eq. $$(3.5)$$ when $$Y=y|X=x$$ is substituted back for $$A$$. On the other hand, we could partition $$Y=y$$ by $$Z$$ first and only then condition on $$X=x$$, thus $$P(Y=y|X=x) = \\ \sum_{z} P(\color{red}{(}Y=y|Z=z\color{red}{)}|X=x)P(Z=z|X=x),$$ i.e. eq. $$(3.4)$$. It appears to me at this point that I got eq. $$(3.4)$$ without invoking independence between $$Z$$ and $$X$$, but probably I am missing something.

Questions

1. Did I indeed prove eq. $$(3.4)$$ without invoking independence between $$Z$$ and $$X$$? If not, what did I miss?
2. Are there cases where eq. $$(3.5)$$ does not hold? (Perhaps if $$Z$$ and $$X$$ are dependent?) If so, why?

$$\color{blue}{^*}$$Pohoua noted that one cannot define an event like this. Let us just define $$A:=(Y=y|X=x)$$ without calling it an event. Not that that should make the following expressions correct, but at least an incorrect use of a term is avoided.

• This seems relevant: stats.stackexchange.com/questions/406950/… May 11, 2020 at 13:06
• I edited to change the layout of some (long) formulas, since, at least on my screen, they were put in scroll boxes, making them difficult to read. May 11, 2020 at 13:14
• Thank you, @kjetilbhalvorsen. May 11, 2020 at 13:15

\begin{align} P(Y=y|X) &=E(1_{Y=y}|X) \\ &\overset{Tower\ property}{=}E\color{green}{\bigg(}E\color{red}{(}1_{Y=y}|X \color{red}{)}|(X,Z)\color{green}{\bigg)} \\ &\overset{Tower\ property}{=}E\color{green}{\bigg(}E\color{red}{(}1_{Y=y}|(X,Z) \color{red}{)}|X\color{green}{\bigg)} \\ &= E\color{green}{\bigg(}g(X,Z) |X\color{green}{\bigg)} \\ &= \sum_{z} g(X,Z=z) P(Z=z|X) \\ &= \sum_{z} E\color{red}{(}1_{Y=y}|(X,Z=z) \color{red}{)} P(Z=z|X) \\ &= \sum_{z} P\color{red}{(}Y=y|(X,Z=z) \color{red}{)} P(Z=z|X) \end{align}

So \begin{align} P(Y=y|X)= \sum_{z} P\color{red}{(}Y=y|(X,Z=z) \color{red}{)} P(Z=z|X) \end{align} an hence \begin{align} P(Y=y|X=x)= \sum_{z} P\color{red}{(}Y=y|(X=x,Z=z) \color{red}{)} P(Z=z|X=x) \end{align}

Detail:Tower Property Conditional_expectation

For sub-σ-algebras $$\mathcal H_{1} \subset H_{2} \subset \mathcal F$$ we have $$E(E(Y\mid \mathcal H_{2})\mid \mathcal H_{1})=E(E(Y \mid \mathcal H_{1})\mid \mathcal H_{2})=E(Y\mid \mathcal H_{1})$$.

In this situation $$\mathcal H_{1}=\sigma(X) \subset \mathcal H_{2}=\sigma(X,Z)$$

so

$$E(E(Y\mid \sigma(X,Z))\mid \sigma(X))=E(E(Y \mid \sigma(X))\mid \mathcal \sigma(X,Z))=E(Y\mid \sigma(X))$$

• Thank you for your answer! I am currently unable to focus on this thread, but I will return to it (hopefully this week) and then will either upvote or ask for further clarification. Feel free to remind me if I do not. Apr 22, 2020 at 8:41
• @Richard Hardy , Feel free. It was a good question and enjoy it. it is enough . Apr 22, 2020 at 15:52
• I wonder how you got $$E\color{green}{\bigg(}g(X,Z) |X\color{green}{\bigg)} = \sum_{z} g(X,Z=z) P(Z=z|X).$$ I also wonder how you got $$E(E(Y\mid \mathcal H_{2})\mid \mathcal H_{1})=E(E(Y \mid \mathcal H_{1})\mid \mathcal H_{2}).$$ May 11, 2020 at 13:23
• $E(E(Y\mid \mathcal H_{2})\mid \mathcal H_{1})=E(Y \mid \mathcal H_{1})$ By Tower property. $E(\color{red}{E(Y \mid \mathcal H_{1})}\mid \mathcal H_{2})=\color{red}{E(Y \mid \mathcal H_{1})}$ because $\mathcal H_{1} \subset \mathcal H_{2}$ (Stability property since $E(Y \mid \mathcal H_{1})$ is $\mathcal H_{2}$measurable). May 11, 2020 at 14:18
• So the expectation is taken over the distribution of $z$? These expectations get confusing when having to guess what is being averaged over, but you have made it clearer now. May 11, 2020 at 14:36

Your purported proof of $$(3.4)$$, without using independent of $$Z$$ and $$X$$, is not correct. It is not valid to form an event that includes a condition, because that condition then escapes the other probability operator in the law of total probability. In fact, the equation is not true in general (i.e., without the independence condition), as can be seen by considering the counterexample with joint mass function:

$$\mathbb{P}(X=x,Y=y,Z=z) = \begin{cases} \tfrac{1}{2} & & \text{if } x = 0, y = 0, z = 1, \\[6pt] \tfrac{1}{2} & & \text{if } x = 1, y = 1, z = 0, \\[6pt] 0 & & \text{otherwise}. \\[6pt] \end{cases}$$

In this case, we have:

$$1 = \mathbb{P}(Y=x|X=x) \neq \sum_z \mathbb{P}(Y=x|X=x, Z=z) \cdot \mathbb{P}(Z=z) = \tfrac{1}{2}.$$

• Thank you, Ben. May 11, 2020 at 14:27

I think there is a mistake in your proof when you define the event $$A$$ as $$Y=y|X=x$$, this definition does not make sense. You cannot include conditionality in an event (what would be a realization of such an event?), you can just talk about probability of an event conditionally to some other event. Conditioning on an event $$X=x$$ defines new probability measures, but does not define new events.

The proof of equation $$(3.3)$$ is just the application of the law of total probability, to which you add a conditionality on $$X=x$$ at every probability (it's the law of total probability applied to the probability measure $$P(.|X=x)$$).

Then you need independence to say that the law of $$Z$$ and the law of $$Z$$ conditionally on $$X=x$$ are the same.

Here is an example were $$X$$ and $$Z$$ are not independent. $$X$$ is the choice (with probabilities $$1/2$$) of a coin between a fair one and a biased one with two tails, $$Y$$ is the result of a toss of the chosen coin, and $$Z=Y$$. Then equation $$(3.5)$$ does not hold. $$P(Y= tail | X= biased) =1$$ and \begin{aligned} &\sum_z P(Y=tail|X=biased , Z=z) P(Z=z) \\ &= P(Y= tail| X=biased, Z= tail) P(Z=tail) \\ &+ P(Y = tail|X= biased, Z= head)P(Z= head) \\ &= 1\times P(Y = tail) + 0 \\ &=3/4 \end{aligned} using that $$Z=Y$$.

I hope this helps.

• Yes it should. I struggled with a very similar problem not so long ago, conditional probability may be quite tricky! Apr 7, 2020 at 8:26
• It really is, especially since I had hardly encountered similar exercises before. The probability courses I once took did not seem to offer many (any?) examples of this kind, and the probability section of Pearl's textbook does not consider them either. Apr 7, 2020 at 8:28
• Thank you for your answer! (+1) May 13, 2020 at 9:18