Iterated expectations theory

Question

I'm a bit stuck on the general concept and calculations of  iterated expectations. A simple example:

$$E[E[Y|X]] = E[Y]$$

I'm not sure how or why this is the case? I have a 3 line proof in my notes using densities and stuff but is there no 'quicker' way. Especially in longer cases like (below) writing everything out is difficult. Are there tricks to spot? 

Example 2:

$Var[Y] = E[Y^2] - (E[Y])^2$ (I get this line!) $$= E[E[Y^2|X]] - \bigg(E[E[Y|X]]\bigg)^2 $$ $$= E\left[Var[Y|X]+(E[Y|X])^2\right] - \bigg(E[E[Y|X]]\bigg)^2$$ $$= E[Var[Y|X]] + E[(E[Y|X])^2] - \bigg(E[E[Y|X]]\bigg)^2$$ $$= E[Var[Y|X]] + Var[E[Y|X]]$$

I'm not at all sure how they go from line to line.

Answer 1

This is a simple, but not so simple concept to understand. I personally find using the example of a table to illustrate what is going on. So we have a $2\times 2$ table with counts in each cell. To keep from the "abstract" nature of the concept, I'll use real numbers instead of letters. So we have a table of "being sick" against "having a sore throat":

$$\begin{array}{c|c} & \text{Sick} & \text{Not Sick}\\ \hline \text{Sore throat} & 12 & 5 \\ \hline \text{No Sore throat} & 4 & 55 \end{array}$$

So in this table we have 12 people who are both sick and have a sore throat, and 5 people who are not sick but still have a sore throat. Now if we condition on "Sore throat", then the chances of being sick are

$$Pr(\text{Sick}|\text{sore throat})=\frac{12}{12+5}=0.706$$

You can think of this quantity as a part of $E[Y|X]$ in the more abstract sense (not exactly the case, but will do for understanding the concept). Now We can calculate the same probability, but on the condition of no sore throat

$$Pr(\text{Sick}|\text{No sore throat})=\frac{4}{4+55}=0.068$$

This quantity is the "other part" of $E[Y|X]$. No suppose you want to calculate the chance of someone being sick, independently of their throat. We can do this (at least) two ways. The simplest in this case is to calculate the total proportion of people who are sick:

$$Pr(\text{Sick})=\frac{4+12}{4+12+55+5}=0.210$$

But You can also use the law of iterated expectations. It is more cumbersome here, but can be much simpler in more complicated problems. We have, by the product rule and sum rule:

$$Pr(\text{Sick})=Pr(\text{Sick}|\text{Sore throat})Pr(\text{Sore throat})$$ $$+Pr(\text{Sick}|\text{No Sore throat})Pr(\text{No Sore throat})$$

Now we need the marginal probabilities $Pr(\text{No Sore throat})=\frac{4+55}{4+12+55+5}=0.776$ and $Pr(\text{Sore throat})=\frac{12+5}{4+12+55+5}=0.224$. Now we have all the ingredients, we just plug them in:

$$Pr(\text{Sick})=0.706 \times 0.224 + 0.068 \times 0.776=0.210$$

Which is the same result. This is an "empirical proof" so to speak. The law simply say that it holds more generally, for arbitrary distributions.

The variance version comes about by "adding zero" to the equations, but as $0=(E[Y|X])^{2}-(E[Y|X])^{2}$, or by noting that $Var[Y|X]=E[Y^{2}|X]-(E[Y|X])^{2}$, which can be re-arranged to give $E[Y^{2}|X]=Var[Y|X]+(E[Y|X])^{2}$. It's just messy because of all the parenthesis, and remembering where the "square" is