# Iterated expectations theory

I'm a bit stuck on the general concept and calculations of  iterated expectations. A simple example:

$$E[E[Y|X]] = E[Y]$$

I'm not sure how or why this is the case? I have a 3 line proof in my notes using densities and stuff but is there no 'quicker' way. Especially in longer cases like (below) writing everything out is difficult. Are there tricks to spot?

Example 2:

$Var[Y] = E[Y^2] - (E[Y])^2$ (I get this line!) $$= E[E[Y^2|X]] - \bigg(E[E[Y|X]]\bigg)^2$$ $$= E\left[Var[Y|X]+(E[Y|X])^2\right] - \bigg(E[E[Y|X]]\bigg)^2$$ $$= E[Var[Y|X]] + E[(E[Y|X])^2] - \bigg(E[E[Y|X]]\bigg)^2$$ $$= E[Var[Y|X]] + Var[E[Y|X]]$$

I'm not at all sure how they go from line to line.

• This question could benefit from clarification. What do you actually want? You have a proof of $E[E[Y|X]] = E[Y]$, but it is not possible for us to know what "densities and stuff" means. It sounds to me like you want to understand without taking the time to understand (the 'quick' way). And what is the problem with Example 2? The line by line argument is "use $E[E[Y|X]] = E[Y]$". Do you have conceptual problems with the concepts of expectations and conditional expectations? Or is there a particular step in the arguments that you don't understand? – NRH Jun 4 '11 at 22:50
• No tricks; just recognize where they've used the identity $E[Y^2] = Var[Y^2]+E[Y]^2$ – JMS Jun 11 '11 at 15:17

This is a simple, but not so simple concept to understand. I personally find using the example of a table to illustrate what is going on. So we have a $2\times 2$ table with counts in each cell. To keep from the "abstract" nature of the concept, I'll use real numbers instead of letters. So we have a table of "being sick" against "having a sore throat":

$$\begin{array}{c|c} & \text{Sick} & \text{Not Sick}\\ \hline \text{Sore throat} & 12 & 5 \\ \hline \text{No Sore throat} & 4 & 55 \end{array}$$

So in this table we have 12 people who are both sick and have a sore throat, and 5 people who are not sick but still have a sore throat. Now if we condition on "Sore throat", then the chances of being sick are

$$Pr(\text{Sick}|\text{sore throat})=\frac{12}{12+5}=0.706$$

You can think of this quantity as a part of $E[Y|X]$ in the more abstract sense (not exactly the case, but will do for understanding the concept). Now We can calculate the same probability, but on the condition of no sore throat

$$Pr(\text{Sick}|\text{No sore throat})=\frac{4}{4+55}=0.068$$

This quantity is the "other part" of $E[Y|X]$. No suppose you want to calculate the chance of someone being sick, independently of their throat. We can do this (at least) two ways. The simplest in this case is to calculate the total proportion of people who are sick:

$$Pr(\text{Sick})=\frac{4+12}{4+12+55+5}=0.210$$

But You can also use the law of iterated expectations. It is more cumbersome here, but can be much simpler in more complicated problems. We have, by the product rule and sum rule:

$$Pr(\text{Sick})=Pr(\text{Sick}|\text{Sore throat})Pr(\text{Sore throat})$$ $$+Pr(\text{Sick}|\text{No Sore throat})Pr(\text{No Sore throat})$$

Now we need the marginal probabilities $Pr(\text{No Sore throat})=\frac{4+55}{4+12+55+5}=0.776$ and $Pr(\text{Sore throat})=\frac{12+5}{4+12+55+5}=0.224$. Now we have all the ingredients, we just plug them in:

$$Pr(\text{Sick})=0.706 \times 0.224 + 0.068 \times 0.776=0.210$$

Which is the same result. This is an "empirical proof" so to speak. The law simply say that it holds more generally, for arbitrary distributions.

The variance version comes about by "adding zero" to the equations, but as $0=(E[Y|X])^{2}-(E[Y|X])^{2}$, or by noting that $Var[Y|X]=E[Y^{2}|X]-(E[Y|X])^{2}$, which can be re-arranged to give $E[Y^{2}|X]=Var[Y|X]+(E[Y|X])^{2}$. It's just messy because of all the parenthesis, and remembering where the "square" is

What you really posted is just a derivation for the law of total variance, taking from granted the law of iterated expectations: $E[E[Y|X]]=E[Y]$.

Line by line, that is what happening:

1. Definition of variance: ${\rm Var}[Y]=E[Y^2]−(E[Y])^2$

2. Replaced $E[Y^2]$ by $E[E[Y^2|X]]$ according to the Law of iterated expectations.

3. As our friend already pointed out, recognize that $E[Y^2]={\rm Var}[Y^2]+(E[Y])^2$, since ${\rm Var}[Y^2]= E[Y^2] - (E[Y])^2$

4. By linearity of expectations, we have that $E[aX+bY]=a.E[X]+bE[Y]$. So, $E[{\rm Var}[Y|X]+(E[Y|X])^2]$ can be expressed as $E[{\rm Var}(X|Y)] + E[(E[Y|X])^2]$.

5. Recognize that $E[(E[Y|X])^2]−(E[E[Y|X]])^2$ is just ${\rm Var}[E[Y|X]]$.

Since this post is so old, tell me if you need a proof for the law of iterated expectations.