Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I recently came across this identity:

$$E \left[ E \left(y|x,z \right) |x \right] =E \left(y | x \right)$$

I am of course familiar with the simpler version of that rule, namely that $E \left[ E \left(y|x \right) \right]=E \left(y\right) $ but I was not able to find justification for its generalization.

I would be grateful if someone could point me to a not-so-technical reference for that fact or, even better, if someone could lay out a simple proof for this important result.

Thank you.

share|improve this question
    
If $y$ was itself conditioned on some $x$ then wouldn't this fall exactly out of the simpler version? –  Mehrdad May 1 at 19:12
add comment

2 Answers 2

up vote 7 down vote accepted

INFORMAL TREATMENT

We should remember that the notation where we condition on random variables is inaccurate, although economical, as notation. In reality we condition on the sigma-algebra that these random variables generate. In other words $E[Y\mid X]$ is meant to mean $E[Y\mid \sigma(X)]$. This remark may seem out of place in an "Informal Treatment", but it reminds us that our conditioning entities are collections of sets (and when we condition on a single value, then this is a singleton set). And what do these sets contain? They contain the information with which the possible values of the random variable $X$ supply us about what may happen with the realization of $Y$.
Bringing in the concept of Information, permits us to think about (and use) the Law of Iterated Expectations (sometimes called the "Tower Property") in a very intuitive way:
The sigma-algebra generated by two random variables, is at least as large as that generated by one random variable: $\sigma (X) \subseteq \sigma(X,Z)$ in the proper set-theoretic meaning. So the information about $Y$ contained in $\sigma(X,Z)$ is at least as great as the corresponding information in $\sigma (X)$.
Now, as notational innuendo, set $\sigma (X) \equiv I_x$ and $\sigma(X,Z) \equiv I_{xz}$. Then the LHS of the equation we are looking at, can be written

$$E \left[ E \left(Y|I_{xz} \right) |I_{x} \right]$$ Describing verbally the above expression we have : "what is the expectation of {the expected value of $Y$ given Information $I_{xz}$} given that we have available information $I_x$ only?"

Can we somehow "take into account" $I_{xz}$? No - we only know $I_x$. But if we use what we have (as we are obliged by the expression we want to resolve), then we are essentially saying things about $Y$ under the expectations operator, i.e. we say "$E(Y\mid I_x)$", no more -we have just exhausted our information.

Hence $$E \left[ E \left(Y|I_{xz} \right) |I_{x} \right] = E\left(Y|I_{x} \right)$$

If somebody else doesn't, I will return for the formal treatment.

A (bit more) FORMAL TREATMENT

Let's see how two very important books of probability theory, P. Billingsley's Probability and Measure (3d ed.-1995) and D. Williams "Probability with Martingales" (1991), treat the matter of proving the "Law Of Iterated Expectations":
Billingsley devotes exactly three lines to the proof. Williams, and I quote, says

"(the Tower Property) is virtually immediate from the definition of conditional expectation".

That's one line of text. Billingsley's proof is not less opaque.

They are of course right: this important and very intuitive property of conditional expectation derives essentially directly (and almost immediately) from its definition -the only problem is, I suspect that this definition is not usually taught, or at least not highlighted, outside probability or measure theoretic circles. But in order to show in (almost) three lines that the Law of Iterated Expectations holds, we need the definition of conditional expectation, or rather, its defining property.

Let a probability space $(\Omega, \mathcal F, \mathbf P)$, and an integrable random variable $Y$. Let $\mathcal G$ be a sub-$\sigma$-algebra of $\mathcal F$, $\mathcal G \subseteq \mathcal F$. Then there exists a function $W$ that is $\mathcal G$-measurable, is integrable and (this is the defining property)

$$E(W\cdot\mathbb 1_{G}) = E(Y\cdot \mathbb 1_{G})\qquad \forall G \in \mathcal G \qquad [1]$$

where $1_{G}$ is the indicator function of the set $G$. We say that $W$ is ("a version of") the conditional expectation of $Y$ given $\mathcal G$, and we write $W = E(Y\mid \mathcal G) \;a.s.$
The critical detail to note here is that the conditional expectation, has the same expected value as $Y$ does, not just over the whole $\mathcal G$, but in every subset $G$ of $\mathcal G$.

(I will try now to present how the Tower property derives from the definition of conditional expectation).

$W$ is a $\mathcal G$-measurable random variable. Consider then some sub-$\sigma$-algebra, say $\mathcal H \subseteq \mathcal G$. Then $G\in \mathcal H \Rightarrow G\in \mathcal G$. So, in an analogous manner as previously, we have the conditional expectation of $W$ given $\mathcal H$, say $U=E(W\mid \mathcal H) \;a.s.$ that is characterized by

$$E(U\cdot\mathbb 1_{G}) = E(W\cdot \mathbb 1_{G})\qquad \forall G \in \mathcal H \qquad [2]$$

Since $\mathcal H \subseteq \mathcal G$, equations $[1]$ and $[2]$ give us

$$E(U\cdot\mathbb 1_{G}) = E(Y\cdot \mathbb 1_{G})\qquad \forall G \in \mathcal H \qquad [3]$$

But this is the defining property of the conditional expectation of $Y$ given $\mathcal H$. So we are entitled to write $U=E(Y\mid \mathcal H)\; a.s.$
Since we have also by construction $U = E(W\mid \mathcal H) = E\big(E[Y\mid \mathcal G]\mid \mathcal H\big)$, we just proved the Tower property, or the general form of the Law of Iterated Expectations - in eight lines.

share|improve this answer
2  
(+1) This is a helpful way to describe an abstract and difficult concept. I believe, though, that the phrase "...is no bigger..." should be "is no smaller." Better yet, that section could be made clearer by removing the negatives and using a parallel construction, as in "The sigma algebra generated by two variables is at least as large as that generated by one random variable ... So the information about $Y$ contained in $\sigma(X,Z)$ is at least as great as the corresponding information in $\sigma(X)$." –  whuber May 1 at 16:37
    
Thank you both, cc @whuber. This is a very useful theorem. –  JohnK May 1 at 17:07
    
@ whuber Thanks for spotting this -and for the suggestion. –  Alecos Papadopoulos May 1 at 17:40
    
Thank you very much for your addendum Alecos. –  JohnK May 2 at 4:58
add comment

In the Law of Iterated Expectation (LIE), $E\left[E[Y \mid X]\right] = E[Y]$, that inner expectation is a random variable which happens to be a function of $X$, say $g(X)$, and not a function of $Y$. That the expectation of this function of $X$ happens to equal the expectation of $Y$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value of $Y$ can be found by averaging the average values of $Y$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $X$ and $Y$ are discrete random variables with joint pmf $p_{X,Y}(x,y)$, then $$\begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align}$$ Notice how that last expectation is with respect to $X$; $E[Y\mid X]$ is a function of $X$, not of $Y$, but nevertheless its mean is the same as the mean of $Y$.

The generalized LIE that you are looking at has on the left $E\left[E[Y \mid X, Z] \mid X\right]$ in which the inner expectation is a function $h(X,Z)$ of two random variables $X$ and $Z$. The argument is similar to that outlined above but now we have to show that the random variable $E[Y\mid X]$ equals another random variable. We do this by looking at the value of $E[Y\mid X]$ when $X$ happens to have value $x$. Skipping the explanations, we have that $$\begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align}$$ Note that the penultimate right side is the formula for the conditional expected value of the random variable $E[Y \mid X, Z]$ (a function of $X$ and $Z$) conditioned on the value of $X$. We are fixing $X$ to have value $x$, multiplying the values of the random variable $E[Y \mid X, Z]$ by the conditional pmf value of $Z$ given $X$, and summing all such terms.

Thus, for each value $x$ of the random variable $X$, the value of the random variable $E[Y\mid X]$ (which we noted earlier is a function of $X$, not of $Y$), is the same as the value of the random variable $E\left[E[Y \mid X,Z]\mid X\right]$, that is, these two random variables are equal. Would I LIE to you?

share|improve this answer
    
Excellent answer. Thank you so much. –  JohnK May 1 at 22:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.