I recently came across this identity:
$$E \left[ E \left(Y|X,Z \right) |X \right] =E \left[Y | X \right]$$
I am of course familiar with the simpler version of that rule, namely that $E \left[ E \left(Y|X \right) \right]=E \left(Y\right) $ but I was not able to find justification for its generalization.
I would be grateful if someone could point me to a not-so-technical reference for that fact or, even better, if someone could lay out a simple proof for this important result.
INFORMAL TREATMENT
We should remember that the notation where we condition on random variables is inaccurate, although economical, as notation. In reality we condition on the sigma-algebra that these random variables generate. In other words $E[Y\mid X]$ is meant to mean $E[Y\mid \sigma(X)]$. This remark may seem out of place in an "Informal Treatment", but it reminds us that our conditioning entities are collections of sets (and when we condition on a single value, then this is a singleton set). And what do these sets contain? They contain the information with which the possible values of the random variable $X$ supply us about what may happen with the realization of $Y$.
Bringing in the concept of Information, permits us to think about (and use) the Law of Iterated Expectations (sometimes called the "Tower Property") in a very intuitive way:
The sigma-algebra generated by two random variables, is at least as large as that generated by one random variable: $\sigma (X) \subseteq \sigma(X,Z)$ in the proper set-theoretic meaning. So the information about $Y$ contained in $\sigma(X,Z)$ is at least as great as the corresponding information in $\sigma (X)$.
Now, as notational innuendo, set $\sigma (X) \equiv I_x$ and $\sigma(X,Z) \equiv I_{xz}$.
Then the LHS of the equation we are looking at, can be written
$$E \left[ E \left(Y|I_{xz} \right) |I_{x} \right]$$ Describing verbally the above expression we have : "what is the expectation of {the expected value of $Y$ given Information $I_{xz}$} given that we have available information $I_x$ only?"
Can we somehow "take into account" $I_{xz}$? No - we only know $I_x$. But if we use what we have (as we are obliged by the expression we want to resolve), then we are essentially saying things about $Y$ under the expectations operator, i.e. we say "$E(Y\mid I_x)$", no more -we have just exhausted our information.
Hence $$E \left[ E \left(Y|I_{xz} \right) |I_{x} \right] = E\left(Y|I_{x} \right)$$
If somebody else doesn't, I will return for the formal treatment.
A (bit more) FORMAL TREATMENT
Let's see how two very important books of probability theory, P. Billingsley's Probability and Measure (3d ed.-1995) and D. Williams "Probability with Martingales" (1991), treat the matter of proving the "Law Of Iterated Expectations":
Billingsley devotes exactly three lines to the proof. Williams, and I quote, says
"(the Tower Property) is virtually immediate from the definition of conditional expectation".
That's one line of text. Billingsley's proof is not less opaque.
They are of course right: this important and very intuitive property of conditional expectation derives essentially directly (and almost immediately) from its definition -the only problem is, I suspect that this definition is not usually taught, or at least not highlighted, outside probability or measure theoretic circles. But in order to show in (almost) three lines that the Law of Iterated Expectations holds, we need the definition of conditional expectation, or rather, its defining property.
Let a probability space $(\Omega, \mathcal F, \mathbf P)$, and an integrable random variable $Y$. Let $\mathcal G$ be a sub-$\sigma$-algebra of $\mathcal F$, $\mathcal G \subseteq \mathcal F$. Then there exists a function $W$ that is $\mathcal G$-measurable, is integrable and (this is the defining property)
$$E(W\cdot\mathbb 1_{G}) = E(Y\cdot \mathbb 1_{G})\qquad \forall G \in \mathcal G \qquad [1]$$
where $1_{G}$ is the indicator function of the set $G$. We say that $W$ is ("a version of") the conditional expectation of $Y$ given $\mathcal G$, and we write
$W = E(Y\mid \mathcal G) \;a.s.$
The critical detail to note here is that the conditional expectation, has the same expected value as $Y$ does, not just over the whole $\mathcal G$, but in every subset $G$ of $\mathcal G$.
(I will try now to present how the Tower property derives from the definition of conditional expectation).
$W$ is a $\mathcal G$-measurable random variable. Consider then some sub-$\sigma$-algebra, say $\mathcal H \subseteq \mathcal G$. Then $G\in \mathcal H \Rightarrow G\in \mathcal G$. So, in an analogous manner as previously, we have the conditional expectation of $W$ given $\mathcal H$, say $U=E(W\mid \mathcal H) \;a.s.$ that is characterized by
$$E(U\cdot\mathbb 1_{G}) = E(W\cdot \mathbb 1_{G})\qquad \forall G \in \mathcal H \qquad [2]$$
Since $\mathcal H \subseteq \mathcal G$, equations $[1]$ and $[2]$ give us
$$E(U\cdot\mathbb 1_{G}) = E(Y\cdot \mathbb 1_{G})\qquad \forall G \in \mathcal H \qquad [3]$$
But this is the defining property of the conditional expectation of $Y$ given $\mathcal H$. So we are entitled to write $U=E(Y\mid \mathcal H)\; a.s.$
Since we have also by construction $U = E(W\mid \mathcal H) = E\big(E[Y\mid \mathcal G]\mid \mathcal H\big)$, we just proved the Tower property, or the general form of the Law of Iterated Expectations - in eight lines.
The way I understand conditional expectation and teach my students is the following:
conditional expectation $E[Y|\sigma(X)]$ is a picture taken by a camera with resolution $\sigma(X)$
As mentioned by Alecos Papadopoulos, the notation $E[Y|\sigma(X)]$ is more precise than $E[Y|X]$. Along the line of camera, one can think of $Y$ as the original object, e.g., a landscape, scenery. $E[Y|\sigma(X,Z)]$ is a picture taken by a camera with resolution $\sigma(X,Z)$. Expectation is an averaging operator ("blurring" operator?). The scenary may contain a lot of stuff, but the picture you took using a camera with low resolution will certainly make some detail go away, e.g., there may be an UFO in the sky that can be seen by your naked eye but it does not appear in your picture taken by (iphone 3?)
If the resolution is so high such that $\sigma(X,Z)=\sigma(Y)$, then this picture is able to capture every detail of the real scenery. In this case, we have $E[Y|\sigma(Y)]=Y$.
Now, $E[E[Y|\sigma(X,Z)]|\sigma(X)]$ can be viewed as: using another camera with resolution $\sigma(X)$ (e.g., iphone 1) which is lower than $\sigma(X,Z)$ (e.g., iphone 3) and take a picture on that picture generated by camera with resolution $\sigma(X,Z)$, then it should be clear that this picture on a picture should be the same as if you originally just use a camera with low resolution $\sigma(X)$ on the scenery.
This provides intuition on $E[E[Y|X,Z]|X]=E[Y|X]$. In fact this same intuition tells us that $E[E[Y|X]|X,Z]=E[Y|X]$ still. This is because: if your first picture is taken by iphone 1 (i.e., low resolution), and now you want to use a better camera (e.g., iphone 3) to generate another photo on the first photo, then there is no way you can improve the quality of the first photo.
In the Law of Iterated Expectation (LIE), $E\left[E[Y \mid X]\right] = E[Y]$, that inner expectation is a random variable which happens to be a function of $X$, say $g(X)$, and not a function of $Y$. That the expectation of this function of $X$ happens to equal the expectation of $Y$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value of $Y$ can be found by averaging the average values of $Y$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $X$ and $Y$ are discrete random variables with joint pmf $p_{X,Y}(x,y)$, then $$\begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align}$$ Notice how that last expectation is with respect to $X$; $E[Y\mid X]$ is a function of $X$, not of $Y$, but nevertheless its mean is the same as the mean of $Y$.
The generalized LIE that you are looking at has on the left $E\left[E[Y \mid X, Z] \mid X\right]$ in which the inner expectation is a function $h(X,Z)$ of two random variables $X$ and $Z$. The argument is similar to that outlined above but now we have to show that the random variable $E[Y\mid X]$ equals another random variable. We do this by looking at the value of $E[Y\mid X]$ when $X$ happens to have value $x$. Skipping the explanations, we have that $$\begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align}$$ Note that the penultimate right side is the formula for the conditional expected value of the random variable $E[Y \mid X, Z]$ (a function of $X$ and $Z$) conditioned on the value of $X$. We are fixing $X$ to have value $x$, multiplying the values of the random variable $E[Y \mid X, Z]$ by the conditional pmf value of $Z$ given $X$, and summing all such terms.
Thus, for each value $x$ of the random variable $X$, the value of the random variable $E[Y\mid X]$ (which we noted earlier is a function of $X$, not of $Y$), is the same as the value of the random variable $E\left[E[Y \mid X,Z]\mid X\right]$, that is, these two random variables are equal. Would I LIE to you?
