2 added 15 characters in body edited May 1 '14 at 22:48 Dilip Sarwate 31.7k33 gold badges5959 silver badges154154 bronze badges In the Law of Iterated Expectation (LIE), $$E\left[E[Y \mid X]\right] = E[Y]$$, that inner expectation is a random variable which happens to be a function of $$X$$, say $$g(X)$$, and not a function of $$Y$$. Thus, thatThat the expectation of this function of $$X$$ happens to equal the expectation of $$Y$$ is a consequence of a a LIE. All that this is, hand-wavingly, just the assertion that the average value of   of $$Y$$ can be found by averaging the average values of $$Y$$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $$X$$ and $$Y$$ are discrete random variables with joint pmf $$p_{X,Y}(x,y)$$, then \begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align} Notice how that last expectation is with respect to $$X$$; $$E[Y\mid X]$$ is a function of $$X$$, not of $$Y$$, but nevertheless its mean is the same as the mean of $$Y$$. The generalized LIE that you are looking at has on the left $$E\left[E[Y \mid X, Z] \mid X\right]$$ in which the inner expectation is a function $$h(X,Z)$$ of two random variables $$X$$ and $$Z$$. The argument is similar to that outlined above but now we have to show that the random variable $$E[Y\mid X]$$ equals another random variable. We do this by looking at the value of $$E[\mid X]$$$$E[Y\mid X]$$ when $$X$$ happens to have value $$x$$. Skipping the explanations, we have that \begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align} Note that the penultimate right side is the formula for the conditional expected value of the random variable $$E[Y \mid X, Z]$$ (a function of $$X$$ and $$Z$$) conditioned on the value of $$X$$. We are fixing $$X$$ to have value $$x$$, multiplying the values of the random variable $$E[Y \mid X, Z]$$ by the conditional pmf value of $$Z$$ given $$X$$, and summing all such terms. Thus, for each value $$x$$ of the random variable $$X$$, the value of the random variable $$E[Y\mid X]$$ (which we noted earlier is a function of $$X$$, not of $$Y$$), is the same as the value of the random variable $$E\left[E[Y \mid X,Z]\mid X\right]$$, that is, these two random variables are equal. Would I LIE to you? In the Law of Iterated Expectation (LIE), $$E\left[E[Y \mid X]\right] = E[Y]$$, that inner expectation is a random variable which happens to be a function of $$X$$, say $$g(X)$$, and not a function of $$Y$$. Thus, that the expectation of this function of $$X$$ happens to equal the expectation of $$Y$$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value of of $$Y$$ can be found by averaging the average values of $$Y$$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $$X$$ and $$Y$$ are discrete random variables with joint pmf $$p_{X,Y}(x,y)$$, then \begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align} Notice how that last expectation is with respect to $$X$$; $$E[Y\mid X]$$ is a function of $$X$$, not of $$Y$$, but nevertheless its mean is the same as the mean of $$Y$$. The generalized LIE that you are looking at has on the left $$E\left[E[Y \mid X, Z] \mid X\right]$$ in which the inner expectation is a function $$h(X,Z)$$ of two random variables $$X$$ and $$Z$$. The argument is similar to that outlined above but now we have to show that the random variable $$E[Y\mid X]$$ equals another random variable. We do this by looking at the value of $$E[\mid X]$$ when $$X$$ happens to have value $$x$$. Skipping the explanations, we have that \begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align} Note that the penultimate right side is the formula for the conditional expected value of the random variable $$E[Y \mid X, Z]$$ (a function of $$X$$ and $$Z$$) conditioned on the value of $$X$$. We are fixing $$X$$ to have value $$x$$, multiplying the values of the random variable $$E[Y \mid X, Z]$$ by the conditional pmf value of $$Z$$ given $$X$$, and summing all such terms. Thus, for each value $$x$$ of the random variable $$X$$, the value of the random variable $$E[Y\mid X]$$ (which we noted earlier is a function of $$X$$, not of $$Y$$), is the same as the value of the random variable $$E\left[E[Y \mid X,Z]\mid X\right]$$, that is, these two random variables are equal. In the Law of Iterated Expectation (LIE), $$E\left[E[Y \mid X]\right] = E[Y]$$, that inner expectation is a random variable which happens to be a function of $$X$$, say $$g(X)$$, and not a function of $$Y$$. That the expectation of this function of $$X$$ happens to equal the expectation of $$Y$$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value   of $$Y$$ can be found by averaging the average values of $$Y$$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $$X$$ and $$Y$$ are discrete random variables with joint pmf $$p_{X,Y}(x,y)$$, then \begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align} Notice how that last expectation is with respect to $$X$$; $$E[Y\mid X]$$ is a function of $$X$$, not of $$Y$$, but nevertheless its mean is the same as the mean of $$Y$$. The generalized LIE that you are looking at has on the left $$E\left[E[Y \mid X, Z] \mid X\right]$$ in which the inner expectation is a function $$h(X,Z)$$ of two random variables $$X$$ and $$Z$$. The argument is similar to that outlined above but now we have to show that the random variable $$E[Y\mid X]$$ equals another random variable. We do this by looking at the value of $$E[Y\mid X]$$ when $$X$$ happens to have value $$x$$. Skipping the explanations, we have that \begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align} Note that the penultimate right side is the formula for the conditional expected value of the random variable $$E[Y \mid X, Z]$$ (a function of $$X$$ and $$Z$$) conditioned on the value of $$X$$. We are fixing $$X$$ to have value $$x$$, multiplying the values of the random variable $$E[Y \mid X, Z]$$ by the conditional pmf value of $$Z$$ given $$X$$, and summing all such terms. Thus, for each value $$x$$ of the random variable $$X$$, the value of the random variable $$E[Y\mid X]$$ (which we noted earlier is a function of $$X$$, not of $$Y$$), is the same as the value of the random variable $$E\left[E[Y \mid X,Z]\mid X\right]$$, that is, these two random variables are equal. Would I LIE to you? 1 answered May 1 '14 at 22:22 Dilip Sarwate 31.7k33 gold badges5959 silver badges154154 bronze badges In the Law of Iterated Expectation (LIE), $$E\left[E[Y \mid X]\right] = E[Y]$$, that inner expectation is a random variable which happens to be a function of $$X$$, say $$g(X)$$, and not a function of $$Y$$. Thus, that the expectation of this function of $$X$$ happens to equal the expectation of $$Y$$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value of of $$Y$$ can be found by averaging the average values of $$Y$$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $$X$$ and $$Y$$ are discrete random variables with joint pmf $$p_{X,Y}(x,y)$$, then \begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align} Notice how that last expectation is with respect to $$X$$; $$E[Y\mid X]$$ is a function of $$X$$, not of $$Y$$, but nevertheless its mean is the same as the mean of $$Y$$. The generalized LIE that you are looking at has on the left $$E\left[E[Y \mid X, Z] \mid X\right]$$ in which the inner expectation is a function $$h(X,Z)$$ of two random variables $$X$$ and $$Z$$. The argument is similar to that outlined above but now we have to show that the random variable $$E[Y\mid X]$$ equals another random variable. We do this by looking at the value of $$E[\mid X]$$ when $$X$$ happens to have value $$x$$. Skipping the explanations, we have that \begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align} Note that the penultimate right side is the formula for the conditional expected value of the random variable $$E[Y \mid X, Z]$$ (a function of $$X$$ and $$Z$$) conditioned on the value of $$X$$. We are fixing $$X$$ to have value $$x$$, multiplying the values of the random variable $$E[Y \mid X, Z]$$ by the conditional pmf value of $$Z$$ given $$X$$, and summing all such terms. Thus, for each value $$x$$ of the random variable $$X$$, the value of the random variable $$E[Y\mid X]$$ (which we noted earlier is a function of $$X$$, not of $$Y$$), is the same as the value of the random variable $$E\left[E[Y \mid X,Z]\mid X\right]$$, that is, these two random variables are equal.