2 added 15 characters in body
source | link

In the Law of Iterated Expectation (LIE), $E\left[E[Y \mid X]\right] = E[Y]$, that inner expectation is a random variable which happens to be a function of $X$, say $g(X)$, and not a function of $Y$. Thus, thatThat the expectation of this function of $X$ happens to equal the expectation of $Y$ is a consequence of a a LIE. All that this is, hand-wavingly, just the assertion that the average value of   of $Y$ can be found by averaging the average values of $Y$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $X$ and $Y$ are discrete random variables with joint pmf $p_{X,Y}(x,y)$, then $$\begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align}$$ Notice how that last expectation is with respect to $X$; $E[Y\mid X]$ is a function of $X$, not of $Y$, but nevertheless its mean is the same as the mean of $Y$.

The generalized LIE that you are looking at has on the left $E\left[E[Y \mid X, Z] \mid X\right]$ in which the inner expectation is a function $h(X,Z)$ of two random variables $X$ and $Z$. The argument is similar to that outlined above but now we have to show that the random variable $E[Y\mid X]$ equals another random variable. We do this by looking at the value of $E[\mid X]$$E[Y\mid X]$ when $X$ happens to have value $x$. Skipping the explanations, we have that $$\begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align}$$ Note that the penultimate right side is the formula for the conditional expected value of the random variable $E[Y \mid X, Z]$ (a function of $X$ and $Z$) conditioned on the value of $X$. We are fixing $X$ to have value $x$, multiplying the values of the random variable $E[Y \mid X, Z]$ by the conditional pmf value of $Z$ given $X$, and summing all such terms.

Thus, for each value $x$ of the random variable $X$, the value of the random variable $E[Y\mid X]$ (which we noted earlier is a function of $X$, not of $Y$), is the same as the value of the random variable $E\left[E[Y \mid X,Z]\mid X\right]$, that is, these two random variables are equal. Would I LIE to you?

In the Law of Iterated Expectation (LIE), $E\left[E[Y \mid X]\right] = E[Y]$, that inner expectation is a random variable which happens to be a function of $X$, say $g(X)$, and not a function of $Y$. Thus, that the expectation of this function of $X$ happens to equal the expectation of $Y$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value of of $Y$ can be found by averaging the average values of $Y$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $X$ and $Y$ are discrete random variables with joint pmf $p_{X,Y}(x,y)$, then $$\begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align}$$ Notice how that last expectation is with respect to $X$; $E[Y\mid X]$ is a function of $X$, not of $Y$, but nevertheless its mean is the same as the mean of $Y$.

The generalized LIE that you are looking at has on the left $E\left[E[Y \mid X, Z] \mid X\right]$ in which the inner expectation is a function $h(X,Z)$ of two random variables $X$ and $Z$. The argument is similar to that outlined above but now we have to show that the random variable $E[Y\mid X]$ equals another random variable. We do this by looking at the value of $E[\mid X]$ when $X$ happens to have value $x$. Skipping the explanations, we have that $$\begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align}$$ Note that the penultimate right side is the formula for the conditional expected value of the random variable $E[Y \mid X, Z]$ (a function of $X$ and $Z$) conditioned on the value of $X$. We are fixing $X$ to have value $x$, multiplying the values of the random variable $E[Y \mid X, Z]$ by the conditional pmf value of $Z$ given $X$, and summing all such terms.

Thus, for each value $x$ of the random variable $X$, the value of the random variable $E[Y\mid X]$ (which we noted earlier is a function of $X$, not of $Y$), is the same as the value of the random variable $E\left[E[Y \mid X,Z]\mid X\right]$, that is, these two random variables are equal.

In the Law of Iterated Expectation (LIE), $E\left[E[Y \mid X]\right] = E[Y]$, that inner expectation is a random variable which happens to be a function of $X$, say $g(X)$, and not a function of $Y$. That the expectation of this function of $X$ happens to equal the expectation of $Y$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value   of $Y$ can be found by averaging the average values of $Y$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $X$ and $Y$ are discrete random variables with joint pmf $p_{X,Y}(x,y)$, then $$\begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align}$$ Notice how that last expectation is with respect to $X$; $E[Y\mid X]$ is a function of $X$, not of $Y$, but nevertheless its mean is the same as the mean of $Y$.

The generalized LIE that you are looking at has on the left $E\left[E[Y \mid X, Z] \mid X\right]$ in which the inner expectation is a function $h(X,Z)$ of two random variables $X$ and $Z$. The argument is similar to that outlined above but now we have to show that the random variable $E[Y\mid X]$ equals another random variable. We do this by looking at the value of $E[Y\mid X]$ when $X$ happens to have value $x$. Skipping the explanations, we have that $$\begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align}$$ Note that the penultimate right side is the formula for the conditional expected value of the random variable $E[Y \mid X, Z]$ (a function of $X$ and $Z$) conditioned on the value of $X$. We are fixing $X$ to have value $x$, multiplying the values of the random variable $E[Y \mid X, Z]$ by the conditional pmf value of $Z$ given $X$, and summing all such terms.

Thus, for each value $x$ of the random variable $X$, the value of the random variable $E[Y\mid X]$ (which we noted earlier is a function of $X$, not of $Y$), is the same as the value of the random variable $E\left[E[Y \mid X,Z]\mid X\right]$, that is, these two random variables are equal. Would I LIE to you?

1
source | link

In the Law of Iterated Expectation (LIE), $E\left[E[Y \mid X]\right] = E[Y]$, that inner expectation is a random variable which happens to be a function of $X$, say $g(X)$, and not a function of $Y$. Thus, that the expectation of this function of $X$ happens to equal the expectation of $Y$ is a consequence of a LIE. All that this is, hand-wavingly, just the assertion that the average value of of $Y$ can be found by averaging the average values of $Y$ under various conditions. In effect, it is all just a direct consequence of the law of total probability. For example, if $X$ and $Y$ are discrete random variables with joint pmf $p_{X,Y}(x,y)$, then $$\begin{align} E[Y] &= \sum_y y\cdot p_Y(y) &\scriptstyle{\text{definition}}\\ &= \sum_y y \cdot \sum_x p_{X,Y}(x,y) &\scriptstyle{\text{write in terms of joint pmf}}\\ &= \sum_y y \cdot \sum_x p_{Y\mid X}(y \mid X=x)\cdot p_X(x) &\scriptstyle{\text{write in terms of conditional pmf}}\\ &= \sum_x p_X(x)\cdot \sum_y y \cdot p_{Y\mid X}(y \mid X=x) &\scriptstyle{\text{interchange order of summation}}\\ &= \sum_x p_X(x)\cdot E[Y \mid X = x] &\scriptstyle{\text{inner sum is conditional expectation}}\\ &= E\left[E[Y\mid X]\right] &\scriptstyle{\text{RV}~E[Y\mid X]~\text{has value}~E[Y\mid X=x]~\text{when}~X=x} \end{align}$$ Notice how that last expectation is with respect to $X$; $E[Y\mid X]$ is a function of $X$, not of $Y$, but nevertheless its mean is the same as the mean of $Y$.

The generalized LIE that you are looking at has on the left $E\left[E[Y \mid X, Z] \mid X\right]$ in which the inner expectation is a function $h(X,Z)$ of two random variables $X$ and $Z$. The argument is similar to that outlined above but now we have to show that the random variable $E[Y\mid X]$ equals another random variable. We do this by looking at the value of $E[\mid X]$ when $X$ happens to have value $x$. Skipping the explanations, we have that $$\begin{align} E[Y \mid X = x] &= \sum_y y\cdot p_{Y\mid X}(y\mid X = x)\\ &= \sum_y y \cdot \frac{p_{X,Y}(x,y)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{X,Y,Z}(x,y,z)}{p_X(x)}\\ &= \sum_y y \cdot \frac{\sum_z p_{Y\mid X,Z}(y \mid X=x, Z=z)\cdot p_{X,Z}(x,z)}{p_X(x)}\\ &= \sum_z \frac{p_{X,Z}(x,z)}{p_X(x)}\sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot \sum_y y \cdot p_{Y\mid X,Z}(y \mid X=x, Z=z)\\ &= \sum_z p_{Z\mid X}(z \mid X=x)\cdot E[Y \mid X=x, Z=z)\\ &= E\left[E[Y\mid X,Z]\mid X = x\right] \end{align}$$ Note that the penultimate right side is the formula for the conditional expected value of the random variable $E[Y \mid X, Z]$ (a function of $X$ and $Z$) conditioned on the value of $X$. We are fixing $X$ to have value $x$, multiplying the values of the random variable $E[Y \mid X, Z]$ by the conditional pmf value of $Z$ given $X$, and summing all such terms.

Thus, for each value $x$ of the random variable $X$, the value of the random variable $E[Y\mid X]$ (which we noted earlier is a function of $X$, not of $Y$), is the same as the value of the random variable $E\left[E[Y \mid X,Z]\mid X\right]$, that is, these two random variables are equal.