6
$\begingroup$

My book states that $\mathrm{E}[X_2] = \mathrm{E}[\mathrm{E}[X_2\mid X_1]]$ and gives a weird proof and a short paragraph about how awesome this result is and how intuitive this is.

I neither understand the proof nor do I understand the "intuition". Can someone give me both?

The textbook proof uses some non-standard notation and weird transformations but I am hoping for a clear lucid proof.

$\endgroup$
  • $\begingroup$ ¿Maybe you should give the name of the textbook? Why is the result intuitive: Consider you have an $N\times M$ matrix and want the mean of the components. You can compute it in three ways: directly, the mean of all the components. Or first the mean of all the rows, and then the mean of the rowmeans. Or you can do the same with row replaced by column. All this give clearly the same answer. The stated theorem is just a theoretical version of this result. $\endgroup$ – kjetil b halvorsen Oct 4 '12 at 23:28
  • 2
    $\begingroup$ What you've written is not true, perhaps simply due to a typo (the RHS is missing an expectation operator). The (correct) result follows almost immediately from the definition of conditional expectation. Could you please cite your source explicitly and edit your post to provide the result actually stated in the book? $\endgroup$ – cardinal Oct 4 '12 at 23:31
  • $\begingroup$ Do you understand what $\mathrm{E}[X_2\mid X_1]$ is? $\endgroup$ – Zen Oct 4 '12 at 23:42
  • 5
    $\begingroup$ Inquest, I don't think @Zen intended to sound threatening with that question. I take it as an abbreviated way of asking, "what is your understanding of the meaning of $\mathbf{E}[X_2|X_2]$?". Your clarification of this will establish a starting point for answers that address your intuition and what you would like to see as a proof. $\endgroup$ – whuber Oct 5 '12 at 1:07
  • 1
    $\begingroup$ My book states (as a particular case) : E[X] = E[E[X|Y]]. My book is Craig and Hogg 7th edition. $\endgroup$ – user8968 Oct 5 '12 at 15:04
11
$\begingroup$

Try to build your intuition with particular cases first. Suppose that $X\in\{x_1,\dots,x_m\}$ with pmf $p_X$ and $Y\in\{y_1,\dots,y_n\}$ with pmf $p_Y$, such that $p_Y(y_i)>0$, for $i=1,\dots,n$. Let $p_{X,Y}$ be the joint pmf of $X$ and $Y$. The conditional pmf of $X$ given $Y$ is defined as $$ p_{X\mid Y}(x_i\mid y_j) = \frac{p_{X,Y}(x_i,y_j)}{p_Y(y_j)} \qquad . \qquad\qquad (*) $$ Note that $(*)$ is a pmf for each "fixed" $y_j$. Now, interpret the expectation $$ \mathrm{E}[X]=\sum_{i=1}^m x_i\,p_X(x_i) $$ intuitively as your best guess about the value of $X$, and extend this interpretation to a new object $\mathrm{E}[X\mid Y=y]$, called the conditional expectation of $X$ given that $Y=y$, which represents your best guess about the value of $X$ when you are given the information that $Y=y$.

The key point is that this conditional expectation is computed from $(*)$ as $$ \mathrm{E}[X\mid Y=y] = \sum_{i=1}^m x_i\,p_{X\mid Y}(x_i\mid y) = g(y) \, , $$ for some nice function $g$. The notation $\mathrm{E}[X\mid Y]$ is just an abreviation for the random variable $g(Y)$ (that is what I've asked you in my comment). But you already now how to compute the expectation of $g(Y)$ using the "law of the unconscious statistician" as $$ \mathrm{E}[g(Y)] = \sum_{i=1}^n g(y_i)\,p_Y(y_i) = \sum_{i=1}^n \mathrm{E}[X\mid Y=y_i]\,p_Y(y_i) $$ $$ = \sum_{i=1}^n \sum_{j=1}^m x_j\,p_{X\mid Y}(x_j\mid y_i)\,p_Y(y_i) $$ $$ = \sum_{j=1}^m x_j \sum_{i=1}^n p_{X,Y}(x_j,y_i) = \sum_{j=1}^m x_j\, p_X(x_j) $$ which can be written as $$ \mathrm{E}[\mathrm{E}[X\mid Y]]=\mathrm{E}[X] \, . $$

The importance of this concept is impossible to overstate. Its generalization to the conditional expectation given a sigma-field dominates the development of modern probability theory, martingale theory, etc. Defining conditional variance $\mathrm{Var}[X\mid Y]$ in the same spirit, the corresponding property $$ \mathrm{Var}[X] = \mathrm{E}[\mathrm{Var}[X\mid Y]] + \mathrm{Var}[\mathrm{E}[X\mid Y]] $$ is the basis for a variance reduction technique extensively used in simulation known as rao-blackwelization.

To give you a taste of the conditional expectation as a tool, consider this problem. Let $N$ be the number of questions posted monthly at Stack Exchange, and let $X_i$ be the number of answers to question $i$. Suppose that the $X_i$'s are IID, and that $N$ and the $X_i$'s are independent. The number of total answers in a month is given by the random sum $\sum_{i=1}^N X_i$ (note that the number of terms in this sum is random). What is $\mathrm{E}\left[\sum_{i=1}^N X_i\right]$?

To solve this we will justify some properties of the conditional expectation intuitively. First, we "use what we know", plus the independence of the $X_i$'s and $N$ to get $$ \mathrm{E}\left[\sum_{i=1}^N X_i \,\Bigg|\, N = n\right] = \mathrm{E}\left[\sum_{i=1}^n X_i \,\Bigg|\, N = n\right] = \mathrm{E}\left[\sum_{i=1}^n X_i\right] = n \,\mathrm{E}\left[X_1\right] \, . $$ Hence, the random variable $\mathrm{E}\left[\sum_{i=1}^N X_i\mid N\right]=N \,\mathrm{E}\left[X_1\right]$, and using the fundamental property we have $$ \mathrm{E}\left[\sum_{i=1}^N X_i\right]=\mathrm{E}\left[\mathrm{E}\left[\sum_{i=1}^N X_i \,\Bigg|\, N\right]\right] = \mathrm{E}[N]\, \mathrm{E}\left[X_1\right] \, . $$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Tks, Prof.! They've told me that 3K is the thing. Let's see. $\endgroup$ – Zen Oct 5 '12 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy