Why Condition on a Random Variable in Iterated Expectation?

Question

Say I wanted to take the following expectation for $C$ a constant and an RV $\theta$ with density $p(\theta)$.

$$E[\theta-C] = \int_{\theta}(\theta-C)p(\theta)d\theta$$

Now given $p(y|\theta)$ and a function $\phi(y)$, I would like to take the following expectation:

$$E[\theta-\phi(y)]$$

I am having trouble proving to myself the following:

$$E[\theta-\phi(y)] \ne \int_{\theta}(\theta-\phi(y))p(\theta)d\theta$$

But rather I believe the correct solution is:

$$E[\theta-\phi(y)] = \int_{y}\int_{\theta}(\theta-\phi(y))p(\theta|y)d\theta\ p(y)dy$$ $$=\int_{\theta}\int_{y}(\theta-\phi(y))\ p(\theta,y)dyd\theta$$ $$=E[E[(\theta-\phi(Y))|Y]]$$

Intuitively, I think the last line makes since, taking the average of $\theta -\phi(y)$ for a given $y$, and then taking the average of that over all $y$.

But I am still not convinced on why I shouldnt take the expectation treating $\phi(y)$ as a constant.

Is it necessary or sufficient to do the iterated expection if $Y$ is a random variable? Or is it only necessary when $Y$ depends on $\theta$ via $p(y|\theta)$?

Thanks for reading and any help on clairification!

Answer 1

In fact both expressions for $E[\theta -\phi(y)]$ are correct, depending on which type of expectation you want to compute. The integral over $\theta$ only computes an expectation with respect to the marginal distribution of $\theta$, while the integral over both, $\theta$ and $y$, provides the expectation over the full joint distribution via the law of total expectation (see https://en.wikipedia.org/wiki/Law_of_total_expectation). Sometimes people use indices to indicate over which of the variables they intend to compute the expectation, so I think you simply got fooled by notation or the lack thereof.

Think of it this way: For most (but not all) probability distributions it is possible to compute an expectation. In the first case this is done for $p(\theta)$, the marginal distribution of $\theta$, while in the latter it is done for $p(\theta,y)$, i.e. the joint distribution of both variables or, equivalently, the random vector $(\theta,y)$.