Say I wanted to take the following expectation for $C$ a constant and an RV $\theta$ with density $p(\theta)$.

$$E[\theta-C] = \int_{\theta}(\theta-C)p(\theta)d\theta$$

Now given $p(y|\theta)$ and a function $\phi(y)$, I would like to take the following expectation:


I am having trouble proving to myself the following:

$$E[\theta-\phi(y)] \ne \int_{\theta}(\theta-\phi(y))p(\theta)d\theta$$

But rather I believe the correct solution is:

$$E[\theta-\phi(y)] = \int_{y}\int_{\theta}(\theta-\phi(y))p(\theta|y)d\theta\ p(y)dy$$ $$=\int_{\theta}\int_{y}(\theta-\phi(y))\ p(\theta,y)dyd\theta$$ $$=E[E[(\theta-\phi(Y))|Y]]$$

Intuitively, I think the last line makes since, taking the average of $\theta -\phi(y)$ for a given $y$, and then taking the average of that over all $y$.

But I am still not convinced on why I shouldnt take the expectation treating $\phi(y)$ as a constant.

Is it necessary or sufficient to do the iterated expection if $Y$ is a random variable? Or is it only necessary when $Y$ depends on $\theta$ via $p(y|\theta)$?

Thanks for reading and any help on clairification!

  • $\begingroup$ I think you should clarify whether you want $E_{\theta|y}(\theta - \phi(y))$ or $E_{\theta,y}(\theta - \phi(y))$. $\endgroup$ – Tim Mak Apr 27 '20 at 6:42

In fact both expressions for $E[\theta -\phi(y)]$ are correct, depending on which type of expectation you want to compute. The integral over $\theta$ only computes an expectation with respect to the marginal distribution of $\theta$, while the integral over both, $\theta$ and $y$, provides the expectation over the full joint distribution via the law of total expectation (see https://en.wikipedia.org/wiki/Law_of_total_expectation). Sometimes people use indices to indicate over which of the variables they intend to compute the expectation, so I think you simply got fooled by notation or the lack thereof.

Think of it this way: For most (but not all) probability distributions it is possible to compute an expectation. In the first case this is done for $p(\theta)$, the marginal distribution of $\theta$, while in the latter it is done for $p(\theta,y)$, i.e. the joint distribution of both variables or, equivalently, the random vector $(\theta,y)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.