0
$\begingroup$

I am trying to figure out how to prove a result about the expectation of a random variable under the posterior predictive distribution, that may or may not be true.

Let $X$ be a random variable, and $D$ be observed data of this random variable. Suppose $X$ belongs to a parametric family with parameter $\theta$. Then I wish to show, \begin{equation} \mathbb{E}_{X|D}[X]=\mathbb{E}_{X|D}[\mathbb{E}_{\theta|D}[X|\theta]] \end{equation}

where the outer expectation is taken under the posterior predictive, and the inner expectation is taken under the posterior distribution. It seems intuitively true to me because under the posterior predictive distribution we don't assume $X$ has any parameter, so when taking the expectation of $X$ under the predictive, we must first average out over the possible values of $\theta$.

My ideas about how to prove it used the tower rule (law of total expectation) and using that the posterior predictive pdf satisfies $f(x|D)=\mathbb{E}_{\theta|D}[f(x|\theta)]$ but neither of these worked. Does anyone have any suggestions about how to prove this, if it is even true?

Thanks.

EDIT: Here is an example of this in action.

Suppose $X \sim N(\theta,\sigma^2)$, with $\theta$ unknown. Then the posterior $\theta|D_n$ is Normal with parameters $\theta_n,\sigma^2_n$ say. The posterior-predictive distribution is also normal with parameters $\theta_n$ and $\sigma^2_n+\sigma^2$.

Then $\mathbb{E}_{X|D}[X]=\theta_n$ and \begin{align} \mathbb{E}_{X|D}[\mathbb{E}_{\theta|D}[X|\theta]]&=\mathbb{E}_{X|D}[\mathbb{E}_{\theta|D}[\theta+{\sigma}Z]]\\ &=\mathbb{E}_{X|D}[\theta_n+{\sigma}Z]=\theta_n+0=\theta_n \end{align}

This is just one example though. Still not convinced it is correct.

$\endgroup$
8
  • $\begingroup$ Since $D$ are observed realizations of $X$, then conditioning on $D$ is not correct since $D$ is not a random variable. Because of this, I do not understand what you mean by $\mathbb{E}_{X|D}[X]$. Could you please explain? $\endgroup$ – mhdadk Jan 12 at 17:24
  • $\begingroup$ @mhdadk Sorry, fixed in post. I just meant an I.I.D. sample $X_1,...,X_n$ such that each $X_i$ is distributed like $X$. $\endgroup$ – jacobe Jan 12 at 17:29
  • $\begingroup$ A random sample is the same as an observed realization. The problem still exists. Could you explain in words what you are trying to prove? $\endgroup$ – mhdadk Jan 12 at 17:31
  • $\begingroup$ @mhdadk I think it is fixed now. According to Casella and Berger, "The random variables $X_1,...,X_n$ are called a random sample if they are mutually independent and identically distributed". But I think we're getting caught up in pedantics here. $\endgroup$ – jacobe Jan 12 at 17:36
  • 1
    $\begingroup$ Let us continue this discussion in chat. $\endgroup$ – mhdadk Jan 12 at 17:41
0
$\begingroup$

I am not sure about $\mathbb{E}_{\theta|D}[X|\theta]$, since the subscript $\theta|D$ implies that the expectation is averaging over all $\theta$, but the $|\theta$ part in the expectation itself implies that $\theta$ has already been observed. Therefore, I will assume that $\mathbb{E}_{\theta|D}[X|\theta] \equiv \mathbb{E}_{\theta|D}[X]$.

Using the definition of a conditional expectation:

$$ \mathbb{E}_{X|D}[\mathbb{E}_{\theta|D}[X]] = \mathbb{E}_{X|D}\left[\int_{\theta} x \ p(\theta|D) \ d\theta \right] $$

Using the definition of conditional expectation again:

$$ \begin{align} \mathbb{E}_{X|D}\left[\int_{\theta} x \ p(\theta|D) \ d\theta \right] &= \int_x p(x|D) \cdot \left[\int_{\theta} x \ p(\theta|D) \ d\theta \right] dx \\ &= \int_x x \ p(x|D) \cdot \left[\int_{\theta} \ p(\theta|D) \ d\theta \right] dx \\ &= \int_x x \ p(x|D) \ dx \\ &= \mathbb{E}_{X|D}[X] \end{align} $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.