Deriving covariance from "embedded" random variable

I'm trying to derive covariance from "embedded" random variable (r.v.). Say we have two Gaussian r.v.'s

$$X_1 \sim N(\mu_1, \sigma^2_1)$$ $$X_2 \sim N(\mu_2, \sigma^2_2)$$

where $\mu_1$ and $\mu_2$ are also Gaussian r.v,

$$\mu_1 \sim N(\theta_1, \delta^2_1)$$ $$\mu_2 \sim N(\theta_2, \delta^2_2)$$

with covariance $\delta_{1,2}$. Here we assume that $\sigma$, $\theta$, and $\delta$ are all known.

I'm trying to get $\mathrm{cov}(X_1, X_2)$ following the definition,

$$\mathrm{cov}(X_1, X_2) = E[X_1, X_2] - E[X_1]E[X_2] \>.$$

However, I'm not sure how to incorporate $\delta_{1,2}$ into $E[X_1, X_2]$ when marginalizing out $\mu_1$ and $\mu_2$.

Or, vice versa, given covariance $\sigma_{1,2}$ between $X_1$ and $X_2$, can we derive $\delta_{1,2}$?

Any hint would be appreciated!

• I see you're still working on this thing. I suggest you construct the joint distribution of $(X_1, X_2, \mu_1, \mu_2)$. Once you have (the mean vector and covariance matrix of) the full joint distribution in hand, calculating marginals and conditionals can be done straightforwardly using formulas in the Wikipedia article on the multivariate normal distribution.
– Cyan
Apr 14, 2012 at 22:49
• If you sample $X_1$ twice, does $\mu_1$ remain the same or does it change? If it changes, does that mean that in fact $X_1 \sim N(\theta_1, \delta_1^2+\sigma_1^2)$ and its variance is $\delta_1^2+\sigma_1^2$? Apr 15, 2012 at 1:08
• @Cyan By creating a joint distribution of $(X_1, X_2, \mu_1, \mu_2)$ I need to give the covariance matrix, which contains $\sigma_{1,2}$ and $\delta_{1,2}$ directly. You see, that's what I'm after. Apr 15, 2012 at 9:52
• @Henry You're absolutely right. But I'm looking for the covariance :) Apr 15, 2012 at 9:54
• @shuaiyuancn I know my comment doesn't provide the answer to your question. What I'm saying is that you should aim to get the whole set of parameters characterizing the distribution, not simply $\mathrm{cov}(X_1, X_2)$. Then you won't have to ask for help with this problem on Cross-Validated again -- you can just apply well-known formulas to answer any question of interest to you.
– Cyan
Apr 16, 2012 at 2:57

Firstly you should be more rigorous when defining the distribution of $X_1$ and $X_2$. I think you mean $$(X_1 \mid \mu_1) \sim N(\mu_1, \sigma^2_1)$$ $$(X_2 \mid \mu_2) \sim N(\mu_2, \sigma^2_2)$$ and even more precisely you should say that $X_1$ and $X_2$ are conditionally independent given $\mu_1, \mu_2$. Right ?

Then use the well-known formula $$\boxed{\mathrm{cov}(X_1, X_2) = \mathbb{E}\left[\mathrm{cov}(X_1, X_2 \mid \mu_1, \mu_2)\right] + \mathrm{cov}\bigl(\mathbb{E}[X_1 \mid \mu_1, \mu_2], \mathbb{E}[X_2 \mid \mu_1, \mu_2]\bigr)}.$$ Now $\mathrm{cov}(X_1, X_2 \mid \mu_1, \mu_2)=0$ (conditional independance) and $\mathbb{E}[X_1 \mid \mu_1, \mu_2]=\mu_1$, $\mathbb{E}[X_2 \mid \mu_1, \mu_2]=\mu_2$, then it is easy to conclude.

• I appreciate your straightforward explanation, as well as the suggestion on rigorousness! Apr 15, 2012 at 10:04

Rewrite $X=(X_1,X_2)$ as $$X = \mu + \epsilon\,,\quad \mu \sim \mathcal{N}_2(\theta,\Delta)\,,\epsilon\sim \mathcal{N}_2(0,\Sigma)\,,$$ with independence between $\mu$ and $\epsilon$. Then $$X \sim \mathcal{N}_2(\theta,\Sigma+\Delta)$$ gives the marginal distribution of the vector $X$.

• Thanks for the helpful answer. But how to get $\Sigma$ (when I cannot sample $X$ simultaneously) ? Apr 15, 2012 at 10:08
• Uh?! You assume independence between the components of $X$ so $\Sigma$ is a diagonal matrix with elements $\sigma_1^2$ and $\sigma_2^2$. Apr 16, 2012 at 8:46
• I see. All three answers are actually the same idea, but different approaches. Thanks! Apr 17, 2012 at 9:18

Based on the answer to my comment, I would have thought you could write

• $X_1 = \theta_1 + V_1 + W_1$

• $X_2 = \theta_2 + V_2 + W_2$

where

• $V_1 =\mu_1 - \theta_1 \sim N(0, \delta_1^2)$,
• $V_2 =\mu_2 - \theta_2 \sim N(0, \delta_2^2)$,
• $W_1 = X_1 - \mu_1 \sim N(0, \sigma_1^2)$, and
• $W_2 = X_2 - \mu_2 \sim N(0, \sigma_2^2)$,

and with $W_1$ and $W_2$ independent of each other and of $V_1$ and $V_2$, but $\mathrm{cov}(V_1,V_2)=\delta_{1,2}$.

In that case it seems obvious $\mathrm{cov}(X_1,X_2)=\delta_{1,2}$.

• Thanks @Henry! I'm grateful to learn another different derivation to the problem. Apr 17, 2012 at 9:19