# How to Derive the Conditional Variance in a Bivariate Normal Distribution After Bayesian Updating?

I'm working with a bivariate normal distribution of two variables, $$\theta_1$$ and $$\theta_2$$ in a Bayesian framework, with an intial joint prior distribution defined as:

$$\begin{pmatrix} \theta_1 \\ \theta_2 \end{pmatrix} \sim \mathcal{N}\left( \begin{pmatrix} \theta_{01} \\ \theta_{02} \end{pmatrix}, \begin{pmatrix} \sigma_{01}^2 & \rho \sigma_{01} \sigma_{02} \\ \rho \sigma_{01} \sigma_{02} & \sigma_{02}^2 \end{pmatrix} \right)$$

Importantly, I want to assume that the correlation $$\rho$$ is a known and fixed quantity/constant.

After observing $$N$$ data points related to $$\theta_1$$, I have updated the variance of $$\theta_1$$ using Bayesian principles, resulting in a new posterior variance $$\sigma_{\text{post}}^2$$:

$$\sigma_{\text{post}}^2 = \left(\frac{1}{\sigma_{01}^2} + \frac{N}{\sigma^2}\right)^{-1}$$

where $$\sigma^2$$ is the variance of the observation errors.

My challenge is to accurately derive the conditional variance of $$\theta_2$$ given $$\theta_1$$ after this update. The traditional formula for the conditional variance in a bivariate normal distribution is:

$$\sigma_{\theta_2|\theta_1}^2 = \sigma_{02}^2 - \frac{(\rho \sigma_{01} \sigma_{02})^2}{\sigma_{01}^2}$$

Given that the variance of $$\theta_1$$ has been updated, how should I adjust this formula correctly? Specifically, how does the updated $$\sigma_{\text{post}}^2$$ impact the conditional variance of $$\theta_2$$?

It definitely can't be:

$$\sigma_{02}^2(1-\rho^2)$$

Because if I only observe 1 observation for $$\theta_1$$, the variance drops down from $$\sigma_{02}^2$$ to $$\sigma_{02}^2(1 - \rho^2)$$, but if I observe a further 100 observations, it will make no difference!

I've read in a paper (related to surrogate endpoints in pharmaceutical clinical trials) a formula which I think boils down to it being:

$$\sigma_{\theta_2 | \text{data}}^2 = \sigma_{02}^2 (1 - \rho^2 \frac{\sigma_{\text{post}}^2}{\sigma_{01}^2})$$

However I can't for the life of me derive it. Can anyone help please?

The posterior precision matrix of $$(\theta_1,\theta_2)$$ can be derived by adding the prior precision matrix and the precision matrix associated with the likelihood. However, the following alternative derivation will perhaps give more insight into what is happening when the observed data only depends on $$\theta_1$$.

To find the conditional variance of $$\theta$$ given the data, consider the law of total variance $$\operatorname{Var}(\theta_2)=E\operatorname{Var}(\theta_2|\theta_1)+\operatorname{Var}E(\theta_2|\theta_1) \tag{1}$$ Like all probability laws, this also holds if conditioning everywhere on any third event or random variable. Thus, $$\operatorname{Var}(\theta_2|\text{data})=E(\operatorname{Var}(\theta_2|\theta_1,\text{data})|\text{data})+\operatorname{Var}(E(\theta_2|\theta_1,\text{data})|\text{data}). \tag{2}$$ Now, since the data and $$\theta_2$$ are conditionally independent given $$\theta_1$$, it follows that $$\operatorname{Var}(\theta_2|\theta_1,\text{data})=\operatorname{Var}(\theta_2|\theta_1)=\sigma_{02}^2(1-\rho^2) \tag{3}$$
(see wikipedia for the last equality).

Similarly, for the same reasons, $$E(\theta_2|\theta_1,\text{data})=E(\theta_2|\theta_1)=\theta_{02}+\rho\frac{\sigma_{02}}{\sigma_{01}}(\theta_1-\theta_{01}). \tag{4}$$ Substituting (3) and (4) into (2) yields \begin{align} \operatorname{Var}(\theta_2|\text{data}) &=E(\sigma_{02}^2(1-\rho^2)|\text{data}) + \operatorname{Var}(\theta_{02}+\rho\frac{\sigma_{02}}{\sigma_{01}}(\theta_1-\theta_{01})|\text{data}) \\&=\sigma_{02}^2(1-\rho^2) + \rho^2\frac{\sigma_{02}^2}{\sigma_{01}^2} \operatorname{Var}(\theta_1|\text{data}) \\&=\sigma_{02}^2(1-\rho^2) + \rho^2\frac{\sigma_{02}^2}{\sigma_{01}^2} \sigma_\text{post}^2 \\&=\sigma_{02}^2(1-\rho^2 + \rho^2\frac{\sigma_\text{post}^2}{\sigma_{01}^2}). \tag{5} \end{align}

As sanity checks, consider the cases of $$\sigma_\text{post}^2=\sigma_{01}^2$$ and $$\sigma_\text{post}^2\rightarrow 0$$ for which (5) produces the expected results.

So I believe your final formula for the posterior variance of $$\theta_2$$ can't be correct.

• I'm going to upvote as soon as my votes reset! Amazing thank you! Commented Jul 12 at 12:56
• Can I ask one clarifying question? Generalising to the multivariate setting, i.e., for $\theta_p$ where $p > 2$. Suppose we have just two observations $(y_1, y_2, \text{missing})$ from a trivariate multivariate normal distribution $(\theta_1, \theta_2, \theta_3)$ with a known correlations - what would be the precision matrix of the likelihood when no data from $\theta_3$ is unobserved? Commented Jul 12 at 14:18
• If each observation $\mathbf{x}\sim N(\boldsymbol\theta_{1:2},\boldsymbol\Sigma)$ you will see (from rules of blockwise matrix multiplication) that the likelihood can be rewritten as something involving a quadratic form in the vector $\boldsymbol\theta_{1:3}$ and a $3\times 3$ precision matrix with $\boldsymbol\Sigma^{-1}$ as its upper left block with zeroes in the third row and column. Commented Jul 12 at 14:55
• Thanks that's amazing. One final question: I'm a bit confused about this statement: $\operatorname{Var}(\theta_2|\theta_1,\text{data})=\operatorname{Var}(\theta_2|\theta_1)=\sigma_{02}^2(1-\rho^2)$ Does this mean $\operatorname{Var}(\theta_2)$ is conditional on the data and the $\theta_1$ is only defined by the prior variance and the correlation (known)? That seems a strange result to me Commented Jul 12 at 16:00
• @statneutrino Yes, this follows from conditional independence of $\theta_2$ and the data given $\theta_1$. Commented Jul 12 at 17:06