How to Derive the Conditional Variance in a Bivariate Normal Distribution After Bayesian Updating?

Question

I'm working with a bivariate normal distribution of two variables, $\theta_1$ and $\theta_2$ in a Bayesian framework, with an intial joint prior distribution defined as:

$$\begin{pmatrix} \theta_1 \\ \theta_2 \end{pmatrix} \sim \mathcal{N}\left( \begin{pmatrix} \theta_{01} \\ \theta_{02} \end{pmatrix}, \begin{pmatrix} \sigma_{01}^2 & \rho \sigma_{01} \sigma_{02} \\ \rho \sigma_{01} \sigma_{02} & \sigma_{02}^2 \end{pmatrix} \right)$$

Importantly, I want to assume that the correlation $\rho$ is a known and fixed quantity/constant.

After observing $N$ data points related to $\theta_1$, I have updated the variance of $\theta_1$ using Bayesian principles, resulting in a new posterior variance $\sigma_{\text{post}}^2$:

$$ \sigma_{\text{post}}^2 = \left(\frac{1}{\sigma_{01}^2} + \frac{N}{\sigma^2}\right)^{-1} $$

where $\sigma^2$ is the variance of the observation errors.

My challenge is to accurately derive the conditional variance of $\theta_2$ given $\theta_1$ after this update. The traditional formula for the conditional variance in a bivariate normal distribution is:

$$\sigma_{\theta_2|\theta_1}^2 = \sigma_{02}^2 - \frac{(\rho \sigma_{01} \sigma_{02})^2}{\sigma_{01}^2}$$

Given that the variance of $\theta_1$ has been updated, how should I adjust this formula correctly? Specifically, how does the updated $\sigma_{\text{post}}^2$ impact the conditional variance of $\theta_2$?

It definitely can't be:

$$ \sigma_{02}^2(1-\rho^2) $$

Because if I only observe 1 observation for $\theta_1$, the variance drops down from $\sigma_{02}^2$ to $\sigma_{02}^2(1 - \rho^2)$, but if I observe a further 100 observations, it will make no difference!

I've read in a paper (related to surrogate endpoints in pharmaceutical clinical trials) a formula which I think boils down to it being:

$$\sigma_{\theta_2 | \text{data}}^2 = \sigma_{02}^2 (1 - \rho^2 \frac{\sigma_{\text{post}}^2}{\sigma_{01}^2})$$

However I can't for the life of me derive it. Can anyone help please?

Answer 1

The posterior precision matrix of $(\theta_1,\theta_2)$ can be derived by adding the prior precision matrix and the precision matrix associated with the likelihood. However, the following alternative derivation will perhaps give more insight into what is happening when the observed data only depends on $\theta_1$.

To find the conditional variance of $\theta$ given the data, consider the law of total variance $$ \operatorname{Var}(\theta_2)=E\operatorname{Var}(\theta_2|\theta_1)+\operatorname{Var}E(\theta_2|\theta_1) \tag{1} $$ Like all probability laws, this also holds if conditioning everywhere on any third event or random variable. Thus, $$ \operatorname{Var}(\theta_2|\text{data})=E(\operatorname{Var}(\theta_2|\theta_1,\text{data})|\text{data})+\operatorname{Var}(E(\theta_2|\theta_1,\text{data})|\text{data}). \tag{2} $$ Now, since the data and $\theta_2$ are conditionally independent given $\theta_1$, it follows that $$ \operatorname{Var}(\theta_2|\theta_1,\text{data})=\operatorname{Var}(\theta_2|\theta_1)=\sigma_{02}^2(1-\rho^2) \tag{3} $$
(see wikipedia for the last equality).

Similarly, for the same reasons, $$ E(\theta_2|\theta_1,\text{data})=E(\theta_2|\theta_1)=\theta_{02}+\rho\frac{\sigma_{02}}{\sigma_{01}}(\theta_1-\theta_{01}). \tag{4} $$ Substituting (3) and (4) into (2) yields \begin{align} \operatorname{Var}(\theta_2|\text{data}) &=E(\sigma_{02}^2(1-\rho^2)|\text{data}) + \operatorname{Var}(\theta_{02}+\rho\frac{\sigma_{02}}{\sigma_{01}}(\theta_1-\theta_{01})|\text{data}) \\&=\sigma_{02}^2(1-\rho^2) + \rho^2\frac{\sigma_{02}^2}{\sigma_{01}^2} \operatorname{Var}(\theta_1|\text{data}) \\&=\sigma_{02}^2(1-\rho^2) + \rho^2\frac{\sigma_{02}^2}{\sigma_{01}^2} \sigma_\text{post}^2 \\&=\sigma_{02}^2(1-\rho^2 + \rho^2\frac{\sigma_\text{post}^2}{\sigma_{01}^2}). \tag{5} \end{align}

As sanity checks, consider the cases of $\sigma_\text{post}^2=\sigma_{01}^2$ and $\sigma_\text{post}^2\rightarrow 0$ for which (5) produces the expected results.

So I believe your final formula for the posterior variance of $\theta_2$ can't be correct.