More correlated information is... more informative?

Question

Suppose I am trying to make inference about a parameter $\mu$. I have a prior $$ \mu \sim N(0,\sigma^2), $$ and I observe two correlated signals about $\mu,$ namely $x_1, x_2$ where $$ \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = N\left(\begin{pmatrix} \mu \\ \mu \end{pmatrix}, \begin{pmatrix} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \end{pmatrix}\right), $$ so $x_1$ and $x_2$ are just "noised up" $\mu$. One can show that the posterior for $\mu$ conditional on $x_1$ and $x_2$ is given by $$ \mu \mid x_1, x_2 \sim N\left(\frac{\tau_1x_1 - \rho\sqrt{\tau_1\tau_2}(x_1+x_2) + \tau_2x_2}{\tau_1 - 2\rho\sqrt{\tau_1\tau_2} + \tau_2 + (1-\rho^2)\tau_\mu}, \frac{1-\rho^2}{\tau_1 - 2\rho\sqrt{\tau_1\tau_2} + \tau_2 + (1-\rho^2)\tau_\mu}\right), $$ where $\tau$ is the precision. I want to know how "informative" $x_1$ and $x_2$ are as a function of $\rho$. I'm going to define the measure of informativeness as the expected reduction in the mean-squared error of our estimate for $\mu$.* Because we want to minimize the MSE, we'll take the estimator for $\mu$ to be the posterior mean (see here).

Now, my intuition (which I guess is wrong) would be that independent signals would be the most informative about the true value of $\mu$ since correlated signals contain "overlapping" information. But we see that as $\lvert\rho\rvert\to1$ the variance on the right hand side goes to $0,$ in other words, the more correlated the signals are, the better our posterior seems to be. What's going on here? Is my intuition wrong, or is my interpretation of the result wrong?

The best "secondary" intuition I can come up with is that perhaps the correlation is allowing us to pinpoint $\mu$. I know that as $\lvert \rho \rvert \to 1,$ the values of $x_2$ become concentrated around the line $x_2 = \mu + \tfrac{\sigma_2}{\sigma_1}(x_1-\mu)$ (see here), so of course if we observed both $x_1$ and $x_2$ we would be able to back out the value of $\mu$ from this.

Finally, if this is true (i.e. that correlated signals here lead to more accurate estimates), what is the minimal change we could make to the setup so that more correlation leads to a lower quality posterior estimate for $\mu$?

*If this notion of informativeness seems unintuitive to you, the same conclusions can be drawn if we consider instead the mutual information between $\mu$ and $x_1, x_2$.

Answer 1

You noticed that the question can be made more precise if we use a precise definition of "information" from the information theory. Let's take that route. Recall that the entropy for multivariate normal distribution is

$$ H(x) = \frac{n}{2} \ln(2\pi) + \frac{1}{2} \ln|\Sigma| + \frac{1}{2} n $$

It depends on the determinant of the covariance matrix $|\Sigma|$, which squashes it to a single number. Hopefully, the determinant for a $2\times 2$ matrix, such as the covariance of the bivariate normal distribution is trivial to calculate

$$ \begin{vmatrix} a & b\\c & d \end{vmatrix}=ad-bc $$

hence it is just $\sigma^2_1\sigma^2_2 - \rho^2$. Now, if you keep everything else constant and treat it as a function of $\rho$, the only moving part is $-\rho^2$, hence the entropy decreases as $|\rho|$ increases. The more correlated the variables are, the less information they carry.

Now, for an intuition about this, consider two cases. First, when the correlation is perfect $\rho=1$, then $x_1 = x_2$. Second, when $\rho=0$, $x_1$ and $x_2$ can be any values on the real line. The average is a linear combination, so it will fall somewhere in between $x_1$ and $x_2$. In the first case, imagine that $x_1$ is higher than $\mu$, then the mean would be equally higher than $\mu$. In the second case, if $x_1$ is higher than $\mu$, $x_2$ can be any value, it can be even higher, slightly less high, less, etc, so $x_2$ has a chance to "correct" the $x_1$ being high.

Even more practically, this is the reason why we prefer to draw our samples randomly. Non-random samples would be correlated, hence more biased, and less "informative".