Suppose I am trying to make inference about a parameter $$\mu$$. I have a prior $$\mu \sim N(0,\sigma^2),$$ and I observe two correlated signals about $$\mu,$$ namely $$x_1, x_2$$ where $$\begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = N\left(\begin{pmatrix} \mu \\ \mu \end{pmatrix}, \begin{pmatrix} \sigma_1^2 & \rho\sigma_1\sigma_2 \\ \rho\sigma_1\sigma_2 & \sigma_2^2 \end{pmatrix}\right),$$ so $$x_1$$ and $$x_2$$ are just "noised up" $$\mu$$. One can show that the posterior for $$\mu$$ conditional on $$x_1$$ and $$x_2$$ is given by $$\mu \mid x_1, x_2 \sim N\left(\frac{\tau_1x_1 - \rho\sqrt{\tau_1\tau_2}(x_1+x_2) + \tau_2x_2}{\tau_1 - 2\rho\sqrt{\tau_1\tau_2} + \tau_2 + (1-\rho^2)\tau_\mu}, \frac{1-\rho^2}{\tau_1 - 2\rho\sqrt{\tau_1\tau_2} + \tau_2 + (1-\rho^2)\tau_\mu}\right),$$ where $$\tau$$ is the precision. I want to know how "informative" $$x_1$$ and $$x_2$$ are as a function of $$\rho$$. I'm going to define the measure of informativeness as the expected reduction in the mean-squared error of our estimate for $$\mu$$.* Because we want to minimize the MSE, we'll take the estimator for $$\mu$$ to be the posterior mean (see here).

Now, my intuition (which I guess is wrong) would be that independent signals would be the most informative about the true value of $$\mu$$ since correlated signals contain "overlapping" information. But we see that as $$\lvert\rho\rvert\to1$$ the variance on the right hand side goes to $$0,$$ in other words, the more correlated the signals are, the better our posterior seems to be. What's going on here? Is my intuition wrong, or is my interpretation of the result wrong?

The best "secondary" intuition I can come up with is that perhaps the correlation is allowing us to pinpoint $$\mu$$. I know that as $$\lvert \rho \rvert \to 1,$$ the values of $$x_2$$ become concentrated around the line $$x_2 = \mu + \tfrac{\sigma_2}{\sigma_1}(x_1-\mu)$$ (see here), so of course if we observed both $$x_1$$ and $$x_2$$ we would be able to back out the value of $$\mu$$ from this.

Finally, if this is true (i.e. that correlated signals here lead to more accurate estimates), what is the minimal change we could make to the setup so that more correlation leads to a lower quality posterior estimate for $$\mu$$?

*If this notion of informativeness seems unintuitive to you, the same conclusions can be drawn if we consider instead the mutual information between $$\mu$$ and $$x_1, x_2$$.

• Do your assertions still apply when $\sigma_1^2=\sigma_2^2$ and $\rho \to +1$? if not, is there something in the way you have set up the question that causes this curiosity? Jul 4 at 2:10
• When $\sigma_1^2 = \sigma_2^2$ we have $\mathrm{Var}(\mu \mid x_1, x_2) = \frac{1+\rho}{2\tau + (1+\rho)\tau_\mu},$ and so in this case you are right that as $\rho\to1,$ my concern disappears– in fact, the variance is increasing in $\rho$ between $0$ and $1,$ but as $\rho\to-1$ we have the same problem I describe in my question. When you say is there something in the way I've set it up that causes this, do you have something in mind? I mean, I am wanting to analyze what happens when $\sigma_1 \neq \sigma_2,$ but even when they're the same unexpected stuff is happening.
– deej
Jul 4 at 2:58
• Negative correlation is intuitively desirable: if one sample being above the unknown mean results in the other being below the mean, then a convex combination of the samples could lead to a substantially better estimate of the mean Jul 4 at 9:03
• I don't see how negative correlation is intuitively desirable. Practically speaking it's no different to positive correlation since if the variables were positively correlated you could just multiply one of them by -1 to get a negative correlation, right?
– deej
Jul 5 at 1:31
• Not really, as multiplying one of them by $-1$ would change its mean which disrupts finding $\mu$. A natural estimator of $\mu$ is $\frac{x_1+x_2}{2}$. If you know $\frac{\sigma_1^2}{\sigma_2^2}$ then a better estimator of $\mu$ may be the weighted means $\frac{x_1+x_2\frac{\sigma_1}{\sigma_2}}{1+\frac{\sigma_1}{\sigma_2}}$ and $\frac{x_1+x_2\frac{\sigma_1^2}{\sigma_2^2}}{1+\frac{\sigma_1^2}{\sigma_2^2}}$ though these are the same when $\sigma_1^2 = \sigma_2^2$. They get better as estimators as the correlation becomes more negative Jul 5 at 8:17

You noticed that the question can be made more precise if we use a precise definition of "information" from the information theory. Let's take that route. Recall that the entropy for multivariate normal distribution is

$$H(x) = \frac{n}{2} \ln(2\pi) + \frac{1}{2} \ln|\Sigma| + \frac{1}{2} n$$

It depends on the determinant of the covariance matrix $$|\Sigma|$$, which squashes it to a single number. Hopefully, the determinant for a $$2\times 2$$ matrix, such as the covariance of the bivariate normal distribution is trivial to calculate

$$\begin{vmatrix} a & b\\c & d \end{vmatrix}=ad-bc$$

hence it is just $$\sigma^2_1\sigma^2_2 - \rho^2$$. Now, if you keep everything else constant and treat it as a function of $$\rho$$, the only moving part is $$-\rho^2$$, hence the entropy decreases as $$|\rho|$$ increases. The more correlated the variables are, the less information they carry.

Now, for an intuition about this, consider two cases. First, when the correlation is perfect $$\rho=1$$, then $$x_1 = x_2$$. Second, when $$\rho=0$$, $$x_1$$ and $$x_2$$ can be any values on the real line. The average is a linear combination, so it will fall somewhere in between $$x_1$$ and $$x_2$$. In the first case, imagine that $$x_1$$ is higher than $$\mu$$, then the mean would be equally higher than $$\mu$$. In the second case, if $$x_1$$ is higher than $$\mu$$, $$x_2$$ can be any value, it can be even higher, slightly less high, less, etc, so $$x_2$$ has a chance to "correct" the $$x_1$$ being high.

Even more practically, this is the reason why we prefer to draw our samples randomly. Non-random samples would be correlated, hence more biased, and less "informative".

• I agree that if we restrict our attention just to the multivariate normal distribution that higher correlation $\implies$ more entropy and therefore more information, but it still seems like more correlated signals lead to a more precise estimate of $\mu$ and I'm not sure how to reconcile that with what you've said. I agree with your intuition, but again, as $\rho\to 1,$ the posterior variance goes to $0,$ so does this not mean we have pinned down the estimate? Also, I think the determinant should be $\sigma_1^2\sigma_2^2(1-\rho^2),$ but your reasoning still works.
– deej
Jul 5 at 1:22
• Perhaps there's a distinction to be made about the informational content in the realisation of $x_1$ and $x_2$ which, as you've noted, is captured by the entropy, versus the information that this realization tells us about the parameter $\mu$. These are different things, no? That's why I was using mutual information rather than entropy.
– deej
Jul 5 at 1:29