Multivariate bayesian inference: learning about the mean of a variable by observing another variable

Question

I want to derive a Bayesian learning procedure where I don't only learn from my own signal, but also from other signals which are correlated to mine.

I thought it could simply work with Bayesian inference about the mean of a multivariate normal distribution (with known covariance matrix $\Sigma$), but I'm quite puzzled by the result. As you may know, if one start from a prior $\mu_0, \Sigma_0$, the posterior will be given by (cf this thread):

$\mu_n = \Sigma_0 \Big( \Sigma_0 + 1/n \Sigma \Big)^{-1} \Big( 1/n \sum^n_{i=1} \textbf{x}_i \Big) + 1/n \Sigma \Big( \Sigma_0 + 1/n \Sigma \Big)^{-1} \mu_0$
$\Sigma_n = \Sigma_0 \Big( \Sigma_0 + 1/n \Sigma \Big)^{-1} 1/n \Sigma$

Now, what puzzles me is that, if I start from a prior $\Sigma_0 = \Sigma$, I end up in a situation where I only learn about the mean of the first variable using observations of the first variable!
Indeed: in this case, $\Sigma_0 + 1/n \Sigma = (n+1)/n \Sigma$, hence $\Sigma_0 \Big( \Sigma_0 + 1/n \Sigma \Big)^{-1} = n/(n+1) Id$. I don't get why one does not use the additional information contained in correlated signals.

It's pretty much the same if one start from uninformed prior $\Sigma_0 (\rightarrow \infty)$ since after one iteration, $\Sigma_n \rightarrow \Sigma$.

So I don't really get the intuition behind it. Imagine $x$ is just bivariate $= (x_1, x_2)'$. Since I know that $x_2$ is correlated to $x_1$, why can't I use the observation of $x_2$ to learn about the mean of $x_1$? I'll just learn "less perfectly" than by observing my own signal.
By the way in the extreme case where I assume that $\Sigma_0$ is a matrix full of $1$ (or any constant) (i.e. correlation $=1$), then I end up learning as well from each variables (as if I only had one variable). But this get destroyed as soon as the covariance $\neq$ variance in the prior.

There's probably something I'm missing here, but I don't know what. I think maybe because the covariance is `net of the mean', and hence it does not help or something like that.

By the way, since Bayesian inference about a multivariate normal does not seem to be the correct way to go, do you know of another example where one learns about the mean of a variable $x_1$ by observing another variable $x_2$ that is only correlated to the first one?

Answer 1

I think it would be good to have the notations cleared first. $$ \begin{aligned} \vec{\mu} &\sim \mathcal{N}(\vec{\mu}_0, \Sigma_0) &&\textrm{prior},\\ \vec{x}_i \vert \vec{\mu} &\sim \mathcal{N}(\vec{\mu}, \Sigma) &&\textrm{likelihood},\\ \vec{\mu} \vert \{\vec{x}_i\} &\sim \mathcal{N}(\vec{\mu}_n, \Sigma_n) &&\textrm{posterior},\\ \end{aligned} $$ where $\vec{x}_i$ is the $i$-th observation with $\vec{x}_i=(y_1, y_2, \dots)$.

One can see that the $\Sigma_0$ and $\Sigma$ correspond to two different distributions. $\Sigma_0$ describes the variation of $\vec{\mu}$ about $\vec{\mu}_0$, while $\Sigma$ describes the variation of $\vec{x}_i$ about $\vec{\mu}$.

For simplicity, let's set $\vec{\mu}_0 = \vec{0}$ and $\Sigma_0 = I$, the resulting posterior distribution's mean and covariance matrix are given by $$ \begin{aligned} \vec{\mu}_n &= \left(I + \frac{1}{n}\Sigma\right)^{-1}\left(\frac{1}{n}\sum_{i=1}^{n}\vec{x}_i\right)\\ &\equiv \left(I + \frac{1}{n}\Sigma\right)^{-1} \langle \vec{x} \rangle,\\ \Sigma_n &= \left(I + \frac{1}{n}\Sigma\right)^{-1}\frac{1}{n}\Sigma. \end{aligned} $$

where $\langle \vec{x} \rangle$ is referring to the sample mean. We can see that the equation for $\vec{\mu}_n$ follows our intuition; as $n\to\infty$, $\vec{\mu}_n = \vec{\mu}_{\infty} = \langle \vec{x} \rangle$. But what is the matrix $\left(I + \frac{1}{n}\Sigma\right)^{-1}$ actually doing? Let's take a look at a two dimensional example;

$\vec{\mu}_n$" />

The plot shows the evolution of the $\vec{\mu}_n$ (in orange) as $n$ increases up to 10 (with the initial $\vec{0}$ added as the starting point), the evolution of the sample mean (in blue), and the true value is marked with the purple cross. Therefore, to go from $\vec{\mu}_0$ to $\vec{\mu}_1$, the $\langle\vec{x}\rangle_1$ is taken as the input. Because the relatively small number of samples is used, it has not converged to the true value yet, but this is sufficient for a demonstration. This plot shows an interesting feature of the matrix term. Instead of jumping right towards the current sample mean, the direction is alternated to counter the known correlation $\Sigma$ between the components (as shown by the dashed line).

Therefore, there has never been an update of the mean based on the correlation between components; instead, it is a correction, so we don't obtain a biased result. And if the prior correlation on $\Sigma_0$ is the same as the likelihood correlation $\Sigma$, it simply means that no directional correction is needed at all.

This finding echoes the linearity of mean, therefore $$ \langle \vec{x} \rangle = (\langle y_1 \rangle, \langle y_2 \rangle, \dots), $$ regardless of any correlation between components of a random variable, the mean of each component is independent of each other.

Answer 2

This might be slightly counterintuitive at first, but the fact that two variables are correlated doesn't mean that you can learn about the mean of one from another.

Suppose for example that $x_1$ is normally distributed with mean $0$ and $x_2 = x_1 + 100$. $x_1$ and $x_2$ are clearly maximally correlated, but you can learn nothing about the mean of $x_2$ (which is 100) from $x_1$.

That is of course unless you have some prior knowledge about the correlation between the means (your $\Sigma_0$). Indeed in that case you get the 'mix' that you expect between the two variables. The fact that this doesn't happen in the case $\Sigma_0=\Sigma$ might be a bit surprising, but as you pointed out this is equivalent to the covariance of the posterior with a non-informative prior, so in a way this doesn't add new information about the correlation between the means that is not already contained in the likelihood function.

So in short, to learn about the mean of one variable from another requires you to have some additional prior knowledge on their correlation.