Intuition for near-decorrelation through centering

Question

Consider a $p\times 1$ random vector $\mathbf u = (u_1,...,u_p)'$ with zero mean vector and variance-covariance matrix

$$E(\mathbf u \mathbf u')\equiv \mathbf{\Sigma}=\sigma^2((1-\rho)I_p+\rho\mathbf i \mathbf i')$$

where $I_p$ is the identity matrix and $\mathbf i = (1,...,1)'$

Namely the elements of $\mathbf u$ are equicorrelated with correlation coefficient $\rho \neq 0$ and have common variance $\sigma^2>0$. We assume that $-\frac{1}{p-1} < \rho < 1$ so that $\Sigma$ is positive definite.

Consider "centering the vector on its own self", which is a made-up description for the variables

$$\tilde u_i = u_i - \frac 1p\sum_{j=1}^p u_j$$

In matrix notation, by using the idempotent and symmetric $p \times p$ matrix

$$\mathbf {M_i} = I_p - \frac 1p \mathbf i \mathbf i'$$

we have

$$\mathbf {\tilde u} = \mathbf {M_i} \mathbf u$$

The variance covariance matrix of the centered vector is

$$E(\mathbf {\tilde u} \mathbf {\tilde u}')=E(\mathbf {M_i} \mathbf u \mathbf u'\mathbf {M_i}) =\mathbf {M_i}\sigma^2((1-\rho)I_p+\rho\mathbf i \mathbf i')\mathbf {M_i}$$

We have $\mathbf {M_i} \mathbf i = 0$ so we arrive at

$$E(\mathbf {\tilde u} \mathbf {\tilde u}') = (1-\rho)\sigma^2\mathbf {M_i}$$

This is a singular matrix (with rank $p-1$), but my issue is something else: It is not difficult to calculate that

$$E(u_i^2) = (1-1/p)(1-\rho)\sigma^2,\;\; E(u_iu_j) = -(1/p)(1-\rho)\sigma^2,\;\; \tilde \rho = \frac {-1}{p-1}$$

So the centered variables are still equicorrelated but now the correlation coefficient is $\tilde \rho = \frac {-1}{p-1}$: irrespective of the initial correlation direction and strength, the correlation here is negative and depends only on the dimension of the random vector, and in a way that, even for small values of $p$, correlation is small to negligible (say, for $p=20 \implies \tilde \rho \approx -0.0527$). And we achieved that by centering.

I admit I did not expect such sweeping changes in the correlation between random variables by using centering, perhaps because I am used to see often arguments and presentations where "location does not matter", for various results to obtain.

The math is clear, but is there any intuition as to why this kind of centering nearly de-correlates the variables here?

Answer 1

Here is a geometric argument that I think provides the required intuition.

Let us consider a $n\times p$ data matrix $\mathbf U$, where each row is one sample of your random variable $\mathbf u$. Usual "centering" refers to centering the columns of $\mathbf U$; as your $\mathbf u$ has zero mean, the data matrix $\mathbf U$ will be approximately centered in this sense.

Your operation makes row means (as opposed to column means) equal to zero; so I will call it "row-centering".

Now, consider columns of $\mathbf U$ as vectors in $\mathbb R^n$. Each of these vectors corresponds to one of the $p$ variables (components of $\mathbf u$). Assuming that $\mathbf U$ is [column-]centered, the length of each vector is equal to the variance of this variable and the cosine of the angle between any two vectors is equal to the correlation between them. This $n$-dimensional geometric view is standard in linear regression, PCA/FA, etc.

---

This was setting-up. Now comes the argument.

So what you have before row-centering is $p$ vectors of equal length and the same angle $\mathrm{arccos}(p)$ between any two of them. The end-points of these vectors form a cloud of points; this cloud lies entirely in one "hyper-quadrant" of $\mathbb R^n$ (because the angles are all below $90^\circ$).

When you do row-centering, you are centering this cloud of points in the usual sense. So you take this cloud of points and shift to zero. Now all vectors, instead of pointing in the sort-of-same direction, are suddenly pointing in various directions away from zero. In other words, correlations become close to $0$.

To get a better idea of it, consider what happens when $p=2$. There are only two points, so after centering the angle between them is $180^\circ$, corresponding to correlation $-1$, as your formula says. For $p=3$ there are three points forming a perfect triangle; after centering the angles are $120^\circ$, corresponding to correlation $-1/2$. Etc. For larger $p$ you will quickly get to near-zero correlations.

(There really should have been a figure here, but I don't have the time for that now.)