Consider a $p\times 1$ random vector $\mathbf u = (u_1,...,u_p)'$ with zero mean vector and variance-covariance matrix

$$E(\mathbf u \mathbf u')\equiv \mathbf{\Sigma}=\sigma^2((1-\rho)I_p+\rho\mathbf i \mathbf i')$$

where $I_p$ is the identity matrix and $\mathbf i = (1,...,1)'$

Namely the elements of $\mathbf u$ are equicorrelated with correlation coefficient $\rho \neq 0$ and have common variance $\sigma^2>0$. We assume that $-\frac{1}{p-1} < \rho < 1$ so that $\Sigma$ is positive definite.

Consider "centering the vector on its own self", which is a made-up description for the variables

$$\tilde u_i = u_i - \frac 1p\sum_{j=1}^p u_j$$

In matrix notation, by using the idempotent and symmetric $p \times p$ matrix

$$\mathbf {M_i} = I_p - \frac 1p \mathbf i \mathbf i'$$

we have

$$\mathbf {\tilde u} = \mathbf {M_i} \mathbf u$$

The variance covariance matrix of the centered vector is

$$E(\mathbf {\tilde u} \mathbf {\tilde u}')=E(\mathbf {M_i} \mathbf u \mathbf u'\mathbf {M_i}) =\mathbf {M_i}\sigma^2((1-\rho)I_p+\rho\mathbf i \mathbf i')\mathbf {M_i}$$

We have $\mathbf {M_i} \mathbf i = 0$ so we arrive at

$$E(\mathbf {\tilde u} \mathbf {\tilde u}') = (1-\rho)\sigma^2\mathbf {M_i}$$

This is a singular matrix (with rank $p-1$), but my issue is something else: It is not difficult to calculate that

$$E(u_i^2) = (1-1/p)(1-\rho)\sigma^2,\;\; E(u_iu_j) = -(1/p)(1-\rho)\sigma^2,\;\; \tilde \rho = \frac {-1}{p-1}$$

So the centered variables are still equicorrelated but now the correlation coefficient is $\tilde \rho = \frac {-1}{p-1}$: irrespective of the initial correlation direction and strength, the correlation here is negative and depends only on the dimension of the random vector, and in a way that, even for small values of $p$, correlation is small to negligible (say, for $p=20 \implies \tilde \rho \approx -0.0527$). And we achieved that by centering.

I admit I did not expect such sweeping changes in the correlation between random variables by using centering, perhaps because I am used to see often arguments and presentations where "location does not matter", for various results to obtain.

The math is clear, but is there any intuition as to why this kind of centering nearly de-correlates the variables here?

  • 1
    $\begingroup$ It's unclear why you describe this initial operation as "centering the vector on its own self." Since it has zero mean, it is already centered. $\endgroup$ – whuber Apr 22 '16 at 14:25
  • $\begingroup$ @whuber This operation transforms a vector $\mathbf u = (u_1,...,u_p)'$ into $\tilde{\mathbf u} = (u_1-\bar u,...,u_p - \bar u)'$, where $\bar u=\sum u_i/p$. It's sort of centering "across dimensions". Or if one thinks of a data matrix, then it's centering the rows (not columns). $\endgroup$ – amoeba Apr 22 '16 at 15:20
  • $\begingroup$ Yes, that's obvious. But I think your question might stem from confusing that with the more common senses of centering. $\endgroup$ – whuber Apr 22 '16 at 16:11
  • $\begingroup$ @whuber Well, it is not centering in the sense of $\tilde x = x-E(x)$ but it is a form of centering, since the tilde variables describe deviations from the group's average value. I still think that the effect on the correlation structure is remarkable. $\endgroup$ – Alecos Papadopoulos Apr 22 '16 at 17:13
  • $\begingroup$ @amoeba Yes exactly, thank you, I added that to the body of the question for total clarity. $\endgroup$ – Alecos Papadopoulos Apr 22 '16 at 17:14

Here is a geometric argument that I think provides the required intuition.

Let us consider a $n\times p$ data matrix $\mathbf U$, where each row is one sample of your random variable $\mathbf u$. Usual "centering" refers to centering the columns of $\mathbf U$; as your $\mathbf u$ has zero mean, the data matrix $\mathbf U$ will be approximately centered in this sense.

Your operation makes row means (as opposed to column means) equal to zero; so I will call it "row-centering".

Now, consider columns of $\mathbf U$ as vectors in $\mathbb R^n$. Each of these vectors corresponds to one of the $p$ variables (components of $\mathbf u$). Assuming that $\mathbf U$ is [column-]centered, the length of each vector is equal to the variance of this variable and the cosine of the angle between any two vectors is equal to the correlation between them. This $n$-dimensional geometric view is standard in linear regression, PCA/FA, etc.

This was setting-up. Now comes the argument.

So what you have before row-centering is $p$ vectors of equal length and the same angle $\mathrm{arccos}(p)$ between any two of them. The end-points of these vectors form a cloud of points; this cloud lies entirely in one "hyper-quadrant" of $\mathbb R^n$ (because the angles are all below $90^\circ$).

When you do row-centering, you are centering this cloud of points in the usual sense. So you take this cloud of points and shift to zero. Now all vectors, instead of pointing in the sort-of-same direction, are suddenly pointing in various directions away from zero. In other words, correlations become close to $0$.

To get a better idea of it, consider what happens when $p=2$. There are only two points, so after centering the angle between them is $180^\circ$, corresponding to correlation $-1$, as your formula says. For $p=3$ there are three points forming a perfect triangle; after centering the angles are $120^\circ$, corresponding to correlation $-1/2$. Etc. For larger $p$ you will quickly get to near-zero correlations.

(There really should have been a figure here, but I don't have the time for that now.)

  • $\begingroup$ A figure would certainly help, but you have provided verbal description with enough detail for one to be able to visualize this. Thanks. $\endgroup$ – Alecos Papadopoulos Apr 23 '16 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.