10
$\begingroup$

Assuming I have two non-independent random variables and I want to reduce covariance between them as much as possible without loosing too much "signal", does mean centering help? I read somewhere that mean centering reduces correlation by a significant factor, so I'm thinking it should do the same for covariance.

$\endgroup$
28
$\begingroup$

If $X$ and $Y$ are random variables and $a$ and $b$ are constants, then $$ \begin{aligned} \operatorname{Cov}(X + a, Y + b) &= E[(X + a - E[X + a])(Y + b - E[Y + b])] \\ &= E[(X + a - E[X] - E[a])(Y + b - E[Y] - E[b])] \\ &= E[(X + a - E[X] - a)(Y + b - E[Y] - b)] \\ &= E[(X - E[X])(Y - E[Y])] \\ &= \operatorname{Cov}(X, Y). \end{aligned} $$ Centering is the special case $a = -E[X]$ and $b = -E[Y]$, so centering does not affect covariance.


Also, since correlation is defined as $$ \operatorname{Corr}(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sqrt{\operatorname{Var}(X) \operatorname{Var}(Y)}}, $$ we can see that $$ \begin{aligned} \operatorname{Corr}(X + a, Y + b) &= \frac{\operatorname{Cov}(X + a, Y + b)}{\sqrt{\operatorname{Var}(X + a) \operatorname{Var}(Y + b)}} \\ &= \frac{\operatorname{Cov}(X, Y)}{\sqrt{\operatorname{Var}(X) \operatorname{Var}(Y)}}, \end{aligned} $$ so in particular, correlation isn't affected by centering either.


That was the population version of the story. The sample version is the same: If we use $$ \widehat{\operatorname{Cov}}(X, Y) = \frac{1}{n} \sum_{i=1}^n \left(X_i - \frac{1}{n}\sum_{j=1}^n X_j\right)\left(Y_i - \frac{1}{n}\sum_{j=1}^n Y_j\right) $$ as our estimate of covariance between $X$ and $Y$ from a paired sample $(X_1,Y_1), \ldots, (X_n,Y_n)$, then $$ \begin{aligned} \widehat{\operatorname{Cov}}(X + a, Y + b) &= \frac{1}{n} \sum_{i=1}^n \left(X_i + a - \frac{1}{n}\sum_{j=1}^n (X_j + a)\right)\left(Y_i + b - \frac{1}{n}\sum_{j=1}^n (Y_j + b)\right) \\ &= \frac{1}{n} \sum_{i=1}^n \left(X_i + a - \frac{1}{n}\sum_{j=1}^n X_j - \frac{n}{n} a\right)\left(Y_i + b - \frac{1}{n}\sum_{j=1}^n Y_j - \frac{n}{n} b\right) \\ &= \frac{1}{n} \sum_{i=1}^n \left(X_i - \frac{1}{n}\sum_{j=1}^n X_j\right)\left(Y_i - \frac{1}{n}\sum_{j=1}^n Y_j\right) \\ &= \widehat{\operatorname{Cov}}(X, Y) \end{aligned} $$ for any $a$ and $b$.

$\endgroup$
  • $\begingroup$ thanks for the detailed answer. Does it mean that for sample covariance the sample size doesn't have any impact either? i.e. reducing the sample size does not reduce the sample covariance? $\endgroup$ – lvdp Feb 11 at 5:45
  • 3
    $\begingroup$ @lvdp That should probably be a separate question. $\endgroup$ – Acccumulation Feb 11 at 16:24
  • $\begingroup$ A reduced sample size can only come with a different sample. A different sample could show different covariance, therefore. But as sample covariance is defined as an average, sample size is scaled for in principle. $\endgroup$ – Nick Cox Feb 11 at 17:00
5
$\begingroup$

The definition of the covariance of $X$ and $Y$ is $E[(X-E[X])(Y-E[Y])]$. The expression $X-E[X]$ in that formula is the centered version of $X$. So we already center $X$ when we take the covariance, and centering is an idempotent operator; once a variable is centered, applying the centering process further times doesn't change it. If the formula didn't take the centered versions of the variables, then there would all sort of weird effects, such as the covariance between temperature and another variable being different depending on whether we measure temperature in Celsius or Kelvin.

$\endgroup$
3
$\begingroup$

"somewhere" tends to be a rather unreliable source...

Covariance/correlation are defined with explicit centering. If you don't center the data, then you are not computing covariance/correlation. (Precisely: Pearson correlation)

The main difference is whether you center based on a theoretical model (e.g., the expected value is supposed to be exactly 0) or based on the data (arithmetic mean). It is easy to see that the arithmetic mean will yield smaller Covariance than any different center.

However, smaller covariance does not imply smaller correlation, or the opposite. Assume that we have data X=(1,2) and Y=(2,1). It is easy to see that with arithmetic mean centering this will yield perfectly negative correlation, while if we know the generating process produces 0 on average, the data is actually positively correlated. So in this example, we are centering - but with the theoretical expected value of 0.

This can arise easily. Consider we have a sensor array, 11x11, with the cells numbered -5 to +5. Rather than taking the arithmetic mean, it does make sense to use the "physical" mean of our sensor array here when looking for the correlation of sensor events (if we enumerated the cells 0 to 10, we'd use 5 as fixed mean, and we would get the exact same results, so that indexing choice disappears from the analysis - nice).

$\endgroup$
  • $\begingroup$ Thanks @Anony-Mousse, will the sample covariance depend on the sample size? I.e. smaller sample size will yield smaller covariance ( before centering). $\endgroup$ – lvdp Feb 11 at 14:01
  • 1
    $\begingroup$ Depends on the sample obviously. On average - I don't know. I'd expect smaller samples to have more variability mostly, so maybe more often more extreme values. But that is just an intuition. $\endgroup$ – Anony-Mousse Feb 12 at 1:48

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.