In all my self-study, I have come across many different ways in which people seem to normalize their data, prior to the computation of the covariance matrix. I am confused as to what ways are 'correct', or if is just application dependent.

For my particular application, I have a data matrix $X$, which is a $9$x$10000$ matrix. (Let $s=9$, and $N = 10000$). This matrix came from 9 sensors on the ground, which measured vibrations across time. I am tasked with looking at "how well" the different sensor outputs correlate with each other, in other words, studying the covariance matrix.

Option 1: The standard way I am aware of, is to subtract the mean of each row of $X$ from itself, and then compute the sample covariance matrix: Thus, let $\bar{X}$ be nothing but $X$, with the mean of each row subtracted out. Then, the covariance matrix is given by $C = (N-1)^{-1}\bar{X} \bar{X}^T$. In this way, the variance of each sensor output is unchanged, but all sensor outputs have zero mean.

Option 2: Another similar way I have seen, it not to simply stop at removing the mean, but also forcing the standard deviation of each sensor output to be 1, before computing $C$. So in this case, I would take my data matrix $X$, would remove the mean from each row, but then I would also divide each row by its own standard deviation, so that now, all rows/sensor outputs are zero mean, and unit variance. Then, we compute the Covariance matrix as before. Here of course, the covariance matrix will look different... so is this 'correct'?

Option 3: While this option is technically not about normalizing the data prior to computation of the covariance matrix, this method, (Pearson correlation coefficient) simply weights each element of the covariance matrix computed using (1), by the inverse of the product of standard deviations of the vectors it represents. In this case, my Pearson correlation matrix is going to have values between -1 and 1. Like (2), this also has the effect of "not caring" about the energy from each sensor.

Now, I am confused as to which method I need to use for this application... Do I simply demean my sensor data and keep their variances before computing $C$, as the textbook method? Do I demean and standardize all their variances to unity before computing $C$, as some machine learning texts do? Or do I look at the Pearson matrix as option (3), which also does a kind of normalizing by sigmas?

Very confused. :-/

  • $\begingroup$ @ttnphns Ah yes, that is true. Also taken from your own post, you said " "covariance shows only the strength of a linear relationship" This is not true. Cov is sensitive to both strength of linearity and to magnitude of variation." Does this mean then, that by standardizing as per options (2) and (3), we are throwing away information relating to the magnitude of variation, but keeping information relating to the linearity measure? $\endgroup$
    – Creatron
    Commented Jul 17, 2013 at 16:57
  • $\begingroup$ Options 2 and 3 both lead to the same result: Pearson correlation matrix. Because r is the covariance between z-standardized variables [rows, in your notation] and at the same time the cosine between centered variables (note that cov/(sigma1*sigma2) boils down to cosine). P.S. simply weights each element of the covariance matrix computed using (1), by the INVERSED product of standard deviations $\endgroup$
    – ttnphns
    Commented Jul 17, 2013 at 16:59
  • 1
    $\begingroup$ @ttnphns Perfect. So then, how do we know, if we should throw away magnitude of variation information, or keep it? When do you keep it and when to throw it away? Why not always keep it? This is the root cause of my confusion. $\endgroup$
    – Creatron
    Commented Jul 17, 2013 at 17:22
  • 1
    $\begingroup$ This is an eternal and not only yours confusion - what to use (e.g. in multivariate analysis): covariance matrix or correlation matrix, when both ways are warranted. I recommend you to look through this discussion, at least for lulz. $\endgroup$
    – ttnphns
    Commented Jul 17, 2013 at 17:30
  • 1
    $\begingroup$ @ttnphns That was a very interesting discussion... It seems though that correlation always has 'less' information than just the covariance. I guess that just means one should stick to covariance if 'magnitude matters'. When, in your experience, do you use one or the other? This is an example of when experience from others might help to answer this. $\endgroup$
    – Creatron
    Commented Jul 17, 2013 at 18:25


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.