7
$\begingroup$

In all my self-study, I have come across many different ways in which people seem to normalize their data, prior to the computation of the covariance matrix. I am confused as to what ways are 'correct', or if is just application dependent.

For my particular application, I have a data matrix $X$, which is a $9$x$10000$ matrix. (Let $s=9$, and $N = 10000$). This matrix came from 9 sensors on the ground, which measured vibrations across time. I am tasked with looking at "how well" the different sensor outputs correlate with each other, in other words, studying the covariance matrix.

Option 1: The standard way I am aware of, is to subtract the mean of each row of $X$ from itself, and then compute the sample covariance matrix: Thus, let $\bar{X}$ be nothing but $X$, with the mean of each row subtracted out. Then, the covariance matrix is given by $C = (N-1)^{-1}\bar{X} \bar{X}^T$. In this way, the variance of each sensor output is unchanged, but all sensor outputs have zero mean.

Option 2: Another similar way I have seen, it not to simply stop at removing the mean, but also forcing the standard deviation of each sensor output to be 1, before computing $C$. So in this case, I would take my data matrix $X$, would remove the mean from each row, but then I would also divide each row by its own standard deviation, so that now, all rows/sensor outputs are zero mean, and unit variance. Then, we compute the Covariance matrix as before. Here of course, the covariance matrix will look different... so is this 'correct'?

Option 3: While this option is technically not about normalizing the data prior to computation of the covariance matrix, this method, (Pearson correlation coefficient) simply weights each element of the covariance matrix computed using (1), by the inverse of the product of standard deviations of the vectors it represents. In this case, my Pearson correlation matrix is going to have values between -1 and 1. Like (2), this also has the effect of "not caring" about the energy from each sensor.


Now, I am confused as to which method I need to use for this application... Do I simply demean my sensor data and keep their variances before computing $C$, as the textbook method? Do I demean and standardize all their variances to unity before computing $C$, as some machine learning texts do? Or do I look at the Pearson matrix as option (3), which also does a kind of normalizing by sigmas?

Very confused. :-/

$\endgroup$
  • $\begingroup$ @ttnphns Ah yes, that is true. Also taken from your own post, you said " "covariance shows only the strength of a linear relationship" This is not true. Cov is sensitive to both strength of linearity and to magnitude of variation." Does this mean then, that by standardizing as per options (2) and (3), we are throwing away information relating to the magnitude of variation, but keeping information relating to the linearity measure? $\endgroup$ – Creatron Jul 17 '13 at 16:57
  • $\begingroup$ Options 2 and 3 both lead to the same result: Pearson correlation matrix. Because r is the covariance between z-standardized variables [rows, in your notation] and at the same time the cosine between centered variables (note that cov/(sigma1*sigma2) boils down to cosine). P.S. simply weights each element of the covariance matrix computed using (1), by the INVERSED product of standard deviations $\endgroup$ – ttnphns Jul 17 '13 at 16:59
  • $\begingroup$ Where did I say "covariance shows only the strength of a linear relationship"? $\endgroup$ – ttnphns Jul 17 '13 at 17:04
  • $\begingroup$ @ttnphns I didnt say you said it. You were telling someone else that it was wrong. So you corrected them by saying it measures linearity, AND magnitude of variation. $\endgroup$ – Creatron Jul 17 '13 at 17:05
  • $\begingroup$ @ttnphns I am talking about what you said here $\endgroup$ – Creatron Jul 17 '13 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.