Does using a covariance matrix of scaled and centered variables compare with using a correlation matrix?

Question

I have some data with features which have different units of measurements. Here, by data, I'm trying to say that the row represents the observations and column the features. There are correlations between the features as well. Hence, principal component analysis would be the best to explain the variance in my data and also identify the variance contribution of each feature in the total variance explained by a principal component.

I read somewhere that principal components analysis (PCA) can be performed on the data by computing both covariance and correlation matrices. If I scale and center my data (Z-scores) and compute covariance and compare it with a correlation matrix, will the results of PCA be different? As both correlation matrix and scale/centered covariances represents standardization of the data, I assume the results (variance contributed by features in a principal component) should be same? Am I wrong in assuming this? I'm a novice and am trying to understand principal component analysis.

Answer 1

you will find a nice summary given by user @ttnphns here: https://stats.stackexchange.com/q/22520.

In particular:

• If you center columns (variables) of $\mathbf{A}$, then $\mathbf{A′A}$ is the scatter (or co-scatter, if to be rigorous) matrix and $\mathbf{A′A}/(n−1)$ is the covariance matrix.
• If you z-standardize columns of a matrix $\mathbf{A}$ (subtract the column mean and divide by the standard deviation), then $\mathbf{AA′}/(n−1)$ is the Pearson correlation matrix: correlation is covariance for standardized variables.

In general you should always center your data when performing PCA. As explained here, not centering your data can give misleading results.

Answer 2

** I have updated this answer as upon reflection i was focused on the right eigenvectors and there is a difference in scaling, so my original answer was not 100% accurate **

Does using a covariance matrix of scaled and centered variables compare with using a correlation matrix?

Yes

If I scale and center my data (Z-scores) and compute covariance and compare it with a correlation matrix, will the results of PCA be different?

Depends on what you consider to be meaningful. There will be a scaling difference for the eigenvalues, but the right eigenvectors will be unchanged while the left eigenvectors are just a rescaled version. If you were to use SVD to calculate your PCA but stopped before combining the left eigenvector and the singular values you would find that the U matrix is identical whether using z score transformed or correlation matrix, it is the singular values that will be uniformly transformed between the two.

If you scale (z-score) your data then your covariance matrix is a correlation matrix. A correlation matrix is a covariance matrix scaled against the diagonal (variance). Scaling to unit variance is scaling the original data to the standard deviation. Thus whether you scale to before or after the covariance matrix, the end result is still the same pattern of variation.

Some Matlab code I wrote to test this:

%% generate a matrix of random variables with some covariance
A = rand(100,1); 
A(:,2) = A(:,1) - rand(100,1); 
A(:,3) = 0.1*A(:,1) + 0.4*A(:,2) + rand(100,1);

%%calculate z-score matrix
A_mc = bsxfun(@minus,A,mean(A,1));
A_z = bsxfun(@rdivide,A_mc,std(A_mc,[],1));

%%validate matrix properties
if any(abs(mean(A_mc))>eps) %check if mean is within machine precision of 0
    error('Oops, matrix did not mean centre correctly')
elseif any(std(A_z)<1-eps | std(A_z)>1+eps)%check if sd is within machine precision of 1
    error('Oops, matrix did not scale to unit variance correctly')
end

%% PCA is based on covariance matrix, therefore it is sufficicent to compare these
A_zIP = A_z'*A_z; %covariance matrix
A_corr = corr(A); %correlation matrix

zIP_Corr = A_zIP./A_corr
zIP_CorrSD = std(zIP_Corr(:))

The output:

zIP_Corr =

   99.0000   99.0000   99.0000
   99.0000   99.0000   99.0000
   99.0000   99.0000   99.0000

zIP_CorrSD =

1.7405e-14

You can see the scaling is pretty close to uniform, with the difference likely due to the cumulative effect of the rounding errors between the two processes. So there is no practically significant change in the pattern of variation, therefore the right eigenvectors will be the same.