I was working on an exercise that asks me to get a k-dimensional approximation of a data vector using PCA (containing 4096 dimensions - the faces data set). I did this in R, but the last section asks for a "short mathematical expression for how you construct the approximation where you need to account appropriately for the mean and standard deviation terms". I am familiar how to derive the PC's of the covariance matrix using eigen decomposition but I don't get the "accounting for the mean and sd" part.

Can someone help?

  • $\begingroup$ I think they are asking you to standardize the data before carrying out the PCA. $\endgroup$ – Vishaal Sudarsan Mar 12 '18 at 1:26

In PCA, our goal is to find linear combinations of the data such that the variance is maximized subject to certain conditions. If our original data is not all on the same scale, and especially if the scale difference is large, then the result can be somewhat meaningless. We can rectify this by standardizing our data first and then running the procedure - this is equivalent to using our original correlation matrix as the covariance matrix in the PCA. This can be shown as follows: $$ Cov\left(\frac{X_1-\mu_1}{\sigma_1}, \frac{X_2-\mu_2}{\sigma_2}\right) = \frac{Cov(X_1,X_2)}{\sigma_1\sigma_2} = Corr(X_1, X_2) $$

As we can see, the correlation matrix is equivalent to the covariance matrix of the standardized variables. Note that, as a result, your conclusions should be about the standardized variables and not your original variables.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.