# Principal component analysis result

I'm trying to understand the result of PCA, thought you can help me to understand better.

us.pca1 <- prcomp(USArrests)
us.pca1$sdev [1] 83.732400 14.212402 6.489426 2.482790  Here I see the standard deviation for the variables Murder, Assault, UrbanPop and Rape are 83, 14, 6 and 2 respectively. But when I use the scale=T option I'm getting different SD. us.pca2 <- prcomp(USArrests,scale=T) us.pca2$sdev
[1] 1.5748783 0.9948694 0.5971291 0.4164494


From the Help, I know that scale=T is used, so that variables should be scaled to have unit variance before the analysis takes place. But does this actually means???
By the way, if I want to calculate SD in usual way, I'm getting different result.

sd(USArrests$Murder) [1] 4.35551  Can someone help me what are these three different SD indicates! Another question regarding the actual result in$roration matrix.

us.pca2$rotation PC1 PC2 PC3 PC4 Murder -0.5358995 0.4181809 -0.3412327 0.64922780 Assault -0.5831836 0.1879856 -0.2681484 -0.74340748 UrbanPop -0.2781909 -0.8728062 -0.3780158 0.13387773 Rape -0.5434321 -0.1673186 0.8177779 0.08902432  Are these Eigenvalues or some percentage? What should I conclude from this result? Any help or link for further reading will be appreciated. ## 3 Answers 1. What is scaling Scaling refers to techniques such as standardization and normalization which change the scale of your data. In this case, it specifically refers to standardization. More specifically, it sounds like Z-score scaling. This means the variables recalculated as (V - mean of V)/s, where "s" is the standard deviation. As a result, all variables in the data set have equal means (0) and standard deviations (1) but different ranges. I would type out the equation for you, but I'm bad with LaTex/MathJax, so please see the links below. http://www.benetzkorn.com/2011/11/data-normalization-and-standardization/ http://www.ats.ucla.edu/Stat/stata/faq/standardize.htm What's the difference between Normalization and Standardization? 2. What are these 3 different Standard Deviations They were: A. The raw standard deviation of the (first 4) principal components. Note that singular values of the data matrix are equal to the square roots of the eigenvalues of the covariance matrix, up to a scaling factor sqrt(N-1) where N is the number of data points. B. The scaled standard deviation of the (first 4) principal components (or, more precisely, the standard deviations of the (first 4) principal components created on scaled data). C. The raw standard deviation of the variable Murder Why scale? Well, there are lots of reasons, in addition to making the data easier to quickly analyze for some people. Another reason is that in methods involving gradient descent or other iterative solvers it leads to quicker convergence. us.pca1$sdev are the square roots of the eigenvalues of the covariance matrix of the unscaled data matrix:

> us.pca1$sdev [1] 83.732400 14.212402 6.489426 2.482790 > sqrt(eigen(var(USArrests))$values)
[1] 83.732400 14.212402  6.489426  2.482790


whereas us.pca2$sdev are the square roots of the eigenvalues of the covariance matrix of the scaled data matrix, which are different because the scaled data has a different variance: > us.pca2$sdev
[1] 1.5748783 0.9948694 0.5971291 0.4164494
> sqrt(eigen(var(scale(USArrests, center=TRUE, scale=TRUE)))$values) [1] 1.5748783 0.9948694 0.5971291 0.4164494  us.pca2$rotation is the new basis. The original data in coordinates with respect to this new basis is as.matrix(USArrests) %*% us.pca1$rotation. The standard deviations us.pca1$sdev are actually the standard deviations of the transformed data:

> sqrt(apply(as.matrix(USArrests) %*% us.pca1$rotation, 2, var)) PC1 PC2 PC3 PC4 83.732400 14.212402 6.489426 2.482790  To answer your questions directly: Q1: But when I use the scale=T option I'm getting different SD. From the Help, I know that scale=T is used, so that variables should be scaled to have unit variance before the analysis takes place. But does this actually means??? The scale=T argument tells prcomp to perform z-scaling (that is, dividing each value by the standard deviation of all the values). You would get the same result if you performed scaling before calling prcomp. scaledUSArrests <- scale(USArrests) prcomp(scaledUSArrests) # Standard deviations: # [1] 1.5748783 0.9948694 0.5971291 0.4164494 # # Rotation: # PC1 PC2 PC3 PC4 # Murder -0.5358995 0.4181809 -0.3412327 0.64922780 # Assault -0.5831836 0.1879856 -0.2681484 -0.74340748 # UrbanPop -0.2781909 -0.8728062 -0.3780158 0.13387773 # Rape -0.5434321 -0.1673186 0.8177779 0.08902432 prcomp(USArrests, scale=T) # Standard deviations: # [1] 1.5748783 0.9948694 0.5971291 0.4164494 # # Rotation: # PC1 PC2 PC3 PC4 # Murder -0.5358995 0.4181809 -0.3412327 0.64922780 # Assault -0.5831836 0.1879856 -0.2681484 -0.74340748 # UrbanPop -0.2781909 -0.8728062 -0.3780158 0.13387773 # Rape -0.5434321 -0.1673186 0.8177779 0.08902432  Q2: By the way, if I want to calculate SD in usual way, I'm getting different result. The standard deviation that you see there is not the standard deviation from running sd(USArrests$Assault), etc.

Q3. Are these Eigenvalues or some percentage? What should I conclude from this result? Any help or link for further reading will be appreciated.

PCA works by projecting your original data set onto new axes called principal components. Because your original data has 4 features (Murder, Assault, UrbanPop, and Rape), there will be 4 new principal components. Think of them as a new coordinate system defined by 4 new axes. In the output, you can seem them labelled as PC1, PC2, PC3, and PC4.

In the figure below, you can see how the blue dots are projected onto a new axis by connecting them with the red lines. Once the blue dots are on the new axis, you can compute the variance over them, as seen by the green segment. That variance is, of course, the square of the standard deviation shown in the output above.

For example, your first axis is defined by the PC1 vector above, which has the coordinates (-0.535, -0.583, -0.278, -0.543). When your data is projected onto this axis, you can compute a standard deviation of 1.5748783.

You can get more information form using the summary() function, as shown below.

summary(prcomp(USArrests, scale=T))
# Importance of components:
#                           PC1    PC2     PC3     PC4
# Standard deviation     1.5749 0.9949 0.59713 0.41645
# Proportion of Variance 0.6201 0.2474 0.08914 0.04336
# Cumulative Proportion  0.6201 0.8675 0.95664 1.00000


What this shows is that the first three axes explain 0.95664 (the bottom row) of the variance. That is, if your data is projected onto the first three axes defined by the PC1, PC2, and PC3 vectors, then the variance would be 0.95664 of all the variance. The missing variance is explained by data projected onto the axis of PC4.

I know this probably seems really difficult to understand. Here are some great tutorials:

http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf