1
$\begingroup$

I'm trying to understand the result of PCA, thought you can help me to understand better.

us.pca1 <- prcomp(USArrests)
us.pca1$sdev
[1] 83.732400 14.212402  6.489426  2.482790

Here I see the standard deviation for the variables Murder, Assault, UrbanPop and Rape are 83, 14, 6 and 2 respectively. But when I use the scale=T option I'm getting different SD.

us.pca2 <- prcomp(USArrests,scale=T)
us.pca2$sdev
[1] 1.5748783 0.9948694 0.5971291 0.4164494

From the Help, I know that scale=T is used, so that variables should be scaled to have unit variance before the analysis takes place. But does this actually means???
By the way, if I want to calculate SD in usual way, I'm getting different result.

sd(USArrests$Murder)
[1] 4.35551

Can someone help me what are these three different SD indicates!

Another question regarding the actual result in $roration matrix.

us.pca2$rotation
           PC1        PC2        PC3         PC4
Murder   -0.5358995  0.4181809 -0.3412327  0.64922780
Assault  -0.5831836  0.1879856 -0.2681484 -0.74340748
UrbanPop -0.2781909 -0.8728062 -0.3780158  0.13387773
Rape     -0.5434321 -0.1673186  0.8177779  0.08902432

Are these Eigenvalues or some percentage? What should I conclude from this result? Any help or link for further reading will be appreciated.

$\endgroup$

migrated from stackoverflow.com Jun 17 '16 at 8:39

This question came from our site for professional and enthusiast programmers.

1
$\begingroup$

1. What is scaling

Scaling refers to techniques such as standardization and normalization which change the scale of your data. In this case, it specifically refers to standardization. More specifically, it sounds like Z-score scaling.

This means the variables recalculated as (V - mean of V)/s, where "s" is the standard deviation. As a result, all variables in the data set have equal means (0) and standard deviations (1) but different ranges.

I would type out the equation for you, but I'm bad with LaTex/MathJax, so please see the links below.

http://www.benetzkorn.com/2011/11/data-normalization-and-standardization/

http://www.ats.ucla.edu/Stat/stata/faq/standardize.htm

What's the difference between Normalization and Standardization?

2. What are these 3 different Standard Deviations

They were:

A. The raw standard deviation of the (first 4) principal components.

Note that singular values of the data matrix are equal to the square roots of the eigenvalues of the covariance matrix, up to a scaling factor sqrt(N-1) where N is the number of data points.

B. The scaled standard deviation of the (first 4) principal components (or, more precisely, the standard deviations of the (first 4) principal components created on scaled data).

C. The raw standard deviation of the variable Murder

Why scale? Well, there are lots of reasons, in addition to making the data easier to quickly analyze for some people. Another reason is that in methods involving gradient descent or other iterative solvers it leads to quicker convergence.

$\endgroup$
0
$\begingroup$

us.pca1$sdev are the square roots of the eigenvalues of the covariance matrix of the unscaled data matrix:

> us.pca1$sdev
[1] 83.732400 14.212402  6.489426  2.482790
> sqrt(eigen(var(USArrests))$values)
[1] 83.732400 14.212402  6.489426  2.482790

whereas us.pca2$sdev are the square roots of the eigenvalues of the covariance matrix of the scaled data matrix, which are different because the scaled data has a different variance:

> us.pca2$sdev
[1] 1.5748783 0.9948694 0.5971291 0.4164494
> sqrt(eigen(var(scale(USArrests, center=TRUE, scale=TRUE)))$values)
[1] 1.5748783 0.9948694 0.5971291 0.4164494

us.pca2$rotation is the new basis. The original data in coordinates with respect to this new basis is as.matrix(USArrests) %*% us.pca1$rotation. The standard deviations us.pca1$sdev are actually the standard deviations of the transformed data:

> sqrt(apply(as.matrix(USArrests) %*% us.pca1$rotation, 2, var))
      PC1       PC2       PC3       PC4 
83.732400 14.212402  6.489426  2.482790 
$\endgroup$
0
$\begingroup$

To answer your questions directly:

Q1: But when I use the scale=T option I'm getting different SD. From the Help, I know that scale=T is used, so that variables should be scaled to have unit variance before the analysis takes place. But does this actually means???

The scale=T argument tells prcomp to perform z-scaling (that is, dividing each value by the standard deviation of all the values). You would get the same result if you performed scaling before calling prcomp.

scaledUSArrests <- scale(USArrests)
prcomp(scaledUSArrests)
# Standard deviations:
# [1] 1.5748783 0.9948694 0.5971291 0.4164494
# 
# Rotation:
#                 PC1        PC2        PC3         PC4
# Murder   -0.5358995  0.4181809 -0.3412327  0.64922780
# Assault  -0.5831836  0.1879856 -0.2681484 -0.74340748
# UrbanPop -0.2781909 -0.8728062 -0.3780158  0.13387773
# Rape     -0.5434321 -0.1673186  0.8177779  0.08902432

prcomp(USArrests, scale=T)
# Standard deviations:
# [1] 1.5748783 0.9948694 0.5971291 0.4164494
#
# Rotation:
#                 PC1        PC2        PC3         PC4
# Murder   -0.5358995  0.4181809 -0.3412327  0.64922780
# Assault  -0.5831836  0.1879856 -0.2681484 -0.74340748
# UrbanPop -0.2781909 -0.8728062 -0.3780158  0.13387773
# Rape     -0.5434321 -0.1673186  0.8177779  0.08902432

Q2: By the way, if I want to calculate SD in usual way, I'm getting different result.

The standard deviation that you see there is not the standard deviation from running sd(USArrests$Assault), etc.

Q3. Are these Eigenvalues or some percentage? What should I conclude from this result? Any help or link for further reading will be appreciated.

PCA works by projecting your original data set onto new axes called principal components. Because your original data has 4 features (Murder, Assault, UrbanPop, and Rape), there will be 4 new principal components. Think of them as a new coordinate system defined by 4 new axes. In the output, you can seem them labelled as PC1, PC2, PC3, and PC4.

In the figure below, you can see how the blue dots are projected onto a new axis by connecting them with the red lines. Once the blue dots are on the new axis, you can compute the variance over them, as seen by the green segment. That variance is, of course, the square of the standard deviation shown in the output above.

enter image description here

For example, your first axis is defined by the PC1 vector above, which has the coordinates (-0.535, -0.583, -0.278, -0.543). When your data is projected onto this axis, you can compute a standard deviation of 1.5748783.

You can get more information form using the summary() function, as shown below.

summary(prcomp(USArrests, scale=T))
# Importance of components:
#                           PC1    PC2     PC3     PC4
# Standard deviation     1.5749 0.9949 0.59713 0.41645
# Proportion of Variance 0.6201 0.2474 0.08914 0.04336
# Cumulative Proportion  0.6201 0.8675 0.95664 1.00000

What this shows is that the first three axes explain 0.95664 (the bottom row) of the variance. That is, if your data is projected onto the first three axes defined by the PC1, PC2, and PC3 vectors, then the variance would be 0.95664 of all the variance. The missing variance is explained by data projected onto the axis of PC4.

I know this probably seems really difficult to understand. Here are some great tutorials:

http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf

http://www.sthda.com/english/wiki/principal-component-analysis-the-basics-you-should-read-r-software-and-data-mining

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy