5
$\begingroup$

I'm having trouble understanding how to interpret/explain the end result of dimensionality reduction via PCA. Namely, I've attempted to code up a simple example in R but can't really say what happened. I started with three vectors (x,y,z) where x and y are highly correlated and z is veeery loosely related to x. After finding the top 2 principal components and multiplying back by the original data, I have what I believe to be the "final result"... but the plot of the new 2-D data appears entirely uncorrelated. What am I seeing? What are the axes? I expected to see something that resembled the data on the x-y plane.

I have plots of the mean-subtracted data in 3d and the reduced plot but not enough points to post them.

$\endgroup$

2 Answers 2

9
$\begingroup$

The previous answer correctly explains what you see, but I would like to add that one of the properties of PCA is that it makes all variables uncorrelated. So after doing PCA each component is guaranteed to be uncorrelated with any other component, and you should not be surprised.

If you are a bit familiar with mathematics behind PCA, you know that it is equivalent to diagonalizing the covariance matrix of the data. If the covariance matrix is diagonal, then covariance (and also correlation) between any two different components is zero.

$\endgroup$
8
$\begingroup$

Your initial data was rotated in the existing three dimensions such that the bulk of the variance was along the X axis, then rotated again such that the remaining variance was predominantly along the Y axis. Then the Z axis was flattened so only the new X and Y axes remained.

This article goes into a really good and accessible explanation of what is going on in PCA, I recommend you check it out: http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf

UPDATE: Regarding your comment below, here's a simple example illustrating why your PCA transformed data doesn't look like the data projected into the X-Y plane:

set.seed(123)
X = rnorm(100)
Y = X + rnorm(100,0,.5)
Z = rnorm(100)
df = data.frame(X,Y,Z)
df.xz = as.matrix(cbind(Y,Z))
plot(df)

enter image description here

cor(X,Y) # .8786993
cor(X,Z) # -0.129176
cor(Y,Z) # -0.1019494

# X and Y are tightly correlated. Z is uncorrelated to both

pc.sd1 = princomp(df)

# plot of first two components looks random.    
plot(pc.sd1$scores[,1:2])

enter image description here

It makes sense that the resulting plot looks random: it is random. The principal components capture the variance in the data. The idea is that if two variables are tightly correlated, we probably aren't adding much information by including both in our model: we really only need one, especially if one variable is actually a function of the other. Principal components is an easy way to ignore those relationships.

The plot in this example is capturing the relationship between X/Y and Z because there's a lot of variance in that (those) relationship(s). There's very little variance in the relationship between X and Y, so PCA allows us to combine those two dimensions into a single dimension so we can focus on the more complex relationship between this new dimension and Z.

Let's look at how each point was transformed in each pair of dimensions to further illustrate this.

# The end point of each line signifies the end point of the PCA transformation.
PCA_transform_plot=function(dims){
  plot(df[,dims], main=paste(names(df)[dims], collapse=""))
  sapply(1:nrow(df), function(i){
    lines(rbind(df[i,dims]
               ,pc.sd1$scores[i,1:2] 
                )
          ,col='red'     
          )      
  })
}

par(mfrow=c(1,3))
PCA_transform_plot(c(1,2))
PCA_transform_plot(c(1,3))
PCA_transform_plot(c(2,3))

enter image description here

As you can see, the points are displaced significantly relative to their initial positions in the XY plane, but much less so from their positions in the XZ and YZ planes. Moreover, the transformations in the XZ and YZ planes look very similar. In fact, the starting positions in the XZ and YZ planes look very similar. This isn't surprising: in this example, X and Y are so tightly correlated, they're practically interchangeable. PCA is a technique that let's us say (in this example), "Hey, these variables are so close, we don't really need both. Let's pretend our data is two dimensional instead of three dimensional, because it may as well be."

$\endgroup$
2
  • $\begingroup$ Thanks for the quick response. I've read though the link you suggested several times-- agreed, it's a fantastic resource. I suspect that I'm over thinking my question. My concern is that x and y are very tightly correlated with plotted against each other, but the "flattened" plot looks like a bunch of random points. Is this just a result of having the z data being so uncorrelated? $\endgroup$
    – Nancy
    Commented Aug 4, 2013 at 19:44
  • $\begingroup$ Updated my answer to demonstrate why your plot looks the way it does. Short version: because X and Y are tightly correlated, PCA combines them into a single dimension to allow you to explore the relationship between those variables and Z. The X-Z and Y-Z relationships are effectively the same thing because X and Y are so tightly correlated. Your PCA transformation is showing you the much less structured (XY)-Z relationship. $\endgroup$
    – David Marx
    Commented Aug 5, 2013 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.