0
$\begingroup$

I have these mean centered points:

X = [-0.7142;-1.7142;-1.7142;-0.7142;1.2857;1.2857;2.2857];
Y = [1.2857;0.2857;-0.7142;0.2857;0.2857;-0.7142;-0.7142];

I then calculated the PCA projection points as:

X_proj = [-2.2316; 0.0648; 1.044; -0.2725; -2.1535; 1.7647; 1.2005];
Y_proj = [2.8746; 3.4734; 1.5356; 1.5142; -2.4040; -1.4030; -2.3826];

with the modified axis given as:

y1 = 0.2053*x1 + 0;
y2 = -0.48719*x2 + 0;

I then did a plot in Matlab showing both the mean centered values, and then the projected values:

enter image description here

Where the red triangles represent the projected data and the blue squares represent the mean centered data.

I find this a little graphically confusing.

Is it true that the projection points should show perpendicular "projection lines" to the dashed axes? How should this be shown better graphically?

$\endgroup$
1
$\begingroup$

I think the problem is that you're plotting the projected data onto the projected basis (provided by PCA). That's confusing. The two ways to think about this are:

1) Original data (original basis) plotted against the new principle components - this will help you see that the principle components provide the best two orthogonal lines which minimize the squared perpendicular distance to your data. In fact, your first PC explains 85% of the variance in your data, so I'd probably just draw the first one only.

enter image description here

2) Projected data (new basis) and standard $\mathbb{R}^2$ - this will help you see how your projected data looks in it's "natural state". That is, if you treated the new basis as the "standard basis" and plotted the projected data onto it.

![enter image description here

See below for more detail on PCA and some R code to generate these plots.

  • The idea is to find the best k dimensional representation of the data, where usually k is less than the number of dimensions you are starting with. In this case, you start with 2 dimensions, so ideally we'd like to find a single dimension to reduce it to. Which is why plotting it on two dimensions might not provide much insight compared to plotting it on the first principle component
  • How do we know it's the best k dimensions? Because we know that the dimension that preserves the most variance in the data is the one orthogonal to your data - that's where the perpendicular part comes in. The next best dimension that preserves the most variance in your data is orthogonal to that first one. And so on. Each new dimension is orthogonal to the others.
  • So perhaps you should plot your projected data onto the first principle component only, as I showed above.

To help illustrate some of the concepts, here is some R code:

X = c(-0.7142,-1.7142,-1.7142,-0.7142,1.2857,1.2857,2.2857)
Y = c(1.2857,0.2857,-0.7142,0.2857,0.2857,-0.7142,-0.7142)
covariance = cov(data.frame(X,Y))

# grab the eigenvalues
lambda1 = eigen(covariance)$values[1]
lambda2 = eigen(covariance)$values[2]
# first principle component captures 85% of the total variance
# this suggests that you can express your data in one dimension
lambda1/(lambda1 + lambda2)  # 0.8461777

# grab the eigenvectors
v1 = eigen(covariance)$vectors[,1] #  x = -0.9795803  y = 0.2010532
v2 = eigen(covariance)$vectors[,2] #  x = -0.2010532  y = -0.9795803


# note that the new basis is orthogonal
v1 %*% v2  # 0

r = seq(-4,4,0.1)
# plot projection of original data into new basis
dim1 = t(v1)%*%t(as.matrix(data.frame(X,Y)))
dim2 = t(v2)%*%t(as.matrix(data.frame(X,Y)))
projection = cbind(t(dim1),t(dim2))
plot(projection,xlim=c(-3,3),ylim=c(-3,3),main='projected data')
abline(h=0,v=0,col='red')

# correlation between projected data is 0
cov(projection)

# plot original data and 2 principle components
plot(data.frame(X,Y),xlim=c(-3,3),ylim=c(-3,3),pch=8,main='original data, 2 PCs')
lines(y=v1[2]/v1[1]*r,x=r,type='l',col='red')
lines(y=v2[2]/v2[1]*r,x=r,col='red') 
$\endgroup$
  • $\begingroup$ thanks for your thorough solution. For the last two lines in your code, you negate the slope so as to obtain the perpendicular bias of the originally generated eigen vectors? $\endgroup$ – Joe Oct 23 '16 at 3:40
  • $\begingroup$ When I ran this problem in Matlab, I got the eigen vectors as [-0.201065872268197 -0.979577722801529; -0.979577722801529 0.201065872268197]. How did you get your eigenvectors to be the opposite?? Thanks for your time! $\endgroup$ – Joe Oct 23 '16 at 5:12
  • 1
    $\begingroup$ I removed the negation. I used it original to try to reproduce your PCs, but in fact it shouldn't be there. As to your Matlab question, did you reverse the order of eigenvectors? Can you see what the associated eigenvalues are with each eigenvector? $\endgroup$ – ilanman Oct 23 '16 at 12:37
  • $\begingroup$ Hello, these were my eigenvalues: [0.4834, 0; 0, 2.659]....as formatted in Matlab form. $\endgroup$ – Joe Oct 23 '16 at 16:11
  • 1
    $\begingroup$ Ok - the order is different thats all. See how your largest eigenvalue is the second element, and the associated eigenvector is (-0.979577722801529 0.201065872268197). This corresponds to what I got as well. $\endgroup$ – ilanman Oct 23 '16 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.