What are principal component scores?

Question

What are principal component scores (PC scores, PCA scores)?

Answer 1

First, let's define a score.

John, Mike and Kate get the following percentages for exams in Maths, Science, English and Music as follows:

      Maths    Science    English    Music    
John  80        85          60       55  
Mike  90        85          70       45
Kate  95        80          40       50

In this case there are 12 scores in total. Each score represents the exam results for each person in a particular subject. So a score in this case is simply a representation of where a row and column intersect.

Now let's informally define a Principal Component.

In the table above, can you easily plot the data in a 2D graph? No, because there are four subjects (which means four variables: Maths, Science, English, and Music), i.e.:

• You could plot two subjects in the exact same way you would with $x$ and $y$ co-ordinates in a 2D graph.
• You could even plot three subjects in the same way you would plot $x$, $y$ and $z$ in a 3D graph (though this is generally bad practice, because some distortion is inevitable in the 2D representation of 3D data).

But how would you plot 4 subjects?

At the moment we have four variables which each represent just one subject. So a method around this might be to somehow combine the subjects into maybe just two new variables which we can then plot. This is known as Multidimensional scaling.

Principal Component analysis is a form of multidimensional scaling. It is a linear transformation of the variables into a lower dimensional space which retain maximal amount of information about the variables. For example, this would mean we could look at the types of subjects each student is maybe more suited to.

A principal component is therefore a combination of the original variables after a linear transformation. In R, this is:

DF <- data.frame(Maths=c(80, 90, 95), Science=c(85, 85, 80),  
                 English=c(60, 70, 40), Music=c(55, 45, 50))
prcomp(DF, scale = FALSE)

Which will give you something like this (first two Principal Components only for sake of simplicity):

                PC1         PC2
Maths    0.27795606  0.76772853 
Science -0.17428077 -0.08162874 
English -0.94200929  0.19632732 
Music    0.07060547 -0.60447104 

The first column here shows coefficients of linear combination that defines principal component #1, and the second column shows coefficients for principal component #2.

So what is a Principal Component Score?

It's a score from the table at the end of this post (see below).

The above output from R means we can now plot each person's score across all subjects in a 2D graph as follows. First, we need to center the original variables by subtracting column means:

      Maths    Science    English    Music    
John  -8.33       1.66       3.33       5  
Mike   1.66       1.66      13.33      -5
Kate   6.66       -3.33    -16.66       0

And then to form linear combinations to get PC1 and PC2 scores:

      x                                                    y
John -0.28*8.33 + -0.17*1.66 + -0.94*3.33  + 0.07*5   -0.77*8.33 + -0.08*1.66 + 0.19*3.33   + -0.60*5 
Mike 0.28*1.66  + -0.17*1.66 + -0.94*13.33 + -0.07*5   0.77*1.66 + -0.08*1.66 + 0.19*13.33  + -0.60*5
Kate 0.28*6.66  + 0.17*3.33  + 0.94*16.66  + 0.07*0    0.77*6.66 +  0.08*3.33 + -0.19*16.66 + -0.60*0

Which simplifies to:

        x       y
John   -5.39   -8.90
Mike  -12.74    6.78
Kate   18.13    2.12

There are six principal component scores in the table above. You can now plot the scores in a 2D graph to get a sense of the type of subjects each student is perhaps more suited to.

The same output can be obtained in R by typing prcomp(DF, scale = FALSE)$x.

EDIT 1: Hmm, I probably could have thought up a better example, and there is more to it than what I've put here, but I hope you get the idea.

EDIT 2: full credit to @drpaulbrewer for his comment in improving this answer.

Answer 2

Principal component analysis (PCA) is one popular approach analyzing variance when you are dealing with multivariate data. You have random variables X1, X2,...Xn which are all correlated (positively or negatively) to varying degrees, and you want to get a better understanding of what's going on. PCA can help.

What PCA gives you is a change of variable into Y1, Y2,..., Yn (i.e. the same number of variables) which are linear combinations of the Xs. For example, you might have Y1 = 2.1 X1 - 1.76 X2 + 0.2 X3...

The Ys the nice property that each of these have zero correlation with each other. Better still, you get them in decreasing order of variance. So, Y1 "explains" a big chunk of the variance of the original variables, Y2 a bit less and so on. Usually after the first few Ys, the variables become somewhat meaningless. The PCA score for any of the Xi is just it's coefficient in each of the Ys. In my earlier example, the score for X2 in the first principal component (Y1) is 1.76.

The way PCA does this magic is by computing eigenvectors of the covariance matrix.

To give a concrete example, imagine X1,...X10 are changes in 1 year, 2 year, ..., 10 year Treasury bond yields over some time period. When you compute PCA you generally find that the first component has scores for each bond of the same sign and about the same sign. This tells you that most of the variance in bond yields comes from everything moving the same way: "parallel shifts" up or down. The second component typically shows "steepening" and "flattening" of the curve and has opposite signs for X1 and X10.

Answer 3

I like to think of principal component scores as "basically meaningless" until you actually give them some meaning. Interpretting PC scores in terms of "reality" is a tricky business - and there can really be no unique way to do it. It depends on what you know about the particular variables that are going into the PCA, and how they relate to each other in terms of interpretations.

As far as the mathematics goes, I like to interpret PC scores as the co-ordinates of each point, with respect to the principal component axes. So in the raw variables you have $\bf{}x_i$ $=(x_{1i},x_{2i},\dots,x_{pi})$ which is a "point" in p-dimensional space. In these co-ordinates, this means along the $x_{1}$ axis the point is a distance $x_{1i}$ away from the origin. Now a PCA is basically a different way to describe this "point" - with respect to its principal component axis, rather than the "raw variable" axis. So we have $\bf{}z_i$ $=(z_{1i},z_{2i},\dots,z_{pi})=\bf{}A(x_i-\overline{x})$, where $\bf{}A$ is the $p\times p$ matrix of principal component weights (i.e. eigenvectors in each row), and $\bf{}\overline{x}$ is the "centroid" of the data (or mean vector of the data points).

So you can think of the eigenvectors as describing where the "straight lines" which describe the PCs are. Then the principal component scores describe where each data point lies on each straight line, relative to the "centriod" of the data. You can also think of the PC scores in combination with the weights/eigenvectors as a series of rank 1 predictions for each of the original data points, which have the form:

$$\hat{x}_{ji}^{(k)}=\overline{x}_j+z_{ki}A_{kj}$$

Where $\hat{x}_{ji}^{(k)}$ is the prediction for the $i$th observation, for the $j$th variable using the $k$th PC.

Answer 4

Say you have a cloud of N points in, say, 3D (which can be listed in a 100x3 array). Then, the principal components analysis (PCA) fits an arbitrarily oriented ellipsoid into the data. The principal component score is the length of the diameters of the ellipsoid.

In the direction in which the diameter is large, the data varies a lot, while in the direction in which the diameter is small, the data varies litte. If you wanted to project N-d data into a 2-d scatter plot, you plot them along the two largest principal components, because with that approach you display most of the variance in the data.

Answer 5

Principal component scores are a group of scores that are obtained following a Principle Components Analysis (PCA). In PCA the relationships between a group of scores is analyzed such that an equal number of new "imaginary" variables (aka principle components) are created. The first of these new imaginary variables is maximally correlated with all of the original group of variables. The next is somewhat less correlated, and so forth until the point that if you used all of the principal components scores to predict any given variable from the initial group you would be able to explain all of its variance. The way in which PCA proceeds is complex and has certain restrictions. Among these is the restriction that the correlation between any two principal components (i.e. imaginary variables) is zero; thus it doesn't make sense to try to predict one principal component with another.

Answer 6

Let $i=1,\dots,N$ index the rows and $j=1,\dots,M$ index the columns. Suppose you linearize the combination of variables (columns):

$$Z_{i,1} = c_{i,1}\cdot Y_{i,1} + c_{i,2}\cdot Y_{i,2} + ... + c_{i,M}\cdot Y_{i,M}$$

The above formula basically says to multiply row elements with a certain value $c$ (loadings) and sum them by columns. Resulting values ($Y$ values times the loading) are scores.

A principal component (PC) is a linear combination $Z_1 = (Z_{1,1}, ..., Z_{N,1}$) (values by columns which are called scores). In essence, the PC should present the most important features of variables (columns). Ergo, you can extract as many PC as there are variables (or less).

An output from R on PCA (a fake example) looks like this. PC1, PC2... are principal components 1, 2... The example below is showing only the first 8 principal components (out of 17). You can also extract other elements from PCA, like loadings and scores.

Importance of components:
                          PC1    PC2    PC3    PC4    PC5    PC6    PC7    PC8
Standard deviation     1.0889 1.0642 1.0550 1.0475 1.0387 1.0277 1.0169 1.0105
Proportion of Variance 0.0697 0.0666 0.0655 0.0645 0.0635 0.0621 0.0608 0.0601
Cumulative Proportion  0.0697 0.1364 0.2018 0.2664 0.3298 0.3920 0.4528 0.5129

Answer 7

The principal components of a data matrix are the eigenvector-eigenvalue pairs of its variance-covariance matrix. In essence, they are the decorrelated pieces of the variance. Each one is a linear combination of the variables for an observation -- suppose you measure w, x, y,z on each of a bunch of subjects. Your first PC might work out to be something like

0.5w + 4x + 5y - 1.5z

The loadings (eigenvectors) here are (0.5, 4, 5, -1.5). The score (eigenvalue) for each observation is the resulting value when you substitute in the observed (w, x, y, z) and compute the total.

This comes in handy when you project things onto their principal components (for, say, outlier detection) because you just plot the scores on each like you would any other data. This can reveal a lot about your data if much of the variance is correlated (== in the first few PCs).