I have a large set (100+) of variables on which I'm performing PCA. The PCA returns me a list of components, each of which is in turn a list of the weights to be placed on my variables in a linear combination.

What I want to do is use these numbers to evaluate the relative importance of each variable in describing the dataset. I don't want to just look at the absolute values of the weights in the first principal component, as that would ignore all the other components. However I also don't want to just sum the weights of each variable in all the principal components, since that doesn't take into account the fact that earlier components are more important than later ones.

One idea I've been told is to, for each variable, do a weighted average of the weights, where each weight is multiplied by the variance that the corresponding component represents. The logic is that that does make the earlier components (i.e. the ones with larger variance) have greater importance than the later ones (the ones with smaller variance). Would this be a sound idea? Is there another, better solution?

  • $\begingroup$ What do you mean by importance? Some average or other function of correlation of one variable with the others, is importance? $\endgroup$ Commented Jul 10, 2017 at 9:42
  • $\begingroup$ That is exactly my question. What is a good measure of the importance and effect of a variable in describing the data set, with regards to PCA? $\endgroup$
    – Bluefire
    Commented Jul 10, 2017 at 9:56
  • 1
    $\begingroup$ The measure of importance depends on what you mean by importance. Why do you want to evaluate importance? What is your underlying purpose? $\endgroup$ Commented Jul 10, 2017 at 9:57
  • $\begingroup$ I want to look at which variables are most critical and which are least critical. In my specific case my variable matrix is a TFIDF matrix constructed from a textual dataset. $\endgroup$
    – Bluefire
    Commented Jul 10, 2017 at 9:59
  • 1
    $\begingroup$ Sorry to be pedantic. What is the meaning of 'critical'? $\endgroup$ Commented Jul 10, 2017 at 10:00

1 Answer 1


A problem with your question is illustrated in the example below.

The points vary mostly in two directions (it is roughly a disk shape) and so the data may be reduced into two dimensions without loosing lot's of information (in terms of variance, possibly that tiny bit of variation may be important, first determine whether that viewpoint of amount of variance=information/importance applies).

Rather than selecting the largest PC's, PC1 and PC2 (which are transformations of x1, x2 and x3), one may consider instead to choose a set of the original parameters that also describes a large amount of the variation. Reducing the number of x's, for instance because it requires time to measure or space to store the information. This seems to be your goal.

Note in this example that x1 and x2 correlate strongly with the PC's and have high weights. Yet, it is better to select x1 + x3 or x2 + x3. This is because x1 and x2 correlate strongly with each other and after selecting one of x1 and x2 as important, the other one does not provide much more value. The contrast X1-X2 correlates only with the small variance of PC3).

What the PCA does is just showing you a structure perpendicular components generated in order of maximum variance. The goals of a PCA is to observe an underlying structure in a complex system of many variables. It does not give you an answer to reduce the dimensionality by selecting less variables (instead, it allows you to reduce the dimensionality by transformation, but this still requires all of the original variables).

What you could better do is write an algorithm selecting and switching variables until a maximum explained variance is achieved.

example of data with two strongly correlated parameters

R code to generate the image

#generating three random PC's
PC1 = rnorm(100,0,1)
PC2 = rnorm(100,0,0.5)
PC3 = rnorm(100,0,0.1)

#transformation back into underlying parameters
x1 = PC1 - PC3  
x2 = PC1 + PC3
x3 = 0.2*PC1 + PC2

for (theta in c(0:120)*3) {
  if (theta < 10) {n = paste0('000',theta)}
  if (theta < 100 && theta >= 10) {n = paste0('00',theta)}
  if (theta >= 100) {n = paste0('0', theta)}


system("convert ~/Desktop/gifs/image*.png -delay 1 -loop 0 ~/Desktop/gifs/3D.gif")
  • $\begingroup$ That's a really insightful take, thank you for putting in the time to do this? $\endgroup$
    – Bluefire
    Commented Jul 10, 2017 at 10:58
  • $\begingroup$ The given example and suggestion for an alternative method are based on a model (with reduced number of variables) that 'maximizes the explained variance in the other variables'. Possibly you wish to optimize a model that 'maximizes the success of a prediction of some kind' (for instance predicting a classification). $\endgroup$ Commented Jul 10, 2017 at 11:10
  • $\begingroup$ The latter is true, but would the former not achieve the same goal? $\endgroup$
    – Bluefire
    Commented Jul 10, 2017 at 11:28
  • $\begingroup$ You can also approach the problem in reverse, finding out which variables are less needed, using techniques such as battery reduction. $\endgroup$ Commented Jul 10, 2017 at 11:36
  • $\begingroup$ Using the example. Imagine that an important classification has only a small effect on the variation, in the contrast X1-X2. Yet some other effect (not seen as important, possibly noise) is causing a large variation on the variables X1+X2. If the effect, classification, has only a small effect on the measured variables than PCA may not reveal this. Or actually, PCA may reveal this by showing that a small PC component can be used as a significant factor for some model. In this way you could use PCA as a filter, although there may be better methods for this. ie LDA, or non-linear techniques. $\endgroup$ Commented Jul 10, 2017 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.