I am using PCA as a dimension reduction method before feeding the data into a classifier.

I am typically selecting only the first two or three PCs. Typical data includes factors such as length, width, height, weight, energy output, and other key characteristics. Ultimately I am using only the first several PCA-scores instead of the original factors.

My question is: If I were to include compound factors such as area = width*length, energy per unit volume or surface area = (length*width + length*height + width*length) in addition to the simple factors that go into the PCA algorithm would this be expected to add, subtract or be neutral in terms of the information expected to be contained in the first several principal components? What about including multiple variations of the same factor such as length^2, length^3?


1 Answer 1


This raises concerns on various levels.

In general, it is a very good idea to use variables that are closer to the scientific (substantive, practical, whatever) part of your problem. If area is meaningful, then it should be an input variable, and so forth. In fact, in some fields, something like a dimensional analysis can be used to identify a minimal set of variables that characterise a problem. However, if that's possible, it is unlikely that you would gain much from PCA that you couldn't find from a theory-guided analysis. PCA tends to be most helpful, or at least most invoked, in data-rich, theory-poor fields.

However, there are, as said, concerns about what you are proposing.

Adding e.g. the square or cube of length might by accident pick up important structure in the data, but it also complicates things. The square and cube of length will be correlated with length, but not perfectly, the size of the correlation depending on the range of variation of length. Correlate length and length$^2$ in your data if you want to quantify this. You may just end up boosting components that correlate strongly with length. There will be side-effects too, which won't be easy to predict or interpret. But what you find may be on a par with finding that adding more chocolate to a cake makes it taste more of chocolate.

The same kind of comment applies to products, ratios, etc. of existing variables.

In essence, PCA works best when there is a structure of correlations linking at least some of the variables. That implies that nonlinearities are more likely to mess up the components than the other way round.

So, there is nothing wrong with picking a version of your variable that is related to a problem, e.g. using length$^2$ not length, but I would be cautious about adding both.

Looking at the correlations between variables and indeed a scatter plot matrix is always to be recommended. For example, it is often a good idea to screen out variables that are only weakly correlated with anything else. Sometimes the correlation and scatter plot matrices show you relatively simple structure that makes the PCA appear unnecessary, or identify strong nonlinearities that will only complicate interpretation unless they are tackled with a transformation.

  • 1
    $\begingroup$ So in a data-rich theory poor domain where I have good data for the simple factors such as length, width, height but where I think that composite factors may be more meaningful I should be cautious of including both and should probably resort to judgement when deciding between simple or composites thereof. Is there any theory/method that could help to decide which set of factors to feed into PCA apart from conducting several analyses and comparing the overall results of each classification against some utility function? $\endgroup$ May 30, 2013 at 10:08
  • 1
    $\begingroup$ The first sentence is an excellent summary of my advice. The second, a question, brings in too many different aspects to be easy to answer, but within the space available: Plenty of accounts may imply otherwise, but in my experience there is usually a major element of judgment about which variables (factors, in your terminology) to feed in. I would distinguish sharply between PCAs and the classifications that might be based on them. PCAs often imply that classifications are of limited use because the pattern of variation is a continuum. $\endgroup$
    – Nick Cox
    May 30, 2013 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.