What is the meaning of using a compound variable in PCA

Question

I am using PCA as a dimension reduction method before feeding the data into a classifier.

I am typically selecting only the first two or three PCs. Typical data includes factors such as length, width, height, weight, energy output, and other key characteristics. Ultimately I am using only the first several PCA-scores instead of the original factors.

My question is: If I were to include compound factors such as area = width*length, energy per unit volume or surface area = (length*width + length*height + width*length) in addition to the simple factors that go into the PCA algorithm would this be expected to add, subtract or be neutral in terms of the information expected to be contained in the first several principal components? What about including multiple variations of the same factor such as length^2, length^3?

Answer 1

This raises concerns on various levels.

In general, it is a very good idea to use variables that are closer to the scientific (substantive, practical, whatever) part of your problem. If area is meaningful, then it should be an input variable, and so forth. In fact, in some fields, something like a dimensional analysis can be used to identify a minimal set of variables that characterise a problem. However, if that's possible, it is unlikely that you would gain much from PCA that you couldn't find from a theory-guided analysis. PCA tends to be most helpful, or at least most invoked, in data-rich, theory-poor fields.

However, there are, as said, concerns about what you are proposing.

Adding e.g. the square or cube of length might by accident pick up important structure in the data, but it also complicates things. The square and cube of length will be correlated with length, but not perfectly, the size of the correlation depending on the range of variation of length. Correlate length and length$^2$ in your data if you want to quantify this. You may just end up boosting components that correlate strongly with length. There will be side-effects too, which won't be easy to predict or interpret. But what you find may be on a par with finding that adding more chocolate to a cake makes it taste more of chocolate.

The same kind of comment applies to products, ratios, etc. of existing variables.

In essence, PCA works best when there is a structure of correlations linking at least some of the variables. That implies that nonlinearities are more likely to mess up the components than the other way round.

So, there is nothing wrong with picking a version of your variable that is related to a problem, e.g. using length$^2$ not length, but I would be cautious about adding both.

Looking at the correlations between variables and indeed a scatter plot matrix is always to be recommended. For example, it is often a good idea to screen out variables that are only weakly correlated with anything else. Sometimes the correlation and scatter plot matrices show you relatively simple structure that makes the PCA appear unnecessary, or identify strong nonlinearities that will only complicate interpretation unless they are tackled with a transformation.