6
$\begingroup$

I am confused about what a component is in PCA and PLS.

Are the components just the original variables but not necessarily in the same order?

For example, in PCA, if I had 8 variables in my data, would PC1 correspond to one of the 8 variables? And in PLSR (PLS regression), if I were to use 4 components, does this mean that I will be using 4 out of 8 variables to build a model?

$\endgroup$
  • $\begingroup$ Yes I think I know PLSR and PCA are two different thing. However as far as I understand both method is to reduce dimension of data(usually when you have small sample size but relatively large factors?). And I am confused in both situation. PCA you pull out principal components that are less than number of factors. In PLSR you use number of components that are less than number of factors. So I guess my question is in PLSR if I were to decide to use 4 components in model out of 8 possible compoents, is this equivalent to say that I am using 4 factors out of 8. If I am wrong, please redirect me. $\endgroup$ – Kane Mar 2 '16 at 21:44
  • $\begingroup$ To make my question clear, lets say I want to perform pls. For example, my data is 8 factors and 25 samples. I want to reduce number of factors in my model. I used plsr in R and I get 8 models, model with 1 comp, 2comps and so on. Max comp I can use is 8. I thought it is because I only have 8 factors. This leads me to think that component and factor are euquivalent term. $\endgroup$ – Kane Mar 3 '16 at 15:16
  • $\begingroup$ Kane, when you say that your data "is 8 factors and 25 samples", you mean that there is 8 measured variables? If so, you shouldn't use the word "factor" for it; "factor" usually means an "underlying factor", estimated e.g. via PCA or factor analysis (FA). So if you have 8 variables, you might do PCA/FA and then conclude that you have e.g. 2 or 3 factors. Does this make sense? This was probably the source of my confusion! Your question makes sense if I replace the word "factor" with the word "variable" everywhere :) $\endgroup$ – amoeba says Reinstate Monica Mar 3 '16 at 15:29
  • $\begingroup$ Yea I think you are right. I have 8 measurements per sample. I simply said each measurement as factor. Then am I thinking right that components in the model is subset of total variables? $\endgroup$ – Kane Mar 3 '16 at 15:33
  • 2
    $\begingroup$ Please see this highly rated page to help clarify, and edit your question if issues are still not clear. Principal components, for example, are new predictors formed by linear combinations of the original predictor variables, with each combination orthogonal (generalization of perpendicular) to each of the others. They are not particular subsets of the original variables. Each principal component may include contributions from all of the original variables. $\endgroup$ – EdM Mar 3 '16 at 15:46
9
$\begingroup$

The possible confusion here, as @amoeba points out in a comment, is the difference between variable selection and dimensionality reduction.

Both PCA and PLS are intended to reduce the dimensionality of the problem. If you have measured 8 variables on each of your cases (and you have more than 8 cases) then the original dimension is 8. PCA and PLS help you choose a lower number of dimensions that will work well enough.

But these procedures do not work by selecting subsets from your original 8 variables. Rather, they construct linear combinations of the 8 variables to make new sets of 8 predictors, then decide how many of these new combinations need to be included in the final model. For either PCA or PLS, these new predictors are designed to be orthogonal (multi-dimensional equivalent of perpendicular) to each other. If there are correlations among predictors, all 8 of your original variables are thus likely to be included to some extent even if you end up with a final dimension of, say, 4. So you are not typically performing all-or-none selection among your original variables. You just get rid of some less-important combinations of them.

PCA simply examines the predictors themselves, finding first the combination that captures the most variance in the predictors, then the (orthogonal) combination that captures the next most, and so on. Several superb explanations of how this works are on this highly rated page.

PLS includes in this type of scheme the relations of predictors to the outcome variable. At each step it finds the combination of predictors, orthogonal to all prior combinations, that maximizes the product of the variance of the predictors times the square of the correlation to the outcome variable. (See ESLII, eq. 3.64, page 81). For the first step, this is a linear combination weighted by each variable's individual correlation to outcome (unlike standard multiple regression, where all variables are considered together). PLS also gives a set of orthogonal predictors, made of linear combinations of the original variables, although different from those provided by PCA.

In either procedure, a decision is made about how many of these new predictors to include, determining the final dimensionality. In either case, if you include all the new predictors, you just get back the original multiple regression.

Also, please note that the above assumes that the predictor variables were first standardized so that differences in scales of the variables do not matter.

$\endgroup$
  • 2
    $\begingroup$ +1. However, the explanation of PLS is somewhat torn. You have not explained how comes that residuals can be further predicted again by a new, orthogonal combination; in other words - what conditions make PLS possible and different from usual multiple regression. $\endgroup$ – ttnphns Mar 3 '16 at 19:25
  • $\begingroup$ @ttnphns I've edited to make the description of PLS a bit more precise, I hope without losing clarity. $\endgroup$ – EdM Mar 3 '16 at 20:27
  • $\begingroup$ +1. Here is one link for further reading about PLS regression (cc @ttnphns): stats.stackexchange.com/questions/179733. $\endgroup$ – amoeba says Reinstate Monica Mar 3 '16 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.