0
$\begingroup$

This may be really a simple question.

I know that if there are correlated variables, they may not be good to use for modeling. We have to use the PCA such that each PC is orthogonal to one meaning every PC is independent of one another.

If you have 4 variables, then you will have 4 PC at maximum. You have to determine by looking at the scree plot to see how many PC you should be using to maximize the variance while minimizing the number of PC you are going to use. My question is I know that the variance for PC1 is far greater than the variance for PC3, for example. But, why is the variance for PC1 greater than PC2? Is there any good explanation?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You wrote:

I know that if there are correlated variables, they may not be good to use for modeling.

If by "modelling" you mean regression, then collinearity can be a problem if it is extreme, but correlation is not the same as collinearity and a moderate degree of collinearity is not a problem (and occurs in a LOT of regressions).

We have to use the PCA such that each PC is orthogonal to one meaning every PC is independent of one another.

No. If you do have problematic collinearity the PCA is one solution, but it's not the only one. For example, you can use ridge regression (which is quite different and keeps the original variables) or you can use partial least squares (which is similar to PCA but includes relations with the dependent variable).

You have to determine by looking at the scree plot to see how many PC you should be using

Again, no. The scree plot is ONE way of choosing a number of PCs, but it is quite subjective. I would rather first figure out what each PC means and then decide which ones to include. Including all four is not unreasonable in many cases -- that 4th one may be important to your research question.

But, why is the variance for PC1 greater than PC2? Is there any good explanation?

The first PC is always biggest, because it is the PC that extracts the maximum amount of variance. Each subsequent PC will always be smaller. But the pattern of how fast they decline varies a lot from one PCA to another.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you Peter for the detailed answer. CAn you please explain the difference between correlation and collinearity? Having a hard time differentiating the two $\endgroup$
    – user392987
    Commented Aug 26, 2023 at 18:54
  • $\begingroup$ Correlation is a measure of the linear relationship between two continuous variables. Collinearity is a measure of how close one variable is to being a linear combination of other variables. You can have low correlations among all variables and still have collinearity (if one variable is a sum of a bunch of uncorrelated variables). And you can have collinearity among categorical variables. $\endgroup$
    – Peter Flom
    Commented Aug 26, 2023 at 23:09
  • $\begingroup$ thank you , Peter. I have one question: you know all PC are not correlated with one another and using PCs can mitigate the issue of collinearity. all PC are not correlated with one another --> PC1 and PC2 are independent of each other (no correlation) using PCs can mitigate the issue of collinearity --> PC is not a linear combination of other x's. can you please confirm my interpretation is correct? thanks in advance $\endgroup$
    – user392987
    Commented Aug 27, 2023 at 2:48
  • $\begingroup$ Yes. PCs (at least if you do orthogonal rotation) are not correlated and can't be colinear. But they may not be the best solution to the problem of collinearity. I think it is rare for them to be the best solution. $\endgroup$
    – Peter Flom
    Commented Aug 27, 2023 at 12:14
  • $\begingroup$ thank you Peter. Long story short, when you have correlated variables, one option is to use the pca other than dropping redundant variables or use AIC/BIC. Then, you now have orthogonal variables as well as PCs are orthogonal. Can you please confirm this? thx $\endgroup$
    – user392987
    Commented Aug 27, 2023 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.