0
$\begingroup$

I have a dataset with 60ish variables. I performed a PCA (4 components) to reduce the dimensionality. The total variance explained by the 4 principal components is 0.79

When I compute the correlations between my original variables and the 4 PC I find that two variables are very highly correlated with PC1 and PC2 (around 0.95) and only few variables are moderately correlated with PC3 & PC4 (around 0.6)

My question is what about the other variables ? From a statistics perspective, does it mean that they don't really matter as they don't explain much ?

Thanks in advance.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Absolutely not, it does not mean that. What is your goal, why are you doing PCA, for what purpose? What will you do after PCA? $\endgroup$
    – user2974951
    Commented Feb 3, 2022 at 11:09
  • $\begingroup$ Well, my dataset consists on worksites with a target variable that says if it's an incident or no. My ultime goal is train a classifier. Now I am in the EDA step where I want to see the distribution of data visually (in 2D) and see if the incidents and not_incidents are visually separated $\endgroup$
    – Othman Bouzahzah
    Commented Feb 3, 2022 at 11:18
  • 3
    $\begingroup$ Keep in mind that PCA does not look at the $Y$ variable (nor is it meant to), so it is completely oblivious to what features distinguish $Y$ values. $\endgroup$
    – Dave
    Commented Feb 3, 2022 at 12:04
  • $\begingroup$ Not quite what you asked, but related scikit-learn.org/stable/auto_examples/cross_decomposition/… $\endgroup$
    – user2974951
    Commented Feb 3, 2022 at 13:57
  • $\begingroup$ @Sycorax forget about classification. All i'm trying to see is explore the dataset. It's simpler visually if reduced to 2D. What's surprising is that the principal components are highly correlated with only a few variables. What does that say about these particular variables ? and what does it say about the others ? $\endgroup$
    – Othman Bouzahzah
    Commented Feb 3, 2022 at 16:00

1 Answer 1

Reset to default
0
$\begingroup$

From your comment about creating a classification model, it seems like you need a supervised learning method like generalized partial least squares (this will be most conceptually similar to PCA, but will provide a model that can predict the response, in this case incident TRUE/FALSE). Another option is generalized lasso or elastic net.

For PLS, lasso, or elastic net, you can look at standardized coefficients to judge which predictors have a positive or negative contribution to incidents, and which effects are stronger or weaker. You won't get a measure of standard error unfortunately, so you will have to make some judgements about how to interpret and present the results.

PCA is useful for identifying relationships and redundancy between your predictor variables, but it is unsupervised, meaning it doesn't consider the response at all.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Arthur. The goal of PCA is not to use it for classification. It's more like an exploration step to visualize the dataset in 2D and get insights about variables correlation $\endgroup$
    – Othman Bouzahzah
    Commented Feb 3, 2022 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.