# Using principal component analysis (PCA) for feature selection

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out noninformative input variables? Basically, I want to be able to order the original features in the data by variance or amount of information contained.

The basic idea when using PCA as a tool for feature selection is to select variables according to the magnitude (from largest to smallest in absolute values) of their coefficients (loadings). You may recall that PCA seeks to replace $p$ (more or less correlated) variables by $k<p$ uncorrelated linear combinations (projections) of the original variables. Let us ignore how to choose an optimal $k$ for the problem at hand. Those $k$ principal components are ranked by importance through their explained variance, and each variable contributes with varying degree to each component. Using the largest variance criteria would be akin to feature extraction, where principal component are used as new features, instead of the original variables. However, we can decide to keep only the first component and select the $j<p$ variables that have the highest absolute coefficient; the number $j$ might be based on the proportion of the number of variables (e.g., keep only the top 10% of the $p$ variables), or a fixed cutoff (e.g., considering a threshold on the normalized coefficients). This approach bears some resemblance with the Lasso operator in penalized regression (or PLS regression). Neither the value of $j$, nor the number of components to retain are obvious choices, though.

The problem with using PCA is that (1) measurements from all of the original variables are used in the projection to the lower dimensional space, (2) only linear relationships are considered, and (3) PCA or SVD-based methods, as well as univariate screening methods (t-test, correlation, etc.), do not take into account the potential multivariate nature of the data structure (e.g., higher order interaction between variables).

About point 1, some more elaborate screening methods have been proposed, for example principal feature analysis or stepwise method, like the one used for 'gene shaving' in gene expression studies. Also, sparse PCA might be used to perform dimension reduction and variable selection based on the resulting variable loadings. About point 2, it is possible to use kernel PCA (using the kernel trick) if one needs to embed nonlinear relationships into a lower dimensional space. Decision trees, or better the random forest algorithm, are probably better able to solve Point 3. The latter allows to derive Gini- or permutation-based measures of variable importance.

A last point: If you intend to perform feature selection before applying a classification or regression model, be sure to cross-validate the whole process (see §7.10.2 of the Elements of Statistical Learning, or Ambroise and McLachlan, 2002).

As you seem to be interested in R solution, I would recommend taking a look at the caret package which includes a lot of handy functions for data preprocessing and variable selection in a classification or regression context.

• There's a lot of good information here, but I'm surprised that there's no mention of EFA. I think of factor analysis as being appropriate to feature selection / dimensionality reduction, & PCA as really being only appropriate for re-representing your data such that the variables are uncorrelated. I guess you disagree? Apr 28, 2012 at 20:12
• I'm reluctant to recommend EFA without knowing what kind of data we are dealing with: introducing a model for the errors (which PCA doesn't) has certainly its advantage when dealing with targeted latent variables, or more generally when trying to uncover latent structures, but PCA (with its caveats) is mostly used to perform dimension reduction, or feature selection in large dimension, AFAICT. In the $n\ll p$ case, EFA would be inappropriate while sophisticated methods for variable selection do exist. I don't know of the OP's case, so I cannot say more, but this is a good remark.
– chl
Apr 28, 2012 at 20:28
• Two comments. First, you mention kPCA as one possible solution to your point 2. But how can kPCA be used for feature selection, when the eigenvectors/loadings are not available there? There is an extra question about that, and I argued there that it can't. Second, your second before last paragraph could improve a lot if you mentioned LASSO, as a preferred (?) way to do feature selection in regression. This thread remains very popular and many questions are closed as its duplicates, so it's important that your answer is as excellent as possible! Feb 5, 2015 at 15:38
• This article questions this approach: "The only way PCA is a valid method of feature selection is if the most important variables are the ones that happen to have the most variation in them. However this is usually not true. As an example, imagine you want to model the probability that an NFL team makes the playoffs. The number of wins an NFL team has (0 to 16) is much more useful for predicting the probability of making the playoffs then the team’s total rushing yards (in the thousands)." Jun 9, 2020 at 7:46
• @cs0815 Some approaches were suggested in various comments to this thread. See also Feature Selection for Unsupervised Learning.
– chl
Nov 9, 2020 at 10:16

Given a set of N features a PCA analysis will produce (1) the linear combination of the features with the highest variance (first PCA component), (2) the linear combination with the highest variance in the subspace orthogonal to the first PCA component etcetera (under the constraint that the coefficients of the combination form a vector with unit norm) Whether the linear combination with maximum variance is a "good" feature really depends on what you are trying to predict. For this reason I would say that being a PCA component and being a "good" features are (in general) two unrelated notions.

• (-1) I don't see how this answers the original question at all. Dec 13, 2014 at 2:21
• @amoeba This answers the question: "Does PCA compute a relative score for each input variable that you can use to filter out noninformative input variables?" with: No, if a feature is low variance it doesn't necessarily mean that it is also low informative. Jun 10, 2020 at 14:29

I skim read through the comments above and I believe quite a few have pointed out that PCA is not a good approach to feature selection. PCA offers dimensionality reduction but it is often misconceived with feature selection (as both tend to reduce the feature space in a sense). I would like to point out the key differences (absolutely open to opinions on this) that I feel between the two:

PCA is a actually a way of transforming your coordinate system to capture the variation in your data. This does not mean that the data is in any way more important than the other ones. It may be true in some cases while it may have no significance in some. PCA will only be relevant in the cases where the features having the most variation will actually be the ones most important to your problem statement and this must be known beforehand. You do normalize the data which tries to reduce this problem but PCA still is not a good method to be using for feature selection. I would list down some of the features that scikit-learn uses for feature selection to just give some direction:

1. Remove highly correlated features (Using Pearson's correlation matrix)
2. Recursive Feature Elimination (sklearn.feature_selection.RFE)
3. SelectFromModel (sklearn.feature_selection.SelectFromBest)

(1) above removes features(except 1) that are highly correlated amongst themselves. (2) and (3) run different algorithms to identify combination of features and checks which set gives the best accuracy while ranking the importance of features accordingly.

I'm not sure which language you are looking to use but there might be similar libraries to these.

Thanks!

You can not order features according to their variance, as the variance used in PCA is basically a multidimensional entity. You can only order features by the projection of the variance to certain direction you choose (which is normally the first principal compnonet.) So, in other word, whether a feature has more variance than anther one depends on how you choose your projection direction.

• I don't understand: each original feature does have a variance, and so one certainly can "order features according to their variance". Moreover, I don't understand how one can order them "by the projection of the variance to certain direction you choose". What do you mean by that? Mar 16, 2015 at 19:18
• You can indeed use variance to order features, just that then have anything to do with PCA, which treats all features together. Mar 17, 2015 at 21:51
• About the projection: If you have n features, a direction vector is just a unit vector in the n-dimensional space; the projection of your m instance vectors is the scale product of instance vector with that unit vector, which results in a m dimensional vector. And the variance of this m-dimensional vector is that "projection" of the variance of the dataset to the chosen direction. Mar 17, 2015 at 22:03
• -1. I think every single one of the three sentences in your answer is either wrong or so unclear that it's misleading. I agree with everything you wrote in the comments, but I have no idea how your answer can be interpreted to mean that. "You can not order features according to their variance" -- ?? "You can only order features by the projection of the variance to certain direction" -- ?? "whether a feature has more variance ... depends ... " -- ?? All of that is wrong. Mar 17, 2015 at 22:07
• I was not aware that this question has been asked so long time ago; and I agree with most of these responses. My point is: PCA is not appropriate for feature selection. Just nobody here want to say this directly. Mar 18, 2015 at 23:11

PCA tells us what features are more important, how?

In short: We find the first principal component one (PC1). Now PC1 is a linear combination of the variables (features). The variable with the highest weight (coefficient)(loading scores) in the linear equation is the most important feature.

Don't miss this wonderful video from StatQuest.

From my perspective, I agree with Sidharth Gurbani's answer.

"The pca is not suitable for variable selection."

You can even construct a dataset in which linear model works well but the performs badly with respect to the first principle component:

df <- data.frame(y=c(1,2,2,2,3),
x1=c(-.87, -1.22, 0,  1.22, .87),
x2=c(-.87, 1.22, 0,  -1.22, .87))
fit <- lm(y ~ x1+x2, data = df)
print(fit)
summary(fit)
pc <- princomp(df[2:3])
biplot(pc)
pc1 <- pc\$scores[,1]
fit2 <- lm(y ~ pc1)
summary(fit2)


Output:

Call:
lm(formula = y ~ x1 + x2, data = df)

Coefficients:
(Intercept)           x1           x2
2.0000       0.5747       0.5747

Warning: essentially perfect fit: summary may be unreliable
Call:
lm(formula = y ~ x1 + x2, data = df)

Residuals:
1          2          3          4          5
1.402e-16 -6.769e-17 -1.451e-16 -6.769e-17  1.402e-16

Coefficients:
Estimate Std. Error   t value Pr(>|t|)
(Intercept) 2.000e+00  8.340e-17 2.398e+16   <2e-16 ***
x1          5.747e-01  9.308e-17 6.175e+15   <2e-16 ***
x2          5.747e-01  9.308e-17 6.175e+15   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.865e-16 on 2 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:      1
F-statistic: 2.876e+31 on 2 and 2 DF,  p-value: < 2.2e-16

Call:
lm(formula = y ~ pc1)

Residuals:
1          2          3          4          5
-1.000e+00 -6.958e-17 -1.406e-17 -1.406e-17  1.000e+00

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2.000e+00  3.651e-01   5.477    0.012 *
pc1         2.503e-17  3.171e-01   0.000    1.000
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.8165 on 3 degrees of freedom
Multiple R-squared:  2.465e-32, Adjusted R-squared:  -0.3333
F-statistic: 7.396e-32 on 1 and 3 DF,  p-value: 1


The R-square of fit is 1, of fit2 is 0!

Since the value of y in the pc1 direction is always 1.

Let me make it more clearly:

The PCA is to reduce the dimension and retain the most variation simultaneously. She works well only if the variation is of interest. She doesn't guarantee that she won't hurt or break down the linear or other kind of relationship in the dataset.

So, use PCA only if the variation is of interest.

Some really great thoughts in the answers so far, and do a good job of explaining that the main job of PCA is to provide a few variables which are linear combinations of our original ones, not to select individual features of our original space. But it is actually possible to do something like that with the CUR decomposition. See CUR matrix decompositions for improved data analysis.

This has as hyperparameter an integer $$K$$ which is the number of PCA components to retain per usual, and then you select features to retain randomly based on their total squared contributions to the first $$K$$ principle components.

In particular, define the normalized statistical leverages to be $$\pi_j=\sum_{k=1}^K\frac{v_{j,k}^2}{K}$$ for each variable $$j$$ (here $$v_{j,k}$$ gives the $$j$$ element of singular vector $$k$$). The theory presented in the referenced article relies on keeping each variable with probability proportional to $$\pi_j$$. Keep this between you and me, but I usually like to just look at the variables with top scores instead of choosing randomly.

• I have added the title of the paper, if you don't mind. Dec 23, 2022 at 16:07