Using principal component analysis (PCA) for feature selection

Question

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out noninformative input variables? Basically, I want to be able to order the original features in the data by variance or amount of information contained.

Answer 1

The basic idea when using PCA as a tool for feature selection is to select variables according to the magnitude (from largest to smallest in absolute values) of their coefficients (loadings). You may recall that PCA seeks to replace $p$ (more or less correlated) variables by $k<p$ uncorrelated linear combinations (projections) of the original variables. Let us ignore how to choose an optimal $k$ for the problem at hand. Those $k$ principal components are ranked by importance through their explained variance, and each variable contributes with varying degree to each component. Using the largest variance criteria would be akin to feature extraction, where principal component are used as new features, instead of the original variables. However, we can decide to keep only the first component and select the $j<p$ variables that have the highest absolute coefficient; the number $j$ might be based on the proportion of the number of variables (e.g., keep only the top 10% of the $p$ variables), or a fixed cutoff (e.g., considering a threshold on the normalized coefficients). This approach bears some resemblance with the Lasso operator in penalized regression (or PLS regression). Neither the value of $j$, nor the number of components to retain are obvious choices, though.

The problem with using PCA is that (1) measurements from all of the original variables are used in the projection to the lower dimensional space, (2) only linear relationships are considered, and (3) PCA or SVD-based methods, as well as univariate screening methods (t-test, correlation, etc.), do not take into account the potential multivariate nature of the data structure (e.g., higher order interaction between variables).

About point 1, some more elaborate screening methods have been proposed, for example principal feature analysis or stepwise method, like the one used for 'gene shaving' in gene expression studies. Also, sparse PCA might be used to perform dimension reduction and variable selection based on the resulting variable loadings. About point 2, it is possible to use kernel PCA (using the kernel trick) if one needs to embed nonlinear relationships into a lower dimensional space. Decision trees, or better the random forest algorithm, are probably better able to solve Point 3. The latter allows to derive Gini- or permutation-based measures of variable importance.

A last point: If you intend to perform feature selection before applying a classification or regression model, be sure to cross-validate the whole process (see §7.10.2 of the Elements of Statistical Learning, or Ambroise and McLachlan, 2002).

---

As you seem to be interested in R solution, I would recommend taking a look at the caret package which includes a lot of handy functions for data preprocessing and variable selection in a classification or regression context.

Answer 2

Given a set of N features a PCA analysis will produce (1) the linear combination of the features with the highest variance (first PCA component), (2) the linear combination with the highest variance in the subspace orthogonal to the first PCA component etcetera (under the constraint that the coefficients of the combination form a vector with unit norm) Whether the linear combination with maximum variance is a "good" feature really depends on what you are trying to predict. For this reason I would say that being a PCA component and being a "good" features are (in general) two unrelated notions.

Answer 3

I skim read through the comments above and I believe quite a few have pointed out that PCA is not a good approach to feature selection. PCA offers dimensionality reduction but it is often misconceived with feature selection (as both tend to reduce the feature space in a sense). I would like to point out the key differences (absolutely open to opinions on this) that I feel between the two:

PCA is a actually a way of transforming your coordinate system to capture the variation in your data. This does not mean that the data is in any way more important than the other ones. It may be true in some cases while it may have no significance in some. PCA will only be relevant in the cases where the features having the most variation will actually be the ones most important to your problem statement and this must be known beforehand. You do normalize the data which tries to reduce this problem but PCA still is not a good method to be using for feature selection. I would list down some of the features that scikit-learn uses for feature selection to just give some direction:

1. Remove highly correlated features (Using Pearson's correlation matrix)
2. Recursive Feature Elimination (sklearn.feature_selection.RFE)
3. SelectFromModel (sklearn.feature_selection.SelectFromBest)

(1) above removes features(except 1) that are highly correlated amongst themselves. (2) and (3) run different algorithms to identify combination of features and checks which set gives the best accuracy while ranking the importance of features accordingly.

I'm not sure which language you are looking to use but there might be similar libraries to these.

Thanks!

Answer 4

From my perspective, I agree with Sidharth Gurbani's answer.

"The pca is not suitable for variable selection."

You can even construct a dataset in which linear model works well but the performs badly with respect to the first principle component:

df <- data.frame(y=c(1,2,2,2,3),
x1=c(-.87, -1.22, 0,  1.22, .87),
x2=c(-.87, 1.22, 0,  -1.22, .87))
fit <- lm(y ~ x1+x2, data = df)
print(fit)
summary(fit)
pc <- princomp(df[2:3])
biplot(pc)
pc1 <- pc$scores[,1]
fit2 <- lm(y ~ pc1)
summary(fit2)

Output:

Call:
lm(formula = y ~ x1 + x2, data = df)

Coefficients:
(Intercept)           x1           x2  
     2.0000       0.5747       0.5747  

Warning: essentially perfect fit: summary may be unreliable
Call:
lm(formula = y ~ x1 + x2, data = df)

Residuals:
         1          2          3          4          5 
 1.402e-16 -6.769e-17 -1.451e-16 -6.769e-17  1.402e-16 

Coefficients:
             Estimate Std. Error   t value Pr(>|t|)    
(Intercept) 2.000e+00  8.340e-17 2.398e+16   <2e-16 ***
x1          5.747e-01  9.308e-17 6.175e+15   <2e-16 ***
x2          5.747e-01  9.308e-17 6.175e+15   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.865e-16 on 2 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:      1 
F-statistic: 2.876e+31 on 2 and 2 DF,  p-value: < 2.2e-16


Call:
lm(formula = y ~ pc1)

Residuals:
         1          2          3          4          5 
-1.000e+00 -6.958e-17 -1.406e-17 -1.406e-17  1.000e+00 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)  
(Intercept) 2.000e+00  3.651e-01   5.477    0.012 *
pc1         2.503e-17  3.171e-01   0.000    1.000  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.8165 on 3 degrees of freedom
Multiple R-squared:  2.465e-32, Adjusted R-squared:  -0.3333 
F-statistic: 7.396e-32 on 1 and 3 DF,  p-value: 1

The R-square of fit is 1, of fit2 is 0!

Since the value of y in the pc1 direction is always 1.

Let me make it more clearly:

The PCA is to reduce the dimension and retain the most variation simultaneously. She works well only if the variation is of interest. She doesn't guarantee that she won't hurt or break down the linear or other kind of relationship in the dataset.

So, use PCA only if the variation is of interest.

Answer 5

You can not order features according to their variance, as the variance used in PCA is basically a multidimensional entity. You can only order features by the projection of the variance to certain direction you choose (which is normally the first principal compnonet.) So, in other word, whether a feature has more variance than anther one depends on how you choose your projection direction.

Answer 6

PCA tells us what features are more important, how?

In short: We find the first principal component one (PC1). Now PC1 is a linear combination of the variables (features). The variable with the highest weight (coefficient)(loading scores) in the linear equation is the most important feature.

Don't miss this wonderful video from StatQuest.

Answer 7

Some really great thoughts in the answers so far, and do a good job of explaining that the main job of PCA is to provide a few variables which are linear combinations of our original ones, not to select individual features of our original space. But it is actually possible to do something like that with the CUR decomposition. See CUR matrix decompositions for improved data analysis.

This has as hyperparameter an integer $K$ which is the number of PCA components to retain per usual, and then you select features to retain randomly based on their total squared contributions to the first $K$ principle components.

In particular, define the normalized statistical leverages to be $\pi_j=\sum_{k=1}^K\frac{v_{j,k}^2}{K}$ for each variable $j$ (here $v_{j,k}$ gives the $j$ element of singular vector $k$). The theory presented in the referenced article relies on keeping each variable with probability proportional to $\pi_j$. Keep this between you and me, but I usually like to just look at the variables with top scores instead of choosing randomly.