54
$\begingroup$

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out noninformative input variables? Basically, I want to be able to order the original features in the data by variance or amount of information contained.

$\endgroup$
75
$\begingroup$

The basic idea when using PCA as a tool for feature selection is to select variables according to the magnitude (from largest to smallest in absolute values) of their coefficients (loadings). You may recall that PCA seeks to replace $p$ (more or less correlated) variables by $k<p$ uncorrelated linear combinations (projections) of the original variables. Let us ignore how to choose an optimal $k$ for the problem at hand. Those $k$ principal components are ranked by importance through their explained variance, and each variable contributes with varying degree to each component. Using the largest variance criteria would be akin to feature extraction, where principal component are used as new features, instead of the original variables. However, we can decide to keep only the first component and select the $j<p$ variables that have the highest absolute coefficient; the number $j$ might be based on the proportion of the number of variables (e.g., keep only the top 10% of the $p$ variables), or a fixed cutoff (e.g., considering a threshold on the normalized coefficients). This approach bears some resemblance with the Lasso operator in penalized regression (or PLS regression). Neither the value of $j$, nor the number of components to retain are obvious choices, though.

The problem with using PCA is that (1) measurements from all of the original variables are used in the projection to the lower dimensional space, (2) only linear relationships are considered, and (3) PCA or SVD-based methods, as well as univariate screening methods (t-test, correlation, etc.), do not take into account the potential multivariate nature of the data structure (e.g., higher order interaction between variables).

About point 1, some more elaborate screening methods have been proposed, for example principal feature analysis or stepwise method, like the one used for 'gene shaving' in gene expression studies. Also, sparse PCA might be used to perform dimension reduction and variable selection based on the resulting variable loadings. About point 2, it is possible to use kernel PCA (using the kernel trick) if one needs to embed nonlinear relationships into a lower dimensional space. Decision trees, or better the random forest algorithm, are probably better able to solve Point 3. The latter allows to derive Gini- or permutation-based measures of variable importance.

A last point: If you intend to perform feature selection before applying a classification or regression model, be sure to cross-validate the whole process (see §7.10.2 of the Elements of Statistical Learning, or Ambroise and McLachlan, 2002).


As you seem to be interested in R solution, I would recommend taking a look at the caret package which includes a lot of handy functions for data preprocessing and variable selection in a classification or regression context.

$\endgroup$
  • 1
    $\begingroup$ There's a lot of good information here, but I'm surprised that there's no mention of EFA. I think of factor analysis as being appropriate to feature selection / dimensionality reduction, & PCA as really being only appropriate for re-representing your data such that the variables are uncorrelated. I guess you disagree? $\endgroup$ – gung Apr 28 '12 at 20:12
  • 3
    $\begingroup$ I'm reluctant to recommend EFA without knowing what kind of data we are dealing with: introducing a model for the errors (which PCA doesn't) has certainly its advantage when dealing with targeted latent variables, or more generally when trying to uncover latent structures, but PCA (with its caveats) is mostly used to perform dimension reduction, or feature selection in large dimension, AFAICT. In the $n\ll p$ case, EFA would be inappropriate while sophisticated methods for variable selection do exist. I don't know of the OP's case, so I cannot say more, but this is a good remark. $\endgroup$ – chl Apr 28 '12 at 20:28
  • $\begingroup$ It's clearly a difficult, nuanced issue. Since you know a lot more about it than I do, I was interested in your opinion. +1, btw. $\endgroup$ – gung Apr 28 '12 at 20:40
  • 2
    $\begingroup$ Two comments. First, you mention kPCA as one possible solution to your point 2. But how can kPCA be used for feature selection, when the eigenvectors/loadings are not available there? There is an extra question about that, and I argued there that it can't. Second, your second before last paragraph could improve a lot if you mentioned LASSO, as a preferred (?) way to do feature selection in regression. This thread remains very popular and many questions are closed as its duplicates, so it's important that your answer is as excellent as possible! $\endgroup$ – amoeba Feb 5 '15 at 15:38
  • $\begingroup$ @chl, thank for your informative answer. You talk about "decide to keep only the first component". Why only the first component? With one component it is easy to rank features/variables. How would you do that with multiple components, say 3? How do you rank variables across components? I suppose you could go through each principal component, and pick the feature with the largest loading from that principal component, from the set of features that haven't already been picked. These again can be picked by a number ( j ) or by a threshold of normalized coefficients. Do you agree? $\endgroup$ – Sother Jun 23 '16 at 5:53
6
$\begingroup$

Given a set of N features a PCA analysis will produce (1) the linear combination of the features with the highest variance (first PCA component), (2) the linear combination with the highest variance in the subspace orthogonal to the first PCA component etcetera (under the constraint that the coefficients of the combination form a vector with unit norm) Whether the linear combination with maximum variance is a "good" feature really depends on what you are trying to predict. For this reason I would say that being a PCA component and being a "good" features are (in general) two unrelated notions.

$\endgroup$
  • $\begingroup$ (-1) I don't see how this answers the original question at all. $\endgroup$ – amoeba Dec 13 '14 at 2:21
-1
$\begingroup$

You can not order features according to their variance, as the variance used in PCA is basically a multidimensional entity. You can only order features by the projection of the variance to certain direction you choose (which is normally the first principal compnonet.) So, in other word, whether a feature has more variance than anther one depends on how you choose your projection direction.

$\endgroup$
  • 2
    $\begingroup$ I don't understand: each original feature does have a variance, and so one certainly can "order features according to their variance". Moreover, I don't understand how one can order them "by the projection of the variance to certain direction you choose". What do you mean by that? $\endgroup$ – amoeba Mar 16 '15 at 19:18
  • $\begingroup$ You can indeed use variance to order features, just that then have anything to do with PCA, which treats all features together. $\endgroup$ – James LI Mar 17 '15 at 21:51
  • $\begingroup$ About the projection: If you have n features, a direction vector is just a unit vector in the n-dimensional space; the projection of your m instance vectors is the scale product of instance vector with that unit vector, which results in a m dimensional vector. And the variance of this m-dimensional vector is that "projection" of the variance of the dataset to the chosen direction. $\endgroup$ – James LI Mar 17 '15 at 22:03
  • 1
    $\begingroup$ -1. I think every single one of the three sentences in your answer is either wrong or so unclear that it's misleading. I agree with everything you wrote in the comments, but I have no idea how your answer can be interpreted to mean that. "You can not order features according to their variance" -- ?? "You can only order features by the projection of the variance to certain direction" -- ?? "whether a feature has more variance ... depends ... " -- ?? All of that is wrong. $\endgroup$ – amoeba Mar 17 '15 at 22:07
  • 2
    $\begingroup$ I was not aware that this question has been asked so long time ago; and I agree with most of these responses. My point is: PCA is not appropriate for feature selection. Just nobody here want to say this directly. $\endgroup$ – James LI Mar 18 '15 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.