2
$\begingroup$

I have a data set with $n=5000$ and $p=40$. It involves both categorical and continuous variables. I want to use PCA as a feature extraction method. I consider 5 components in PCA. My question is: Do the components resulted from PCA become the new variables and represent the variables instead of the original ones? In other words, should I use these 5 components instead of my original 40 variables in the next stages of data mining including clustering, rule association and so on?

My other question is: when should we use PCA as feature extraction? Specifically in my case, should I use it for feature extraction or the number of my variables aren't large enough to make PCA necessary and the downstream analysis do not require any feature extraction step?

$\endgroup$
1
  • $\begingroup$ By "feature selection" do you mean "feature extraction"? $\endgroup$
    – Michael M
    Commented Nov 7, 2017 at 11:42

2 Answers 2

2
$\begingroup$

PCA is a common tactic of feature extraction for further analyses. The scores of the first few principal components are then considered as derived variables, just as if you would take logs, ratios or other transformations of the original set of variables.

Here are a few typical situations where principal components are used for subsequent analyses:

  1. A group of highly correlated covariables in a modelling setting (e.g. number of rooms in a house, living surface, volume in a house price model): These might be troublesome for interpretation of the regression coefficients and their associated inference. Here you could try to compress the three "size" variables (after taking natural logs each) through a PCA and use just the first PC to represent size. Often, a better tactic is to "decorrelate" the covariables by hand, i.e. by using log(room number), log(surface living / room number), log(volume / surface living). Transformations like this are almost always possible and provide much easier interpretation (which is actually the aim of all the fuzz here).

  2. Overfitting in a modelling setting, i.e. too many covariables compared to the number of independent observations. E.g. 1000 observations and 100 covariables. One idea is to "compress" the 100 covariables to the first few relevant PCAs and hope that they retain most statistical association with the response variable. This is basically the tactic you have proposed. It is done quite frequently and is called principal component regression. Because everything is linear, you can even map the coefficients of the regression to the original variables (if required). Alternatives are e.g. penalized modelling like ridge- or Lasso-regression.

  3. Any other sort of "higher" dimensional analysis. Imagine e.g. you have a large questionnaire with 200 questions along with demographic information like sex, age etc. Instead of running 200 Wilcoxon- or t-tests to test for association between the answers to the questionnaire and sex, quite commonly, only the first few PCs are chosen instead of the 200 questions. Ideally, this increases the power and does not suffer from extreme multiple testing issues.

These settings all share the same problems:

  • There is a danger of not looking enough into the raw data, so the chance of not detecting wrong values could be larger than with explicit transformations.
  • PCA is prone to outliers, so log-transforms are often your friend.
  • Unclear how categorical variables should enter PCA. As dummy variables? This is sometimes being considered as bad habit, but what are the alternatives...?
  • More difficult to interpret the results. What does it finally mean if the third PC is significantly associated with sex? Of course you can say the relevant PC is just a weighted sum of the standardized original variables, but it will stay clumsy.

So the answer to the first question is clearly "yes". And the second one, maybe not unexpected, depends on the situation.

$\endgroup$
2
  • 1
    $\begingroup$ Ha, thanks@amoeba: When I reach your stunning 40k, I will be in pension or even one step further! $\endgroup$
    – Michael M
    Commented Nov 7, 2017 at 15:05
  • 1
    $\begingroup$ @Micheal M:Thanks for your response.We can use PCAMIX(PCA for mixed data) or FAMD(Factor Analysis of mixed data) when having mixed data. $\endgroup$
    – far
    Commented Nov 7, 2017 at 15:41
0
$\begingroup$

I would say no. Principal components analysis is a form of exploratory factor analysis (EFA) and does not develop a formal scale for research purposes. The end-goal of EFA typically is to conduct a confirmatory factor analysis (CFA) which tests a specific structure rather than one that is merely optimal in a given dataset.

To avoid this mess, a pragmatic research approach that is well received in the research community is parceling. Parceling is done by taking a sum-score of appropriately transformed data. An important aspect to parceling is that the researcher must specify a priori how many such scales, which indicators are used for them, and what representation thereof are included in a parcel. With 40 measures, this is easily done and precludes machine learning.

PCA is capable of ranking features in terms of their factor loading. If you select the top $k<p$ features in terms of their eigenvalues, this combination of features and their predictions of other features results in minimum $R^2$ in that sample. That has pretty limited interpretation and generalizability for research. Daniela Witten has an R package SparCl for sparse clustering based on using a lasso ($\mathcal{L}_1$ penalty) for this purpose. Here, if a factor loading is very close to 0, it is simply set to zero, and this affects the non-zero loadings' estimates. I wager this has better out-of-sample generalizability, and better MSE.

$\endgroup$
5
  • 2
    $\begingroup$ PCA is an extremely common tactic of feature extraction for further analyses, be it regression, clustering, visualization, etc. Answering "No" to the title question is sort of misleading in that it makes the impression that this is never or at least rarely done. $\endgroup$
    – amoeba
    Commented Nov 7, 2017 at 15:01
  • $\begingroup$ @amoeba I described what eigenvalue-ranked feature selection via PCA does and what it doesn't do. I offered a critique and alternative approach from a ML perspective. Whether PCA feature-selection can be done or whether it has been done are separate questions, and I'd agree that historically it has been an exact answer to the approximate problem of feature selection. But the favorable reception to and performance of penalized regression moves me to offer what I consider a better alternative. $\endgroup$
    – AdamO
    Commented Nov 7, 2017 at 15:45
  • $\begingroup$ There might be some confusion here because in your comment you are talking about feature selection (I fully agree that PCA is terrible for feature selection) whereas I meant feature "construction". E.g. when one does PCA to reduce dimensionality from 10k to 50 before running clustering, or before running regression ("principal component regression"), there is no feature selection going on. The question text was a bit unclear in the beginning but the OP clarified that they did not mean feature selection. Maybe you could edit your answer to clarify which of the two you mean when you say "No". $\endgroup$
    – amoeba
    Commented Nov 7, 2017 at 15:57
  • $\begingroup$ @amoeba I think the preferred terminology for what you are referring to is a construct, latent construct, scale, psychometric quantity and so on. For instance, if I pilot a questionnaire of 100 questions related to depression, PCA would be interested in: "How do I measure depression?" The description of the probability model is that there are one or more latent variables for which a participants' response to those 100 questions are manifest. You may have to select all 100 questions to represent 1 or more dimensions of depression. Feature selection asks a very different question, (ctd) $\endgroup$
    – AdamO
    Commented Nov 7, 2017 at 16:16
  • $\begingroup$ @amoeba Feature selection is not concerned with estimating a latent construct, we would instead say, "what variable(s) most effectively cluster participant responses to all 100 questions?" when applied with appropriate ML methodology, like cross-validation, I think the generalizability and interpretability of those responses may be better than forward-imputing PCA scales in subsequent analyses. $\endgroup$
    – AdamO
    Commented Nov 7, 2017 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.