-1
$\begingroup$

I tried running prcomp() on my training set, which contains some categorical/factor predictors (as well as a binary response), and was given an error saying my data needs to be numeric. Can PCA not be performed at all with categorical features? Or is there a way to work around the function...

Thank you!

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
1
$\begingroup$

The short answer is yes, PCA is intended for continuously scaled features. Categorical features which are nominally scaled (e.g., the set of teams in the NFL is a nominally scaled variable) and stored as such would cause this function to give an error.

One workaround would be to impose a ranking on a categorical feature that would convert it into an acceptable ordinal scale but that would be an imposition of a rule.

Another workaround would be to convert the categorical features into a set of dummy variables or numeric effects as described in the link below. There are those that contend that PCA is invariant to mixtures of numeric variables such as dummy, ordinal and higher scaled features but that has not been my experience. My experience is that you get results with imbalanced latent dimensions which, based on the loadings, are heavily skewed towards the dummy variables.

My opinion is that the best solution would be to use correspondence analysis (CA). CA is described as 'PCA for categorical features'. It is well reviewed in the second link below. It may require grouping the continuous variables with a large range of possible values into a smaller set of buckets, depending on the software used.

ADDITIONAL CODING SYSTEMS FOR CATEGORICAL VARIABLES IN REGRESSION ANALYSIS https://stats.idre.ucla.edu/sas/webbooks/reg/chapter5/regression-with-saschapter-5-additional-coding-systems-for-categorical-variables-in-regressionanalysis/

Correspondence Analysis in R: The Ultimate Guide for the Analysis, the Visualization and the Interpretation - R software and data mining http://www.sthda.com/english/wiki/print.php?id=228

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you very much! I am reading into correspondence analysis now, my ultimate objective is to predict my test set using principle components, do you think this will work? $\endgroup$
    – SugarMarsh
    Commented Aug 2, 2019 at 11:35
  • $\begingroup$ Sure, just expect there to be an inevitable loss in predictive power since the resulting dimensions will recover a smaller percentage of the total variance in the inputs. Given that, why do CA or PCA at all? If you want to find a reduced set of predictors why not use a variable selection method such as the Lasso? See r-bloggers.com/ridge-regression-and-the-lasso $\endgroup$
    – user234562
    Commented Aug 2, 2019 at 12:07
  • $\begingroup$ I have already selected my variables, I am trying to build a predictive model, then given the same predictors, estimate my response variable (not sure if this makes sense), if CA lacks predictive power, should I just not use this method altogether? I appreciate your help! $\endgroup$
    – SugarMarsh
    Commented Aug 2, 2019 at 12:13
  • $\begingroup$ Why not try both CA and the predictors as separate models? Then see which one provides the better empirical fit. It makes more work for you but, per Gelman's concept of the multiverse, it might prove to be more insightful. Increasing Transparency Through a Multiverse Analysis stat.columbia.edu/~gelman/research/published/… $\endgroup$
    – user234562
    Commented Aug 2, 2019 at 15:15
1
$\begingroup$

Take my answer as a comment more than a true answer (I am a new contributor so i cannot comment yet). If you can compute the varcov of the variables, then you can use PCA on that varcov matrix: of course you can compute the covariances between random variables even when they are binomial variables that numerically represent a categoriacal variable referred to two categories only (the same holds for multi-category variables). So be sure that you are representing your categorical variable via numbers (0-1 for a binomial categoriacal variable or 0-1-2,... for a categorical variable with more than 2 categories) and calculating the varcov correctly. Having said that, personally, I would prefer to keep them outside the PCA, especially if they are binomial and especially if you have just a few of them compared to the total number of features: for example, transform the set of non-categorical features via PCA to obtain a set of orthogonal features, then add the categorical variables to the set of simplified orthogonal features.

Code in python for the spectral decomposition of a simple varcov matrix obtained from two binomial variables

import numpy as np
import scipy.linalg as la
a=[0,1,0,1,0,1]
b=[0,0,0,0,1,1]
eigval, eigvec=la.eig(np.cov(a,b))
Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.