# PCA/MFA for (graphical) dimension reduction: what to do with very small explained variance?

I ran a Multiple Factor Analysis on a data set with 3,924 rows and 96 columns, of which six are (unordered) categorical, with 12-14 categories in each, and the rest are numeric, mean-centered and scaled by one-standard-deviation. My goal is dimension reduction, in order to visualize the results of PAM clustering by plotting the first two or three dimensions and coloring the points by assigned partition, as well as highlighting each medoid.

I found that no one dimension of PCA space explains more than a small fraction of variance in the data:

       eigenvalue percentage of variance cumulative percentage of variance
comp 1  1.0350075               2.466873                          2.466873
comp 2  0.8243004               1.964666                          4.431539
comp 3  0.8093599               1.929057                          6.360596
comp 4  0.7587070               1.808329                          8.168924
comp 5  0.6495978               1.548274                          9.717198
comp 6  0.6328384               1.508329                         11.225527


What should I make of this situation? Can I still use the first two PCA dimensions as a quick 2D approximation of the data set, or will they just fail to represent the data accurately?

Is there an alternative dimension reduction technique I could/should use? All of the reviews of nonlinear dimension reduction I've read were somewhat equivocal on their usefulness compared to PCA, except on fabricated data like the swiss roll data set, so I've been hesitant to use them.

Edit: here are the PCA results from just the numerical variables:

        eigenvalue percentage of variance cumulative percentage of variance
comp 1   5.1704992              5.7449991                          5.744999
comp 2   4.0469449              4.4966055                         10.241605
comp 3   3.8800122              4.3111247                         14.552729
comp 4   3.0606430              3.4007144                         17.953444
comp 5   2.7176048              3.0195609                         20.973005
comp 6   2.4725503              2.7472781                         23.720283

• It appears that the variables (96 columns) are not related to each other. What does the result of correlation matrix suggest? Are there any significant correlations (at least amongst numeric variables)? If so, what are the correlation coefficients (r values) like? It will be a big matrix and would take some time to check.
– rnso
Apr 9, 2015 at 4:45
• @rnso they're very weakly correlated overall. I hadn't even thought to put that together. If the variables are very weakly correlated, then the variance-maximizing basis is not much different from the original basis, right? Maybe then a nonlinear technique would be better after all. Apr 9, 2015 at 4:57
• It may be useful to convert all categorical variables to numeric (e.g. by using R command: var1 = as.numeric(var1) ) and try simple principal component analysis using R commands: res = prcomp(mydf, scale = TRUE); res; biplot(res) . It may be helpful if you post output of res and this plot here.
– rnso
Apr 9, 2015 at 6:21
• What are the results of simple PCA on numeric data only (excluding categorical variables)? I want to know if that also shows first and second component with very low variance explanation.
– rnso
Apr 9, 2015 at 13:52
• @rnso You could also pretty easily try using Spearman correlation or mutual information to generate your correlation (or more generically, similarity matrix) and get a sense for whether using nonlinear approaches would be more successful. If you decide to try some nonlinear methods for dimensions reduction, NMF may also be worth considering. Aug 31, 2016 at 11:26

• @ssdecontrol: I'm aware of the answer that you referenced. However, based on both that answer and this answer, PCA $\Rightarrow$ FA, when n $\Rightarrow$ $\infty$ or upon some other conditions. Thus, I would give EFA a try and share the findings. Apr 9, 2015 at 4:58
• I just tried it with the FAMD function in R's FactoMineR package, and it ran for a long time without finishing. Is there another, faster implementation that can handle mixed numerical and unordered categorical data? Apr 9, 2015 at 5:07
• @ssdecontrol: Sure. There are some options. For EFA, which you're currently interested in, you could start with standard function stats::factanal(). However, I recommend psych package as a better alternative (you might want to load GPArotation, if you want to try some rotations, not included in psych). Since your data set contains a mixture of continuous and categorical variables, you need to calculate polychoric correlations. For that, use hetcor() function from the polycor package. Apr 9, 2015 at 5:24
• Polychoric correlation still assumes that the categorical variables are ordered. The issue is still the fact that there's no way to calculate a correlation when unordered categorical variables are involved; FactoMineR provides a workaround by allowing the user to specify "groups" of dummy variables that will be analyzed together. Apr 9, 2015 at 13:49