2
$\begingroup$

I ran a Multiple Factor Analysis on a data set with 3,924 rows and 96 columns, of which six are (unordered) categorical, with 12-14 categories in each, and the rest are numeric, mean-centered and scaled by one-standard-deviation. My goal is dimension reduction, in order to visualize the results of PAM clustering by plotting the first two or three dimensions and coloring the points by assigned partition, as well as highlighting each medoid.

I found that no one dimension of PCA space explains more than a small fraction of variance in the data:

       eigenvalue percentage of variance cumulative percentage of variance
comp 1  1.0350075               2.466873                          2.466873
comp 2  0.8243004               1.964666                          4.431539
comp 3  0.8093599               1.929057                          6.360596
comp 4  0.7587070               1.808329                          8.168924
comp 5  0.6495978               1.548274                          9.717198
comp 6  0.6328384               1.508329                         11.225527

What should I make of this situation? Can I still use the first two PCA dimensions as a quick 2D approximation of the data set, or will they just fail to represent the data accurately?

Is there an alternative dimension reduction technique I could/should use? All of the reviews of nonlinear dimension reduction I've read were somewhat equivocal on their usefulness compared to PCA, except on fabricated data like the swiss roll data set, so I've been hesitant to use them.

Edit: here are the PCA results from just the numerical variables:

        eigenvalue percentage of variance cumulative percentage of variance
comp 1   5.1704992              5.7449991                          5.744999
comp 2   4.0469449              4.4966055                         10.241605
comp 3   3.8800122              4.3111247                         14.552729
comp 4   3.0606430              3.4007144                         17.953444
comp 5   2.7176048              3.0195609                         20.973005
comp 6   2.4725503              2.7472781                         23.720283
Improve this question
$\endgroup$
15
  • 1
    $\begingroup$ It appears that the variables (96 columns) are not related to each other. What does the result of correlation matrix suggest? Are there any significant correlations (at least amongst numeric variables)? If so, what are the correlation coefficients (r values) like? It will be a big matrix and would take some time to check. $\endgroup$
    – rnso
    Commented Apr 9, 2015 at 4:45
  • $\begingroup$ @rnso they're very weakly correlated overall. I hadn't even thought to put that together. If the variables are very weakly correlated, then the variance-maximizing basis is not much different from the original basis, right? Maybe then a nonlinear technique would be better after all. $\endgroup$
    – shadowtalker
    Commented Apr 9, 2015 at 4:57
  • 1
    $\begingroup$ It may be useful to convert all categorical variables to numeric (e.g. by using R command: var1 = as.numeric(var1) ) and try simple principal component analysis using R commands: res = prcomp(mydf, scale = TRUE); res; biplot(res) . It may be helpful if you post output of res and this plot here. $\endgroup$
    – rnso
    Commented Apr 9, 2015 at 6:21
  • 1
    $\begingroup$ What are the results of simple PCA on numeric data only (excluding categorical variables)? I want to know if that also shows first and second component with very low variance explanation. $\endgroup$
    – rnso
    Commented Apr 9, 2015 at 13:52
  • 1
    $\begingroup$ @rnso You could also pretty easily try using Spearman correlation or mutual information to generate your correlation (or more generically, similarity matrix) and get a sense for whether using nonlinear approaches would be more successful. If you decide to try some nonlinear methods for dimensions reduction, NMF may also be worth considering. $\endgroup$
    – Keith Hughitt
    Commented Aug 31, 2016 at 11:26

1 Answer 1

Reset to default
2
$\begingroup$

Despite the term multiple factor analysis (MFA), used to describe the factor analysis (FA) that you've performed, it seems to me like a standard PCA approach (or, FA via PCA, at best), which focuses on principal components. Instead, I suggest you to use exploratory factor analysis (EFA) and then confirmatory factor analysis (CFA), both of which focus on latent variables approach. EFA serves as an alternative dimensionality reduction technique with an added benefits of discovering latent factor structure, which has more explanatory power. Let me know, if you need further help.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Yes, it's just PCA that is generalized to "groups" of variables. I'm not sure why the name "multiple factor analysis" was chosen; it seems ill-suited. As for EFA instead of PCA, I can try it but it doesn't seem like it would help much. $\endgroup$
    – shadowtalker
    Commented Apr 9, 2015 at 4:22
  • $\begingroup$ @ssdecontrol: I'm aware of the answer that you referenced. However, based on both that answer and this answer, PCA $\Rightarrow$ FA, when n $\Rightarrow$ $\infty$ or upon some other conditions. Thus, I would give EFA a try and share the findings. $\endgroup$
    – Aleksandr Blekh
    Commented Apr 9, 2015 at 4:58
  • $\begingroup$ I just tried it with the FAMD function in R's FactoMineR package, and it ran for a long time without finishing. Is there another, faster implementation that can handle mixed numerical and unordered categorical data? $\endgroup$
    – shadowtalker
    Commented Apr 9, 2015 at 5:07
  • $\begingroup$ @ssdecontrol: Sure. There are some options. For EFA, which you're currently interested in, you could start with standard function stats::factanal(). However, I recommend psych package as a better alternative (you might want to load GPArotation, if you want to try some rotations, not included in psych). Since your data set contains a mixture of continuous and categorical variables, you need to calculate polychoric correlations. For that, use hetcor() function from the polycor package. $\endgroup$
    – Aleksandr Blekh
    Commented Apr 9, 2015 at 5:24
  • $\begingroup$ Polychoric correlation still assumes that the categorical variables are ordered. The issue is still the fact that there's no way to calculate a correlation when unordered categorical variables are involved; FactoMineR provides a workaround by allowing the user to specify "groups" of dummy variables that will be analyzed together. $\endgroup$
    – shadowtalker
    Commented Apr 9, 2015 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.