I have a reasonably large (5k variables x 120k cases) that I'd like to run a dimensionality reduction algorithm on. I tried doing a simple Factor Analysis on it in SPSS, but it (predictably) barfed on a 3GB machine. I got an answer by truncating the data set to 2.5k variables and 25k cases, but I started wondering if there is an algorithm other than Factor Analysis/PCA that would handle the data set better.

My background in software development is much stronger than my background in statistics. It's my hope that there's an algorithm that could handle the original, unsampled data set more easily in 3GB of RAM. Does anyone know of such an algorithm?

Forgive me if this is a FAQ; I tried finding the answer before I posted.

EDIT: Ideally I'd like an existing implementation in something like R or SPSS, but since I'm a dev, even a software-based solution (like "try this feature of numpy") would be extremely helpful.

  • 1
    $\begingroup$ If you are reducing dimensionality, i.e. variables, the data in SPSS should be 120K rows x 5K columns: the 5K x 5K corr. matrix will be analyzed in PCA/Factor. 5K x 5K x 8 = you need roughly only 200Mb RAM (pure minimum; with all front-end expences, 1Gb will suffice surely). Number of rows matter much less in SPSS Factor because the analysis reads row by row. So, there should be no problem except that it will take time to compute and process the corr. matrix. And it will be quicker if you know the number of factors in advance and the number is quite small. $\endgroup$
    – ttnphns
    Apr 11, 2013 at 4:10
  • 1
    $\begingroup$ +1 just for the barfed line. lol. Have you looked into sparse matrix formats in R? $\endgroup$ Apr 11, 2013 at 10:06
  • $\begingroup$ not sure what spss does, but you may be running into memory problems because of the default output produced. Try to get all output as data sets, and supress any printed/html output. And only output the eigenvectors and eigenvalues. $\endgroup$ Apr 11, 2013 at 10:12
  • $\begingroup$ Are you interested in dimension reduction for the sake of visualizing the data i.e. learning about latent structure in the data? Or are you ultimately interested in using this data for predicting some outcome? What your goal is would determine what algorithms would be worth pursuing. $\endgroup$
    – Sameer
    Apr 11, 2013 at 19:37
  • $\begingroup$ Thanks for all the responses! @ ttnphns Having done a bit of research, it appears that the issue is simply storing the whole data set in memory. 120K*5K*8bytes = 4.8G of memory. @ probabilityislogic Those are great advice. I think my best bet is going to be to use another algorithm or to try a clever encoding of the data set in memory, per your first suggestion. @ Sameer Great points. We're interested in understanding the relationships among the variables, which is why Factor Analysis in particular is so interesting. $\endgroup$
    – sigpwned
    Apr 11, 2013 at 20:54

1 Answer 1


Random forests are robust. They are not impacted by outliers.
Gradient boosted trees are great at fitting or over fitting the data. The combination is fast, handles classical or categorical data, and can handle very large data.

Random forests of gradient boosted trees easily handle problems of this complexity and size. http://dl.acm.org/citation.cfm?id=1755828

  • $\begingroup$ Very interesting. I haven't played with random forests. I'll look into them. Thank you! $\endgroup$
    – sigpwned
    Apr 11, 2013 at 20:58
  • 1
    $\begingroup$ I have seen them effectively and conveniently handle data that is 50k rows and 50k cols of mixed type (categorical, integer, continuous). For going from 50k dimensions to 30 they are great. Better tools are then used to go from 30 to below. $\endgroup$ Apr 11, 2013 at 21:18
  • $\begingroup$ Just interested: How are random forests used for dimensionality reduction? $\endgroup$
    – Michael M
    Oct 5, 2013 at 16:54
  • $\begingroup$ @Michael - lets say you have 50k variables, and only 30 are relevant. How do you find the important variables, or maybe the 200 variables with the important ones as an unknown but more accessible subset. Random forest. In this way you hae reduced from a 50k dimensional set to a 200 dimensional set. $\endgroup$ Oct 7, 2013 at 14:14
  • $\begingroup$ @EngrStudent: How would you choose the response variable of the forest? $\endgroup$
    – Michael M
    Oct 7, 2013 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.