# Dimensionality Reduction Algorithm for Large Dataset?

I have a reasonably large (5k variables x 120k cases) that I'd like to run a dimensionality reduction algorithm on. I tried doing a simple Factor Analysis on it in SPSS, but it (predictably) barfed on a 3GB machine. I got an answer by truncating the data set to 2.5k variables and 25k cases, but I started wondering if there is an algorithm other than Factor Analysis/PCA that would handle the data set better.

My background in software development is much stronger than my background in statistics. It's my hope that there's an algorithm that could handle the original, unsampled data set more easily in 3GB of RAM. Does anyone know of such an algorithm?

Forgive me if this is a FAQ; I tried finding the answer before I posted.

EDIT: Ideally I'd like an existing implementation in something like R or SPSS, but since I'm a dev, even a software-based solution (like "try this feature of numpy") would be extremely helpful.

• If you are reducing dimensionality, i.e. variables, the data in SPSS should be 120K rows x 5K columns: the 5K x 5K corr. matrix will be analyzed in PCA/Factor. 5K x 5K x 8 = you need roughly only 200Mb RAM (pure minimum; with all front-end expences, 1Gb will suffice surely). Number of rows matter much less in SPSS Factor because the analysis reads row by row. So, there should be no problem except that it will take time to compute and process the corr. matrix. And it will be quicker if you know the number of factors in advance and the number is quite small. Apr 11, 2013 at 4:10
• +1 just for the barfed line. lol. Have you looked into sparse matrix formats in R? Apr 11, 2013 at 10:06
• not sure what spss does, but you may be running into memory problems because of the default output produced. Try to get all output as data sets, and supress any printed/html output. And only output the eigenvectors and eigenvalues. Apr 11, 2013 at 10:12
• Are you interested in dimension reduction for the sake of visualizing the data i.e. learning about latent structure in the data? Or are you ultimately interested in using this data for predicting some outcome? What your goal is would determine what algorithms would be worth pursuing. Apr 11, 2013 at 19:37
• Thanks for all the responses! @ ttnphns Having done a bit of research, it appears that the issue is simply storing the whole data set in memory. 120K*5K*8bytes = 4.8G of memory. @ probabilityislogic Those are great advice. I think my best bet is going to be to use another algorithm or to try a clever encoding of the data set in memory, per your first suggestion. @ Sameer Great points. We're interested in understanding the relationships among the variables, which is why Factor Analysis in particular is so interesting. Apr 11, 2013 at 20:54