I have a 0/1 sparse matrix with 500 k columns and 3 M rows and I want to do reduce the number of columns.
Clearly I cannot load this into
R, as is, so
prcomp is out.
(I cannot even create the Gram matrix in
too many elements specified.)
However, it appears that I should be able follow this path:
- Reduce the number of columns to 200 k by dropping very sparse ones
- Compute the Gram matrix by scanning the input file without loading the whole thing
- Compute the Gram matrix eigenvectors in
Any better suggestions?