How can one extract meaningful factors from a sparse matrix? I am interested in finding some practical (and reasonably well accepted) techniques for finding the underlying factors of a sparse matrix.  
Specifically, I have a very large sparse matrix whose cells appear to be populated from an approximately geometric distribution.  In its natural form the matrix is square.  The values in the cells represent item x item co-occurrences under case 1 over the diagonal and under case 2 under the diagonal.  If necessary I can subset the matrix to particularly interesting items in order to make it rectangular.  I believe that there are meaningful factors underlying this structure.  However my understanding is that because the matrix is sparse factor analysis is not an appropriate approach.  What approach can I take that will make it most likely that I can find interpretable patterns in the data?
I saw that there was another question asking for references on sparse variants of PCA, but I think I'm looking for something more akin to an obliquely rotated factor solution.  I'm willing to dig into suggested readings somewhat, but my prior experience with factor analysis (and related techniques) is limited, and I prefer a relatively straightforward answer (one with R code is even better).
 A: I might suggest non-negative matrix factorization. The iterative algorithm of Lee and Seung is easy to implement and should be amenable to sparse matrices (although it involves Hadamard products, which some sparse matrix packages may not support.).
A: One has to be careful about the meaning of the word sparse. Your matrix contains many zeroes and one may represent such a matrix in a sparse way (to save on storage). But since the figures represent co-occurrences these zeroes are still to be considered informative (they are not missing; they are not structurally zero) and should therefore be taken into account when modeling the content of the matrix. The many zeroes and the skewness (approximately geometric) would suggest to use generalized forms of bilinear models (see de Falguerolles/Gabriel : Generalized Linear-Bilinear Models). The R-package gnm supports this type of models. The sparse variants of PCA/SVD you are referring to rather relate to L1-regularisations of the factorial representation such that estimated loadings come out as sparse (many zeroes).
A: I had the same problem with a sparce matrix in NLP and what we did was select the columns that where more useful to clasify our rows (that gave more information for discerning the result), if you want I can explain it to you in more detail but it is really simple you can figure it out. 
But your problem does not seem to be a classification one, I actually am a little confused about what you said about above the diagonal and below it. But I was thinking that you can use the Apriori data mining algorithm to discover the more important alliances between any number of items.
A: I suggest that you look at the 2009 paper by Leng and Wang in JCGS:
http://pubs.amstat.org/toc/jcgs/18/1
If this is what you want, the authors supply R code in the supplementary materials.
