1
$\begingroup$

I am trying to calculate the pseudoinverse of a large sparse matrix in R using the singular value decomposition. The matrix is roughly 240,000 x 240,000, and I have it stored as type dgCMatrix. I have tried using pinv, ginv, and other standard pseudoinverse functions, but they error out due to memory constraints. I then tried to opt for the sparse matrix svd provided by the package irlba, which I was then going to use to compute the pseudoinverse using the standard formula after converting all outputs to sparse matrices. My code is here:

lim = 40
digits = 4
SVD =irlba(L,lim)
tU = round(SVD$u,digits)
nonZeroU = which(abs(U)>0,arr.ind = T)
sparseU = sparseMatrix(i=nonZeroU[,2],j=nonZeroU[,1],x = U[nonZeroU])
V = round(SVD$v,digits)
nonZeroV = which(abs(V)>0,arr.ind = T)
sparseV = sparseMatrix(i=nonZeroV[,1],j=nonZeroV[,2],x = U[nonZeroV])
D = as(Diagonal(x=1/SVD$d),"sparseMatrx")
pL =D%*%sparseU
pL = sparseV%*%pL

I am able to get to the last line without an issue, but then I get an error due to memory constraints that says

Error in sparseV %*% pL : 
  Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 105

Of course I could piece together the pseudoinverse entry by entry using a for loop and vector multiplications, but I would like to be able to calculate it using a simple function that takes advantage of the sparsity of the resultant pseudoinverse matrix. Due to the nature of the original matrix (it is a graph laplacian), I know that the pseudoinverse should also be a sparse matrix. Any help would be greatly appreciated!

$\endgroup$

closed as off-topic by Michael Chernick, Stephan Kolassa, kjetil b halvorsen, John, Peter Flom Apr 17 '17 at 13:09

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Michael Chernick, Stephan Kolassa, John, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ 1. What is the type of the pL matrix (and its dimensions)? 2. How much RAM do you have available generally? 3. How sparse are we talking? <1%? <5%? 5-10%? 10%+ (so not really). 4. I hope you notice that you are doing a dense calculation so I somewhat suspect that you try to evaluate a 240k-by-240k dense matrix as a final result. Are you sure that diag(1/SVD$d) is a sparse matrix? 5. Maybe you want to try a smaller problem first, say that all variable types are as expected and then try to throw in and the kitchen-sink. (+1 fun problem) $\endgroup$ – usεr11852 Apr 16 '17 at 20:33
  • $\begingroup$ Seems like a use-case for solving the problem with power iteration + Gram-Schmidt, both of which can be done to use sparse matrix multiplication. (+1) $\endgroup$ – Sycorax Apr 16 '17 at 20:50
  • $\begingroup$ @usεr11852 I am working with 16 gb of ram, and the type of pL is inherited from the type of the type of sparseU, which is dgCMatrix. The matrix L has only 400,000 entries, so it is extremely sparse-- <1%. D has dimensions 40x40 because I only took the first 40 singular values. The reason I thought it should be doable is that if the svd can be taken, I would have thought this should be roughly the same complexity problem... $\endgroup$ – Paul Apr 16 '17 at 21:06
  • $\begingroup$ ( By matrix LI guess you mean pL right? ) D doesn't seem to come into play here. Are you sure that pL is not dgeMatrix? $\endgroup$ – usεr11852 Apr 16 '17 at 21:10
  • $\begingroup$ Try ensuring that diag(1/SVD$d) is a ddiMatrix before multiplying with sparseU. $\endgroup$ – usεr11852 Apr 16 '17 at 21:17

Browse other questions tagged or ask your own question.