7
$\begingroup$

The data files from http://www.csie.ntu.edu.tw/~cjlin/libsvm/ are in 'svm' format. I am trying to read this in to sparse matrix representation in R. Is there an easy/efficient way to do this?

Here is what I am doing now: read in file line by line (800,000 lines), for each line separate classes, values, and cols. Store the classes as a list and the features as a .csr sparse matrix (1 row), then rbind the feature row with all previous rows.

This is terribly inefficient and basically won't finish (12 minutes for 1000 lines). I think it comes from rbinding the sparse matrices once the number of rows starts to get large.

Note: the matrix (800000*48000) is too big to build and then convert to sparse format.

Thanks!

$\endgroup$
4
$\begingroup$

The e1071 package has a means for exporting to the libsvm "svm" format in the write.svm function. But to the best of my knowledge, there is no read.svm function.

$\endgroup$
  • 1
    $\begingroup$ Yes I found it interesting that there was a write function but no read function. What would be ideal is if there was a read function and then you could write it as a sparse matrix. $\endgroup$ – Glen Jan 31 '11 at 23:01
4
$\begingroup$

I found a way that is at least now feasible.

Instead of the sparseM package I use the Matrix package to build the sparse matrices. Store the entries and columns in separate lists and then build the matrix by:

data=sparseMatrix(i=rep(1,length[[1]]),j=columns[[loop]],
    x=entires[[loop]],dims=c(120000,47235))
for(loop in 2:120000){
  if(loop %% 1000==0){
      print(loop)
      print(Sys.time())
  }
  data[loop,columns[[loop]]]=entries[[loop]]  
}

This still takes awhile (about 2 hours) but at least it works.

I store the corresponding classes in another list not in the sparse matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.