I am going to be using R for text analysis (mostly clustering, classification and some visualization) and was wondering what mechanisms R provides for handling high dimensional, sparse data sets. If I understand correctly, R does provide some packages (e.g., matrix library) for handling large and sparse matrices - which brings me to my question.

Specifically, I would like to know:

  1. Which R libraries are most appropriate for storing and processing high dimensional sparse data? Just FYI, my data will fit into memory.

  2. Do such libraries inter-operate with existing text analysis (clustering/classification) packages? Would I need to convert these sparse data structures to and from data frames if I need to text analysis? Wouldn't that add additional time overhead to the computations?

I am fairly new to R, so please excuse me if this sounds vague (or too general).

  • $\begingroup$ Any thoughts on this one? I will appreciate any pointers. Thanks. $\endgroup$ – Andy Nov 20 '11 at 1:54

This does not help you? http://tiny.cc/ns73n Moreover, you can install sos R package to search any function in R, by using findFn() function in the sos package.

  • $\begingroup$ Well the problem is that tm library does not scale well. RWeka provides access to only select WEKA functionality (no way to call WEKA feature selection methods, for example). I don't see any other packages that would let you do pre-processing on text data in a scalable fashion. And then there's always this question of how to store and process highly sparse data... $\endgroup$ – Andy Nov 23 '11 at 21:53

A quick google for R text analysis lead me to the following article in the Journal for Statistical Software from 2008:

Text Mining Infrastructure in R

I'm not sure what is in it exactly, and it might be a bit outdated, but it should be helpful...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.