8
$\begingroup$

I know that R is not particularly helpful for analysing large datasets given that R loads all the data in memory whereas something like SAS does sequential analysis. That said, there are packages like bigmemory that allows users to perform large data analysis (statistical analysis) more efficiently in R.

I wanted to know, apart from all the theoretical information, has anyone used / is using R for analysing large datasets in an enterprise environment and what are the typical issues that could arise. By large datasets I am referring to datasets that are ~ 200 GB in size. Also, any thoughts on real-life examples of migrating from SAS to R in such use cases would be helpful.

$\endgroup$
  • 1
    $\begingroup$ R 3.0 (which is the current release) can access large arrays. On a 64 bit machine should be able to address that much memory if you have that much. $\endgroup$ – Glen_b -Reinstate Monica Apr 4 '13 at 2:30
  • $\begingroup$ There are some packages to increase memory, such as ff and bigmemory. @Glen_b, do you think the new R (with a 64 bit machine) will rivalize with SAS (in terms of possible datasets size) ? $\endgroup$ – Stéphane Laurent Apr 4 '13 at 7:48
  • $\begingroup$ @StéphaneLaurent SAS uses a different approach which means its not limited to addressable memory (this approach isn't really possible for R because of its flexibility), so the limits won't be the same ... but I don't actually know what SAS' limits are. $\endgroup$ – Glen_b -Reinstate Monica Apr 4 '13 at 9:00
4
$\begingroup$

I have done work on very large data sets in R, and not had problems.

There are several approaches that work, but my basic paradigm is that I find ways to process the data "sequentially". Obviously SAS has the same fundamental memory constraints if you're using it on the same machine, using R is just a little more DIY.

In every case that I have ever encountered I'm either doing analysis on some kind of summary of the data, or I'm doing analysis on chunks of the data and then summarizing the results. Either way that's easy to accomplish in R.

It's pretty easy to create summaries if you have your data structured in some way (really in any way). Hadoop is a leading tool for creating summaries, but it's easy to do batch processing on R Data files, and if your data will fit on your local storage device, it's also faster to batch process it that way (in terms of both processing time and development time).

It's also pretty easy to batch your analysis by chunk as well using the same thought process.

If you're really dying to do a linear model directly on a gigantic data set, then I think bigmemory is your answer, as suggested by Stéphane Laurent.

I don't really think there is one "answer" to "how do you deal with memory constraints" or "move to a new platform", but this is my long winded two cents.

$\endgroup$
3
$\begingroup$

I do not have hands-on on the revolutionary analytics part but there is a blog on this

http://www.r-bloggers.com/allstate-compares-sas-hadoop-and-r-for-big-data-insurance-models/

It uses hadoop (distributed computing) to solve this problem of memory.

$\endgroup$
  • 1
    $\begingroup$ Please provide more information. If the link doesn't work anymore, the answer should still be helpful. $\endgroup$ – Sven Hohenstein Apr 4 '13 at 8:00
  • $\begingroup$ I don't think their "solution" was Hadoop (80 Hadoop cores was twice as slow as 16-core SAS), but it sounds like LSF (en.wikipedia.org/wiki/Platform_LSF) is roughly the same kind of thing. $\endgroup$ – Darren Cook Apr 5 '13 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.