Statistics and data mining software tools for dealing with large datasets

Question

Currently I have to analyze approximately 20M records and build prediction models. So far I have tried out Statistica, SPSS, RapidMiner and R. Among these Statistica seems to be most suitable to deal with data mining and RapidMiner user interface is also very handy but it seems that Statistica, RapidMiner and SPSS are suitable just for smaller datasets.

Could anyone recommend a good tool for large datasets?

Thanks you!

Answer 1

I'll second @suncoolsu comment: The dimensionality of your data set is not the only criterion that should orient you toward a specific software. For instance, if you're just planning to do unsupervised clustering or use PCA, there are several dedicated tools that cope with large data sets, as commonly encountered in genomic studies.

Now, R (64 bits) handles large data pretty well, and you still have the option to use disk storage instead of RAM access, but see CRAN Task View High-Performance and Parallel Computing with R. Standard GLM will easily accomodate 20,000 obs. (but see also speedglm) within reasonable time, as shown below:

> require(MASS)
> n <- 20000
> X <- mvrnorm(n, mu=c(0,0), Sigma=matrix(c(1,.8,.8,1), 2, 2))
> df <- cbind.data.frame(X, grp=gl(4, n/4), y=sample(c(0,1), n, rep=TRUE))
> system.time(glm(y ~ ., data=df))
   user  system  elapsed
  0.361   0.018    0.379

To give a more concrete illustration, I used R to process and analyse large genetic data (800 individuals x 800k SNPs, where the main statistical model was a stratified GLM with several covariates (2 min); that was made possible thanks to efficient R and C codes available in the snpMatrix package (in comparison, the same kind of model took about 8 min using a dedicated C++ software (plink). I also worked on a clinical study (12k patients x 50 variables of interest) and R fits my needs too. Finally, as far as I know, the lme4 package is the only software that allow to fit mixed-effects model with unbalanced and large data sets (as is the case in large-scale educational assessment).

Stata/SE is another software that can handle large data set. SAS and SPSS are file based software, so they will handle large volumes of data. A comparative review of software for datamining is available in Data Mining Tools: Which One is Best for CRM. For visualization, there are also plenty of options; maybe a good start is Graphics of large datasets: visualizing a million (reviewed in the JSS by P Murrell), and all related threads on this site.

Answer 2

Most of the algorithms on Apache Mahout scale way beyond 20M records, even with high-dimensional data. If you only need to build a prediction model, there are specific tools like Vowpal Wabbit (http://hunch.net/~vw/) that can easily scale to billions of records on a single machine.

Answer 3

There is the RHIPE package (R-Hadoop integration). It is can make it very easy (with exceptions) to analyze large amounts of data in R.

Answer 4

It is hard to give a good answer without knowing what kind of models you have in mind.

For linear regression, I have successfully used the biglm package in R.

Answer 5

We trained 3.5M observations and 44 features using 64-bit R on an EC2 instance with 32GB ram and 4 cores. We used random forests and it worked well. Note that we had to preprocess/manipulate the data before training.

Answer 6

SAS Enterprise Miner version 6.2 would have no problem handling 20 million observations, and a variety of models which can be adapted to your situation. The issue with SAS is usually the cost however. Here's a summary of what SAS EM can do: SAS EM 6.2: What's New

Answer 7

Since you are building predictive models from large datasets you might benefit from Google's BigQuery (a hosted version of the technology from Google's research paper on massive dataset analysis with Dremel). You can export the query results as CSV for ingestion into a predictive classifier, for example.

BigQuery has a WebUI that allows you to run queries and export results. The beta (v1) version of BigQuery featured a R client, and the production version (v2) will eventually have an R client as well.