# Statistics and data mining software tools for dealing with large datasets

Currently I have to analyze approximately 20M records and build prediction models. So far I have tried out Statistica, SPSS, RapidMiner and R. Among these Statistica seems to be most suitable to deal with data mining and RapidMiner user interface is also very handy but it seems that Statistica, RapidMiner and SPSS are suitable just for smaller datasets.

Could anyone recommend a good tool for large datasets?

Thanks you!

-
Can you pls be a bit more specific? What do you actually want to data mine and how do you plan to do it? I have used R to analyze similar size records as yours, and it wasn't a bad experience at all. –  suncoolsu Oct 14 '10 at 10:31
This should be community wiki. –  Shane Oct 14 '10 at 10:49
I agree with @Shane; this is merely open to equally valid answers, unless it is clearly stated what kind of analyses or questions come with this data set. (Shouldn't have post an answer without further information in fact!) –  chl Oct 14 '10 at 12:20
This question is going to get dated quickly. It might be good to have a semi-annual roundup (wiki) for the latest in such tools. –  Iterator Aug 6 '11 at 0:28

I'll second @suncoolsu comment: The dimensionality of your data set is not the only criterion that should orient you toward a specific software. For instance, if you're just planning to do unsupervised clustering or use PCA, there are several dedicated tools that cope with large data sets, as commonly encountered in genomic studies.

Now, R (64 bits) handles large data pretty well, and you still have the option to use disk storage instead of RAM access, but see CRAN Task View High-Performance and Parallel Computing with R. Standard GLM will easily accomodate 20,000 obs. (but see also speedglm) within reasonable time, as shown below:

> require(MASS)
> n <- 20000
> X <- mvrnorm(n, mu=c(0,0), Sigma=matrix(c(1,.8,.8,1), 2, 2))
> df <- cbind.data.frame(X, grp=gl(4, n/4), y=sample(c(0,1), n, rep=TRUE))
> system.time(glm(y ~ ., data=df))
user  system  elapsed
0.361   0.018    0.379


To give a more concrete illustration, I used R to process and analyse large genetic data (800 individuals x 800k SNPs, where the main statistical model was a stratified GLM with several covariates (2 min); that was made possible thanks to efficient R and C codes available in the snpMatrix package (in comparison, the same kind of model took about 8 min using a dedicated C++ software (plink). I also worked on a clinical study (12k patients x 50 variables of interest) and R fits my needs too. Finally, as far as I know, the lme4 package is the only software that allow to fit mixed-effects model with unbalanced and large data sets (as is the case in large-scale educational assessment).

Stata/SE is another software that can handle large data set. SAS and SPSS are file based software, so they will handle large volumes of data. A comparative review of software for datamining is available in Data Mining Tools: Which One is Best for CRM. For visualization, there are also plenty of options; maybe a good start is Graphics of large datasets: visualizing a million (reviewed in the JSS by P Murrell), and all related threads on this site.

-
@chl: Have you yet found an effective parallel computing solution for 64-bit R? When I last looked (late this summer) the only non-commercial ones appeared to work only in 32-bit R. –  whuber Oct 14 '10 at 15:07
@whuber Nope. I had to switch to 64 bits last year to manage large genetic data sets, but the statistical models we used do not call for parallelization (as far as I know). I thought there was an OpenMP binding for R but did not investigate this further. I know Revolution Analytics have made effort in this sense (j.mp/d7dFb5), but still in 32 bits (this is probably what you referred to). I found R/parallel (rparallel.org) in the meantime, but I don't know how reliable/mature it is. –  chl Oct 14 '10 at 16:47
@chl I tried them all but couldn't get any of them to work. –  whuber Oct 14 '10 at 17:53
@Whuber: are you on windows or a *nix box (mac, linux,...) –  user603 Oct 16 '10 at 9:07
@kwak: Win 7. (Thanks for asking.) –  whuber Oct 16 '10 at 14:15

Most of the algorithms on Apache Mahout scale way beyond 20M records, even with high-dimensional data. If you only need to build a prediction model, there are specific tools like Vowpal Wabbit (http://hunch.net/~vw/) that can easily scale to billions of records on a single machine.

-
Great... I wasn't aware of that! –  chl Oct 16 '10 at 8:47

There is the RHIPE package (R-Hadoop integration). It is can make it very easy (with exceptions) to analyze large amounts of data in R.

-
Do you have success with it? If yes, for what kind of application? –  chl Oct 15 '10 at 5:50
Yes, RHIPE is great. Some of my friends use it to analyze internet traffic data. One of their aims is to model break-in attempts. Data is huge in such cases, petabytes is common! –  suncoolsu Oct 15 '10 at 14:37

It is hard to give a good answer without knowing what kind of models you have in mind.

For linear regression, I have successfully used the biglm package in R.

-

We trained 3.5M observations and 44 features using 64-bit R on an EC2 instance with 32GB ram and 4 cores. We used random forests and it worked well. Note that we had to preprocess/manipulate the data before training.

-