27
$\begingroup$

Currently I have to analyze approximately 20M records and build prediction models. So far I have tried out Statistica, SPSS, RapidMiner and R. Among these Statistica seems to be most suitable to deal with data mining and RapidMiner user interface is also very handy but it seems that Statistica, RapidMiner and SPSS are suitable just for smaller datasets.

Could anyone recommend a good tool for large datasets?

Thanks you!

$\endgroup$
  • 6
    $\begingroup$ Can you pls be a bit more specific? What do you actually want to data mine and how do you plan to do it? I have used R to analyze similar size records as yours, and it wasn't a bad experience at all. $\endgroup$ – suncoolsu Oct 14 '10 at 10:31
  • $\begingroup$ This question is going to get dated quickly. It might be good to have a semi-annual roundup (wiki) for the latest in such tools. $\endgroup$ – Iterator Aug 6 '11 at 0:28
19
$\begingroup$

I'll second @suncoolsu comment: The dimensionality of your data set is not the only criterion that should orient you toward a specific software. For instance, if you're just planning to do unsupervised clustering or use PCA, there are several dedicated tools that cope with large data sets, as commonly encountered in genomic studies.

Now, R (64 bits) handles large data pretty well, and you still have the option to use disk storage instead of RAM access, but see CRAN Task View High-Performance and Parallel Computing with R. Standard GLM will easily accomodate 20,000 obs. (but see also speedglm) within reasonable time, as shown below:

> require(MASS)
> n <- 20000
> X <- mvrnorm(n, mu=c(0,0), Sigma=matrix(c(1,.8,.8,1), 2, 2))
> df <- cbind.data.frame(X, grp=gl(4, n/4), y=sample(c(0,1), n, rep=TRUE))
> system.time(glm(y ~ ., data=df))
   user  system  elapsed
  0.361   0.018    0.379

To give a more concrete illustration, I used R to process and analyse large genetic data (800 individuals x 800k SNPs, where the main statistical model was a stratified GLM with several covariates (2 min); that was made possible thanks to efficient R and C codes available in the snpMatrix package (in comparison, the same kind of model took about 8 min using a dedicated C++ software (plink). I also worked on a clinical study (12k patients x 50 variables of interest) and R fits my needs too. Finally, as far as I know, the lme4 package is the only software that allow to fit mixed-effects model with unbalanced and large data sets (as is the case in large-scale educational assessment).

Stata/SE is another software that can handle large data set. SAS and SPSS are file based software, so they will handle large volumes of data. A comparative review of software for datamining is available in Data Mining Tools: Which One is Best for CRM. For visualization, there are also plenty of options; maybe a good start is Graphics of large datasets: visualizing a million (reviewed in the JSS by P Murrell), and all related threads on this site.

$\endgroup$
  • $\begingroup$ @chl: Have you yet found an effective parallel computing solution for 64-bit R? When I last looked (late this summer) the only non-commercial ones appeared to work only in 32-bit R. $\endgroup$ – whuber Oct 14 '10 at 15:07
  • 1
    $\begingroup$ @whuber Nope. I had to switch to 64 bits last year to manage large genetic data sets, but the statistical models we used do not call for parallelization (as far as I know). I thought there was an OpenMP binding for R but did not investigate this further. I know Revolution Analytics have made effort in this sense (j.mp/d7dFb5), but still in 32 bits (this is probably what you referred to). I found R/parallel (rparallel.org) in the meantime, but I don't know how reliable/mature it is. $\endgroup$ – chl Oct 14 '10 at 16:47
  • $\begingroup$ @chl I tried them all but couldn't get any of them to work. $\endgroup$ – whuber Oct 14 '10 at 17:53
  • $\begingroup$ @Whuber: are you on windows or a *nix box (mac, linux,...) $\endgroup$ – user603 Oct 16 '10 at 9:07
  • 2
    $\begingroup$ I would advise you to install ubuntu (google 'download ubuntu') and to run your windows-only apps from within ubuntu via virtualbox (youtube.com/watch?v=KXgKnd-u2R4). R and latex editors run on ubuntu like a charm. $\endgroup$ – user603 Oct 17 '10 at 10:41
8
$\begingroup$

Most of the algorithms on Apache Mahout scale way beyond 20M records, even with high-dimensional data. If you only need to build a prediction model, there are specific tools like Vowpal Wabbit (http://hunch.net/~vw/) that can easily scale to billions of records on a single machine.

$\endgroup$
  • $\begingroup$ Great... I wasn't aware of that! $\endgroup$ – chl Oct 16 '10 at 8:47
7
$\begingroup$

There is the RHIPE package (R-Hadoop integration). It is can make it very easy (with exceptions) to analyze large amounts of data in R.

$\endgroup$
  • $\begingroup$ Do you have success with it? If yes, for what kind of application? $\endgroup$ – chl Oct 15 '10 at 5:50
  • $\begingroup$ Yes, RHIPE is great. Some of my friends use it to analyze internet traffic data. One of their aims is to model break-in attempts. Data is huge in such cases, petabytes is common! $\endgroup$ – suncoolsu Oct 15 '10 at 14:37
4
$\begingroup$

It is hard to give a good answer without knowing what kind of models you have in mind.

For linear regression, I have successfully used the biglm package in R.

$\endgroup$
4
$\begingroup$

Since you are building predictive models from large datasets you might benefit from Google's BigQuery (a hosted version of the technology from Google's research paper on massive dataset analysis with Dremel). You can export the query results as CSV for ingestion into a predictive classifier, for example.

BigQuery has a WebUI that allows you to run queries and export results. The beta (v1) version of BigQuery featured a R client, and the production version (v2) will eventually have an R client as well.

$\endgroup$
3
$\begingroup$

We trained 3.5M observations and 44 features using 64-bit R on an EC2 instance with 32GB ram and 4 cores. We used random forests and it worked well. Note that we had to preprocess/manipulate the data before training.

$\endgroup$
3
$\begingroup$

SAS Enterprise Miner version 6.2 would have no problem handling 20 million observations, and a variety of models which can be adapted to your situation. The issue with SAS is usually the cost however. Here's a summary of what SAS EM can do: SAS EM 6.2: What's New

$\endgroup$
0
$\begingroup$

Can you look at ScaVis (http://jwork.org/scavis)? I did not look at 20M, but you may try to check it.

$\endgroup$
0
$\begingroup$

RHIPE is a great solution, and I would probably choose this one, if having this issue! but have you considered NCSS? As far as I know, the newest version 10 can build these models. The full ver. is very expensive, but on several remote desktop services you can run the app only for a small fee but I dunno.. rather check that out

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.