I need to run some statistical hypothesis testing, Anova, student's, least square fit, median, data mining, clustering... on a very large quantity of distributed data. (>100TB, Maybe columnar or key/value, mostly numerical, mostly reading ).

I wouldn't like to write all that functions again but I'd prefer to use something already implemented (out-of-the-box) on a "big data" database or on a software able to connect to that database and use its functions or packages on it.

What DB or software would you suggest? Anything able to do it?. Maybe SAS?

There are many interesting DBs, such as SicDB, Hyperdex, Aerospike, Couchbase, (I could even consider Redis, Postgres+analytic addon or SQL Server Analysis Services for smaller data sizes). (Or maybe Cassandra or Hadoop but I prefer to avoid Java based solutions).

The problem is that most of them implement only a small quantity of simple analysis or functions. If you need something more advanced you have to write it yourself (if possible, spend a long time building them from basic functions) or try to connect it to a statistical software (such as R, SPSS,...) to do it, but most functions on that software maybe don't accept that source of data. Then you would need to break it on smaller chunks and process it..., but that's much more difficult and prone to errors and not always possible.

For example, I've found that you can connect R to SciDB but you can only use the functions provided by SciDB, and they don't include data mining or Anova. You can convert some subsets of data to R but if you try to to it with a large column you'll end up with an out of memory error.

Imagine I want to sort the 200TB database, calculate the median value of the 1st column and do a linear least squares regression. How would you do it?


1 Answer 1


For those data set sizes you probably want to check out distributed platforms like Hadoop or Spark. Redis and co are not suitable for such tasks.

Spark would probably be easiest. Its Python interface makes implementing routines like ANOVA based on RDD's fairly painless. Linear regression is already available in Spark's MLlib.

The powered by page of Spark shows that it is being adopted by some internet giants like Amazon, eBay, Baidu, Tencent and Yahoo. I consider that to be a pretty good indication of its performance. Hadoop also boasts impressive lists of companies powered by it.

In terms of comparisons, the typical one is Spark vs Hadoop. Spark is essentially much more modern, so it's not surprising that it wins in terms of performance for most (all?) benchmarks. Some common points are raised in this SO post: https://stackoverflow.com/questions/25267204/hadoop-vs-spark

A pretty good presentation about the nitty gritty details of Spark can be found here.

  • 1
    $\begingroup$ Good advice (+1), albeit a bit on the brief side. I'm curious about documented examples of production level big (hundreds of TBs) data analysis setups, based on Hadoop, Spark as well as other frameworks (preferably, open source). Any references? Benchmark and comparison studies are also of interest. $\endgroup$ Commented Mar 1, 2015 at 10:50
  • 1
    $\begingroup$ @AleksandrBlekh I've updated my answer. You can find documented examples in many recent presentations at Spark/Hadoop summits. $\endgroup$ Commented Mar 1, 2015 at 13:12
  • $\begingroup$ And if I want to start with a smaller quantity of data, let's say 10TB. What DB would you recomend? I prefer something not based on Java. $\endgroup$
    – skan
    Commented Mar 1, 2015 at 16:53
  • $\begingroup$ Did anybody tried SciDB?. Did you compared Spark to Couchbase or other DBs? $\endgroup$
    – skan
    Commented Mar 1, 2015 at 17:02
  • $\begingroup$ Marc, thank you for the update. I'll take a look at the resources you've recommended. $\endgroup$ Commented Mar 1, 2015 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.