I need to run some statistical hypothesis testing, Anova, student's, least square fit, median, data mining, clustering... on a very large quantity of distributed data. (>100TB, Maybe columnar or key/value, mostly numerical, mostly reading ).
I wouldn't like to write all that functions again but I'd prefer to use something already implemented (out-of-the-box) on a "big data" database or on a software able to connect to that database and use its functions or packages on it.
What DB or software would you suggest? Anything able to do it?. Maybe SAS?
There are many interesting DBs, such as SicDB, Hyperdex, Aerospike, Couchbase, (I could even consider Redis, Postgres+analytic addon or SQL Server Analysis Services for smaller data sizes). (Or maybe Cassandra or Hadoop but I prefer to avoid Java based solutions).
The problem is that most of them implement only a small quantity of simple analysis or functions. If you need something more advanced you have to write it yourself (if possible, spend a long time building them from basic functions) or try to connect it to a statistical software (such as R, SPSS,...) to do it, but most functions on that software maybe don't accept that source of data. Then you would need to break it on smaller chunks and process it..., but that's much more difficult and prone to errors and not always possible.
For example, I've found that you can connect R to SciDB but you can only use the functions provided by SciDB, and they don't include data mining or Anova. You can convert some subsets of data to R but if you try to to it with a large column you'll end up with an out of memory error.
Imagine I want to sort the 200TB database, calculate the median value of the 1st column and do a linear least squares regression. How would you do it?
