The stat packages which I've been using are not very well suited for working with large data sets, i.e. more than 1 billion records. They all tend to try loading all into memory, and install one one machine.

Is there a statistical package that's comparable in uesability to Stata or MATLAB that can distribute work across multiple nodes (machines) and run regressions (panels, longitudinal) on large data sets? The idea's that it's got to be an interactive environment, but able to distribute processing when necessary.

  • 1
    $\begingroup$ My understanding is that statistical analysis on data sets that are too large to load into memory is one of the distinguishing features of SAS. $\endgroup$ – Matthew Gunn Jun 17 '16 at 19:27

Spark will do exactly what you want but I think you can make do with simpler tools. Most general scripting languages, such as Python, have the tools to deal with mid to large size data.

Also don't discount Matlab. It has numerous awesome big data tools such as memory mapping and Hadoop support. Check out their site on large data.

  • 1
    $\begingroup$ Do you use any of this? They're all good on paper, I wonder what do people actually use $\endgroup$ – Aksakal Jun 17 '16 at 18:51
  • $\begingroup$ I haven't used Matlab's tools for largish data but my colleagues regularly utilize it for out of RAM tasks. I have used both Spark(read Python's Spark wrapper) and Python is my regular tool. But your main question is inherently difficult. From a scripting optimization to more efficient data storage to cloud computing's costs (if you're not part of Google or Amazon), the solutions are manifold and often specific to the question/resources at hand. I highly encourage you to learn Python or Julia if largish data if it's is a recurrent issue. Even R will work in cases. $\endgroup$ – Gene Burinsky Jun 17 '16 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.