Out-of-Core Data Analysis Options

Question

I have been using SAS professionally for close to 5 years now. I have it installed on my laptop and frequently have to analyze datasets with 1,000-2,000 variables and hundreds of thousands of observations.

I have been looking for alternatives to SAS that allow me to conduct analyses on similar sized data sets. I am curious what other people use for situations such as this. This certainly isn't "Big Data" in the manner that is used today. Nor are my datasets small enough to hold in memory. I need a solution that can apply algorithms to data stored on a hard drive. These are the things I have investigated to no avail:

1. R - BigMemory can create matrices stored out of memory, but the elements have to be the same mode. I work with data that is almost a 50/50 split between character and numeric. The FF package gets closer to what I need, but I don't quite understand which procedures are compatible with it. I think support is somewhat limited.
2. Pandas - I was very excited about a Pythonic alternative to R. However, it too has to hold all of the data in memory.
3. Revolution R - This one shows quite a bit of promise. I have a copy on my home computer (free if you sign up for Kaggle) and have yet to test it as viable alternative to SAS. Comments on Revolution R as a SAS alternative are much appreciated.

Thanks

UPDATE 1

Editing to add that I am looking for real-life, practical solutions that people have used successfully. For the most part, SAS lets me chug through big files without worrying one bit about memory constraints. However SAS is implemented, they figured out how to make memory management transparent to the user. But, it is with a heavy heart that I used SAS for my job (I have to) and would LOVE a FOSS alternative that allows me to work on "large" data without having to think too hard about where the data is located at a specific time (in memory or on disk).

The closest things I have come across are R's FF package and something on the horizon for Python called Blaze. And yet, these problems have existed for many years so what have analysts been doing in the mean time? How are they handling these same issues with Memory limits? The majority of solutions on offer seem to be:

• Get more RAM -- This isn't a good solution, imo. It's easy to find a dataset that can exceed RAM yet still fit on a hard-drive. Furthermore, the work flow has to accommodate all of the structures that are created during exploratory data analysis.
• Subset the data -- This is fine for exploration but not for finalizing results and reporting. Eventually, whatever processes are developed on a subset will have to be applied to the entire dataset (in my case, anyway).
• Chunk through the data -- This is what I would like to know more about from people who actually implement this work-flow. How is it done? With what tools? Can it be done in a way that's transparent to the user? (i.e., create some on-disk data structure and the frame-work takes care of the chunking under the hood).

Answer 1

Maybe it is not so much about the applications/problems you are aiming for, and its characteristics, but more about the algorithms and variants you are using. More concretely, in order to handle big data, many variants based on stochastic gradient descent of popular algorithms, like SVM, have appear which are able to handle that.

Scikit offers support for some of this algorithms (SVM, kNN, kmeans, ...). I guess this nice graph can help you to quickly figure out, if scikit makes sense to you at all.

Hope that helps

NOTE: what follows is a reply on the comment by zelazny7

Now I understood you. What you are looking for is pandas. Take a look at the talks section. There is a presentation comparing Panda's workflow and panda's briefly. Panda lets you import data in different formats and handle bgu files through HDF5 tables integration. Besides, you can interface Scikit.

Answer 2

You already seem comfortable with SAS, and your datasets are small enough to fit in RAM, but maybe you can't fit enough RAM into your laptop. If you don't mind sticking with SAS, how about you just connect to SAS running remotely on a computer with lots of RAM? I have no idea how that works, but these links might get you started.

• http://support.sas.com/documentation/cdl/en/camref/61896/HTML/default/viewer.htm#a000382050.htm
• http://support.sas.com/documentation/cdl/en/hostunx/61879/HTML/default/viewer.htm#sasrem.htm

There are other great reasons to use Pandas or R, but I don't think you need to worry about memory limits. If you can't fit enough memory on your laptop, run the Python or R instance elsewhere and connect with SSH, iPython Notebook or RStudio.

Answer 3

Graphchi is excellent, and can handle huge datasets. It's a bit of a pain to work with, but it can handle graphical and non-graphical data.

Answer 4

Have you considered a "Real", non-interpreted language like Fortran?

It seems like the suggestions so far are either very vendor dependent or interpreted. Interpreted methods are notoriously bad at memory intense applications. MatLab may be much higher level of a language than "C" but the memory handling optimizations in C can make it handle 100's of times faster data sets that are millions of times larger.

Both "R" and "Python" are wonderful, high-level, technically rich and highly used languages. They are also interpreted.

You might consider one of the R-on-Hadoop instances. (Rhipe, others) This has the advantage of being able to translate R (high level, easy to program) into MapReduce/Hadoop instructions. Hadoop can make an interesting poor-mans multiprocessing cluster.

http://www.datadr.org/ <- (Rhipe link)

Fortran has been being developed for decades. It has very efficient memory handling, and compiling. It also has some higher level libraries so it can do very technially sophisticated operations pretty simply. I might to a toybox CFD in MatLab, but for something realistic and self-coded, I would use Fortran for the "big-iron" processing and soemthing like MatLab or R for presenting/summarizing data.

Fortran and Rhipe are accessible. MatLab costs money and if my job didn't pay for it then I would be using R or Python right now.