Out-of-Core Data Analysis Options

I have been using SAS professionally for close to 5 years now. I have it installed on my laptop and frequently have to analyze datasets with 1,000-2,000 variables and hundreds of thousands of observations.

I have been looking for alternatives to SAS that allow me to conduct analyses on similar sized data sets. I am curious what other people use for situations such as this. This certainly isn't "Big Data" in the manner that is used today. Nor are my datasets small enough to hold in memory. I need a solution that can apply algorithms to data stored on a hard drive. These are the things I have investigated to no avail:

1. R - BigMemory can create matrices stored out of memory, but the elements have to be the same mode. I work with data that is almost a 50/50 split between character and numeric. The FF package gets closer to what I need, but I don't quite understand which procedures are compatible with it. I think support is somewhat limited.
2. Pandas - I was very excited about a Pythonic alternative to R. However, it too has to hold all of the data in memory.
3. Revolution R - This one shows quite a bit of promise. I have a copy on my home computer (free if you sign up for Kaggle) and have yet to test it as viable alternative to SAS. Comments on Revolution R as a SAS alternative are much appreciated.

Thanks

UPDATE 1

Editing to add that I am looking for real-life, practical solutions that people have used successfully. For the most part, SAS lets me chug through big files without worrying one bit about memory constraints. However SAS is implemented, they figured out how to make memory management transparent to the user. But, it is with a heavy heart that I used SAS for my job (I have to) and would LOVE a FOSS alternative that allows me to work on "large" data without having to think too hard about where the data is located at a specific time (in memory or on disk).

The closest things I have come across are R's FF package and something on the horizon for Python called Blaze. And yet, these problems have existed for many years so what have analysts been doing in the mean time? How are they handling these same issues with Memory limits? The majority of solutions on offer seem to be:

• Get more RAM -- This isn't a good solution, imo. It's easy to find a dataset that can exceed RAM yet still fit on a hard-drive. Furthermore, the work flow has to accommodate all of the structures that are created during exploratory data analysis.
• Subset the data -- This is fine for exploration but not for finalizing results and reporting. Eventually, whatever processes are developed on a subset will have to be applied to the entire dataset (in my case, anyway).
• Chunk through the data -- This is what I would like to know more about from people who actually implement this work-flow. How is it done? With what tools? Can it be done in a way that's transparent to the user? (i.e., create some on-disk data structure and the frame-work takes care of the chunking under the hood).
-
The newer versions of Stata on 64 bit machines have no problem with datasets this size (because 5GB easily fits in RAM these days) but have you any interest in commercial alternatives to SAS or are you looking only at FOSS? – whuber Mar 26 at 1:58
Yes, FOSS solutions are what I'm looking for. I agree that a dataset could fit into 5 gigs of RAM, but it also has to handle of the operations and additional data structures that are created during exploratory data analysis. Couple this with commercial laptops equipped with a meager 4GB of RAM and the overhead consumed by the OS and Memory Errors show up pretty quickly. – Zelazny7 Mar 26 at 2:34
@Zelazny7: Adding another 4Gigs to a laptop should be pretty cheap these days. :) – curious_cat Mar 27 at 5:23
I've found PyTables to be a reasonable solution in the past when working with data which wouldn't fit in memory. – alto Mar 27 at 13:52
Another option is to store the data in a database (SQL or otherwise). Often, an analysis will only require a subset of data (variables 1:10, but not 1:1000) which will fit in memory. If the subset is still larger than memory, the analysis can be done in chunks (load in the data from the database 1000 observations at a time, appropriately collate the results together at the end). – jthetzel Mar 27 at 18:12

Maybe it is not so much about the applications/problems you are aiming for, and its characteristics, but more about the algorithms and variants you are using. More concretely, in order to handle big data, many variants based on stochastic gradient descent of popular algorithms, like SVM, have appear which are able to handle that.

Scikit offers support for some of this algorithms (SVM, kNN, kmeans, ...). I guess this nice graph can help you to quickly figure out, if scikit makes sense to you at all.

Hope that helps

NOTE: what follows is a reply on the comment by zelazny7

Now I understood you. What you are looking for is pandas. Take a look at the talks section. There is a presentation comparing Panda's workflow and panda's briefly. Panda lets you import data in different formats and handle bgu files through HDF5 tables integration. Besides, you can interface Scikit.

-
 Thanks! The algorithms would definitely have to work on-line or on chunks of the data read into memory and written back to disk. Scikit is great and that's actually what I'd LOVE to use, but what tools/work-flows/approaches exist for the very necessary steps of exploration, munging and data prep before applying these algos? These languages can handle these steps, but I'm really looking for an actual example from someone who has to deal with these issues out of memory. – Zelazny7 Mar 27 at 11:53 I edited my reply with what you are looking for (or so I believe!) – juampa Mar 27 at 12:46

You already seem comfortable with SAS, and your datasets are small enough to fit in RAM, but maybe you can't fit enough RAM into your laptop. If you don't mind sticking with SAS, how about you just connect to SAS running remotely on a computer with lots of RAM? I have no idea how that works, but these links might get you started.

There are other great reasons to use Pandas or R, but I don't think you need to worry about memory limits. If you can't fit enough memory on your laptop, run the Python or R instance elsewhere and connect with SSH, iPython Notebook or RStudio.

-

Graphchi is excellent, and can handle huge datasets. It's a bit of a pain to work with, but it can handle graphical and non-graphical data.

-

Have you considered a "Real", non-interpreted language like Fortran?

It seems like the suggestions so far are either very vendor dependent or interpreted. Interpreted methods are notoriously bad at memory intense applications. MatLab may be much higher level of a language than "C" but the memory handling optimizations in C can make it handle 100's of times faster data sets that are millions of times larger.

Both "R" and "Python" are wonderful, high-level, technically rich and highly used languages. They are also interpreted.

You might consider one of the R-on-Hadoop instances. (Rhipe, others) This has the advantage of being able to translate R (high level, easy to program) into MapReduce/Hadoop instructions. Hadoop can make an interesting poor-mans multiprocessing cluster.