Out-of-core data analysis options

Question

I have been using SAS professionally for close to 5 years now. I have it installed on my laptop and frequently have to analyze datasets with 1,000-2,000 variables and hundreds of thousands of observations.

I have been looking for alternatives to SAS that allow me to conduct analyses on similar sized data sets. I am curious what other people use for situations such as this. This certainly isn't "Big Data" in the manner that is used today. Nor are my datasets small enough to hold in memory. I need a solution that can apply algorithms to data stored on a hard drive. These are the things I have investigated to no avail:

1. R - BigMemory can create matrices stored out of memory, but the elements have to be the same mode. I work with data that is almost a 50/50 split between character and numeric. The FF package gets closer to what I need, but I don't quite understand which procedures are compatible with it. I think support is somewhat limited.
2. Pandas - I was very excited about a Pythonic alternative to R. However, it too has to hold all of the data in memory.
3. Revolution R - This one shows quite a bit of promise. I have a copy on my home computer (free if you sign up for Kaggle) and have yet to test it as viable alternative to SAS. Comments on Revolution R as a SAS alternative are much appreciated.

Thanks

UPDATE 1

Editing to add that I am looking for real-life, practical solutions that people have used successfully. For the most part, SAS lets me chug through big files without worrying one bit about memory constraints. However SAS is implemented, they figured out how to make memory management transparent to the user. But, it is with a heavy heart that I used SAS for my job (I have to) and would LOVE a FOSS alternative that allows me to work on "large" data without having to think too hard about where the data is located at a specific time (in memory or on disk).

The closest things I have come across are R's FF package and something on the horizon for Python called Blaze. And yet, these problems have existed for many years so what have analysts been doing in the mean time? How are they handling these same issues with Memory limits? The majority of solutions on offer seem to be:

• Get more RAM -- This isn't a good solution, imo. It's easy to find a dataset that can exceed RAM yet still fit on a hard-drive. Furthermore, the work flow has to accommodate all of the structures that are created during exploratory data analysis.
• Subset the data -- This is fine for exploration but not for finalizing results and reporting. Eventually, whatever processes are developed on a subset will have to be applied to the entire dataset (in my case, anyway).
• Chunk through the data -- This is what I would like to know more about from people who actually implement this work-flow. How is it done? With what tools? Can it be done in a way that's transparent to the user? (i.e., create some on-disk data structure and the frame-work takes care of the chunking under the hood).

Answer 1

if you're maxing out at 500,000 records x 2,000 variables, i would spend a little more money on RAM for your laptop and be done with it. if you have 16GB, you can probably read the data set you're describing into R directly. and at that point, you'll be able to do far more - and very quickly.. but you say that's not an option, so:

look at SQL-based packages for R. these allow you to connect to external databases and access those tables via SQL. since SQL is pretty universal (and since R is open-source), your code won't be lost if you change jobs or lose access to SAS. the easiest external database to set up is RSQLite but by far the fastest is MonetDB.R (speed tests)

there are probably a few good solutions to your stated problem, my guess is that just about all of them involve R ;)

Answer 2

Maybe it is not so much about the applications/problems you are aiming for, and its characteristics, but more about the algorithms and variants you are using. More concretely, in order to handle big data, many variants based on stochastic gradient descent of popular algorithms, like SVM, have appear which are able to handle that.

Scikit offers support for some of this algorithms (SVM, kNN, kmeans, ...). I guess this nice graph can help you to quickly figure out, if scikit makes sense to you at all.

Hope that helps

NOTE: what follows is a reply on the comment by zelazny7

Now I understood you. What you are looking for is pandas. Take a look at the talks section. There is a presentation comparing Panda's workflow and panda's briefly. Panda lets you import data in different formats and handle bgu files through HDF5 tables integration. Besides, you can interface Scikit.

Answer 3

You already seem comfortable with SAS, and your datasets are small enough to fit in RAM, but maybe you can't fit enough RAM into your laptop. If you don't mind sticking with SAS, how about you just connect to SAS running remotely on a computer with lots of RAM? I have no idea how that works, but these links might get you started.

• http://support.sas.com/documentation/cdl/en/camref/61896/HTML/default/viewer.htm#a000382050.htm
• http://support.sas.com/documentation/cdl/en/hostunx/61879/HTML/default/viewer.htm#sasrem.htm

There are other great reasons to use Pandas or R, but I don't think you need to worry about memory limits. If you can't fit enough memory on your laptop, run the Python or R instance elsewhere and connect with SSH, iPython Notebook or RStudio.

Answer 4

Graphchi is excellent, and can handle huge datasets. It's a bit of a pain to work with, but it can handle graphical and non-graphical data.

Answer 5

I recently came across SFrames and GraphLab Create. These are libraries for Python that offer the kind of functionality you seem to be looking for From the Pypi site: "SFrame is an scalable, out-of-core dataframe, which allows you to work with datasets that are larger than the amount of RAM on your system." So think of it as the data manipulation functionality and API in Pandas but without puttin all the data in memory first. SFrame is free and open source as far as I know. On the other hand GraphLab builds on the SFrame functionality to provide algorithms for doing descriptive and predictive (machine learning) analytics on data that is stored in SFrames. GraphLab Create create is not free / open source, but has a free demo license. In any case, depending on how sophisticated your algorithms need to be, SFrame might suffice for you. Or if you require for instance a quick and dirty linear / logistic / ridge regression on your data, you can implement your self it on the SFrame with some simple SGD method, for example, and avoid having to pay for Graphlab Create license.

Answer 6

Have you considered a "Real", non-interpreted language like Fortran?

It seems like the suggestions so far are either very vendor dependent or interpreted. Interpreted methods are notoriously bad at memory intense applications. MatLab may be much higher level of a language than "C" but the memory handling optimizations in C can make it handle 100's of times faster data sets that are millions of times larger.

Both "R" and "Python" are wonderful, high-level, technically rich and highly used languages. They are also interpreted.

You might consider one of the R-on-Hadoop instances. (Rhipe, others) This has the advantage of being able to translate R (high level, easy to program) into MapReduce/Hadoop instructions. Hadoop can make an interesting poor-mans multiprocessing cluster.

http://www.datadr.org/ <-- (Rhipe link)

Fortran has been being developed for decades. It has very efficient memory handling, and compiling. It also has some higher level libraries so it can do very technically sophisticated operations pretty simply. I might do a toy CFD in MatLab, but for something realistic and self-coded, I would use Fortran for the "big-iron" processing and something like MatLab or R for presenting/summarizing data. Nobody makes commercial CFD software whose "engine" is interpreted instead of compiled. Several vendors have their moneymaker CFD coded in C or Fortran. SAS was originally written in C (link).

Fortran and Rhipe are accessible. MatLab costs money and if my job didn't pay for it then I would be using R or Python right now.

UPDATE:
My point was "compiled". Python has cython that can literally run the same (fundamental) code ~1000x faster. That means you can have data that is ~1000x larger and process it in nearly the same time. Getting Cython to work clean can be a challenge, but tools like "Sage" wrap it well. Rcpp allegedly has similar, but I don't personally know that it is as well developed.

If you think about it, the fundamentals of nearly everything you run in scientific computing, even in interpreted languages is compiled Fortran or compiled C. BLAS is BLAS; you aren't re-inventing the code every time you do EDA. Your interpreted language is calling those libraries, albeit very inefficiently, when it runs.

As an aside, you might look at JMP. It has very easy to use interface and is excellent for very visual exploratory data analysis (EDA).