I would like to start learning about doing computations on large data sets. I have read lots of posts about things like map reduce and Hadoop but nothing that shows examples or even clearly explains how these things work. Hadoop to still seems like a magic word people wave around. So suppose I want to do something like basic regression in R but in stead of a dozen variables and a few hundred data points, I want to do it on 800 variables and 1 million data points. How would I attempt such a problem. Can R still be used for something like this? Would I need Amazon's cloud? Are there any tutorials out there that walk you through this type of problem?
I would have a look at the High Performance computing Task View, which suggests
biglm as a means to analyze big datasets that cannot fit in your computers RAM. Alternatively, you can develop your algorithms on a subset of your data, and then perform the real calculations on the entire dataset using an Amazon EC2 instance with 32 GB of RAM.
Hadoop is just a platform for distributed data processing. If you have data that you can't process and/or store locally then Hadoop is a (possible) solution for you. There are several R libraries such as RHadoop,a collection of libraries that allow R users to interact with Hadoop:
- rhdfs, for connectivity to HDFS (Hadoop Distributed File System)
- rmr2, for implementation of the Map-Reduce framework in R
- plyrmr, for manipulation of data stored in Hadoop
- rhbase, for connectivity to Apache Hbase distributed database
Amazon AWS could be used (either alone, or with Hadoop or another distributed processing framework). For example if you can't process your model on the full dataset locally, use subsets of the data to test the run-time, and from these tests you can estimate run time on AWS/EC2 or, if the data is too big then you could try AWS/EMR (Elastic Map Reduce)
@skan mentions random effects or mixed effects models, although I can't see if/where you asked about it. Anyway, mixed models may present some difficulties, because can be computationally intensive and not easy to parallelize, particularly if you have crossed random effects, but there are strategies if this is what you want.