Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial variables. Let's say there is one regression problem, one unbalanced binary classification problem, and one task of "find out which predictors are most important." My thought for how to approach the problem is:

Fit some predictive model on progressively larger and larger (random) sub-samples of the data until:

  1. Fitting and cross-validating the model becomes computationally difficult (e.g., unreasonbly slow on my laptop, R runs out of memory, etc.), OR

  2. The training and test RMSE or precision/recall values stabilize.

If the training and test errors did not stabilize (1.), use a simpler model and/or implement multicore or multinode versions of the model and restart from the beginning.

If the training and test errors stabilized (2.):

  • If $N_{subset} \ll N$ (i.e., I can still run algorithms on $X_{subset}$ as it's not too large yet), try to improve performance by expanding the feature space or using a more complex model and restarting from the beginning.

  • If $N_{subset}$ is 'large' and running further analyses is costly, analyze variable importance and end.

I plan to use packages like biglm, speedglm, multicore, and ff in R initially, and later use more complicated algorithms and/or multinode (on EC2) as necessary.

Does this sound like a reasonable approach, and if so, do you have any specific advice or suggestions? If not, what would you try instead for a data set of this size?


3 Answers 3


You should check out online methods for regression and classification for datasets of this size. These approaches would let you use the whole dataset without having to load it into memory.

You might also check out Vowpal Wabbit (VW):


It uses an out of core online method, so it should be able to handle a dataset of this size. You can do regression and classification and it has support for sparse formats. You can also do penalized versions (e.g. lasso-type regression/classification) in VW, which could improve your model's accuracy.


I would suggest using Hadoop and RMR (a specific package for Map Reduce in R). With this strategy you can run large datasets on comodity computers with an affordable configuration (probably in two hours you come up with both Hadoop and RMR (RHadoop) installed and running).

In fact, if you have more than one computer you can create a cluster, reducing the processing time.

I give you some links supporting my suggestion:

  • This link will lead you to a tutorial for installing Hadoop on a single-node cluster (one computer).
  • This link and this link will show you how to install RMR on your Hadoop cluster.
  • And finally, here you may find an example of logistic regression by means of RHadoop.

So, my advice is to follow these guidelines as it is certainly worthy if your data is huge.


This is more a comment than an answer, but I can't post it as a comment (requires 50 rep)..

Have you tried to use PCA on your dataset? It can help you to reduce the variable space and find a possible direction on which variable exclude from you regression model. Doing so, the model will be easier to compute. Here you can find an interesting discussion on using PCA with categorical variables: Can principal component analysis be applied to datasets containing a mix of continuous and categorical variables?

Also, I imagine you're using R for many reasons (I use R too), but it may be easier to use a software like SAS or STATA. They perform better with big data and you don't have to deal with multi-core and parallel computing.

Finally, try to think if it makes sense to use as much rows as possible from your dataset. This is a population dataset, a quasi-population dataset or a sampled dataset? You may obtain better results with a good sampling on your dataset than using the whole data. Take a look at this post: Is sampling relevant in the time of 'big data'?

Hope this helps


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.