# First step for big data ($N = 10^{10}$, $p = 2000$)

Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial variables. Let's say there is one regression problem, one unbalanced binary classification problem, and one task of "find out which predictors are most important." My thought for how to approach the problem is:

Fit some predictive model on progressively larger and larger (random) sub-samples of the data until:

1. Fitting and cross-validating the model becomes computationally difficult (e.g., unreasonbly slow on my laptop, R runs out of memory, etc.), OR

2. The training and test RMSE or precision/recall values stabilize.

If the training and test errors did not stabilize (1.), use a simpler model and/or implement multicore or multinode versions of the model and restart from the beginning.

If the training and test errors stabilized (2.):

• If $N_{subset} \ll N$ (i.e., I can still run algorithms on $X_{subset}$ as it's not too large yet), try to improve performance by expanding the feature space or using a more complex model and restarting from the beginning.

• If $N_{subset}$ is 'large' and running further analyses is costly, analyze variable importance and end.

I plan to use packages like biglm, speedglm, multicore, and ff in R initially, and later use more complicated algorithms and/or multinode (on EC2) as necessary.

Does this sound like a reasonable approach, and if so, do you have any specific advice or suggestions? If not, what would you try instead for a data set of this size?

-
That sounds like a reasonable approach. I found a talk about a similar approach: youtube.com/watch?v=X9YVSDWQokQ . – alfa Apr 17 '12 at 12:21
jmlr.csail.mit.edu/proceedings/papers/v7/miller09/miller09.pdf This might be helpful. More details on the maths here: ms.unimelb.edu.au/~millerh/papers/gencor_JCGS.pdf – dcl Apr 17 '12 at 13:56
@dcl: Thanks -- the idea is a nice, computationally cheap, nonlinear variable selection method that will parallelize well (although it is univariate). I might give it a try because it is fast and very simple to code. I was originally thinking of using this very cool algorithm: code.google.com/p/rf-ace. – lockedoff Apr 17 '12 at 18:58
@alfa: I really like this idea! Using a sequential test on larger and larger subsets of the data to sequentially eliminate the worst-performing meta-parameters. This should things up considerably. Thanks! p.s. here is a paper version of the talk: biglearn.org/files/papers/biglearn2011_submission_2.pdf – lockedoff Apr 17 '12 at 19:19