Basically, there are two common ways to learn against huge datasets (when you're confronted by time/space restrictions):
- Cheating :) - use just a "manageable" subset for training. The loss of accuracy may be negligible because of the law of diminishing returns - the predictive performance of the model often flattens out long before all the training data is incorporated into it.
- Parallel computing - split the problem into smaller parts and solve each one on a separate machine/processor. You need a parallel version of the algorithm though, but good news is that a lot of common algorithms are naturally parallel: nearest-neighbor, decision trees, etc.
Are there other methods? Is there any rule of thumb when to use each? What are the drawbacks of each approach?