10
$\begingroup$

Basically, there are two common ways to learn against huge datasets (when you're confronted by time/space restrictions):

  1. Cheating :) - use just a "manageable" subset for training. The loss of accuracy may be negligible because of the law of diminishing returns - the predictive performance of the model often flattens out long before all the training data is incorporated into it.
  2. Parallel computing - split the problem into smaller parts and solve each one on a separate machine/processor. You need a parallel version of the algorithm though, but good news is that a lot of common algorithms are naturally parallel: nearest-neighbor, decision trees, etc.

Are there other methods? Is there any rule of thumb when to use each? What are the drawbacks of each approach?

$\endgroup$
10
$\begingroup$

Stream Mining is one answer. It is also called:

$\endgroup$
  • $\begingroup$ agreed, the MOA toolbox would be a good place to start $\endgroup$ – tdc Feb 20 '12 at 8:46
7
$\begingroup$

Instead of using just one subset, you could use multiple subsets as in mini-batch learning (e.g. stochastic gradient descent). This way you would still make use of all your data.

$\endgroup$
  • $\begingroup$ Aha that's a good point - I clarified the question. I'm interested in a scenario when you're confronted with time/space restrictions and "cannot afford" mini-batch learning. $\endgroup$ – andreister Feb 16 '12 at 7:54
1
$\begingroup$

Ensembles like bagging or blending -- no data is wasted, the problem automagically becomes trivially parallel and there might be significant accuracy/robustness gains.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.