What software (paid or free) exists for learning large datasets?

Question

Is there available software (or even just relevant papers) that can perform multiclass learning on datasets of 200m+ samples with 50+ classes and 1000+ features?

What are the limits on dataset sizes for neural networks? Decision tree ensembles? SVM?

As an example: Microsoft has developed code that can build decision trees on a 1kilonode cluster of 600m samples per tree over 32 classes with 2000 features. It takes a day to do the training of 3 trees.

Are there publicly available programs that can do this for ANY of the above learning algorithms?

Answer 1

You could try using Weka. It has implemented a large collection of classification algorithms. In your case you definitely want to experiment on the speed of algorithms given your dataset. The Naive Bayes and (Lib)SVM algorithms are known to be quite fast. Also try the LibLINEAR algorithm instead of LibSVM, it is sometimes better suited for large datasets. [NOTE: the LibLINEAR and LibSVM packages are not installed in Weka by default, but Weka's development version 3.7.6 offers a package manager to easily install them]

You might also want to use Weka's Select attributes option to find the most informative features and remove unnecessary features.

In general; I would start out to learn on only a fraction of the dataset and scale up from there. It might be the case the your performance won't go up with more data (though an often heard machine learning rule of thumb says "the more data the better").