# What software (paid or free) exists for learning large datasets?

Is there available software (or even just relevant papers) that can perform multiclass learning on datasets of 200m+ samples with 50+ classes and 1000+ features?

What are the limits on dataset sizes for neural networks? Decision tree ensembles? SVM?

As an example: Microsoft has developed code that can build decision trees on a 1kilonode cluster of 600m samples per tree over 32 classes with 2000 features. It takes a day to do the training of 3 trees.

Are there publicly available programs that can do this for ANY of the above learning algorithms?

## 2 Answers

You could try using Weka. It has implemented a large collection of classification algorithms. In your case you definitely want to experiment on the speed of algorithms given your dataset. The Naive Bayes and (Lib)SVM algorithms are known to be quite fast. Also try the LibLINEAR algorithm instead of LibSVM, it is sometimes better suited for large datasets. [NOTE: the LibLINEAR and LibSVM packages are not installed in Weka by default, but Weka's development version 3.7.6 offers a package manager to easily install them]

You might also want to use Weka's Select attributes option to find the most informative features and remove unnecessary features.

In general; I would start out to learn on only a fraction of the dataset and scale up from there. It might be the case the your performance won't go up with more data (though an often heard machine learning rule of thumb says "the more data the better").

• If I'm not mistaken, the first step of Weka is to load the entire dataset into memory. I believe every value is stored as double, so that's 8 bytes per value. With 1000 features and 200 million records, that's 200 billion values and thus 1.6 trillion bytes. In 2012, it would be difficult to get even a small portion of that into memory, so it seems to me that perhaps plain-vanilla Weka is not the best place to start. – Michael McGowan Jun 11 '12 at 17:52
• Correct. That is why I suggest to start out with a fraction of the dataset, which can be created without Weka. Some initial tests can then be done on this data and the author can then decide if some features can be pruned. From there on larger and larger datasets can be used until (and I haven't explicitly mention this indeed) the limits of Weka are reached. – Sicco Jun 11 '12 at 20:23
• Unfortunately watering down the dataset is not an option. Ive already done that and it doesnt work. "The more data the better". – twerdster Jun 11 '12 at 21:35
• mmh, have you checked this page? – user603 Jun 1 '14 at 13:27

Similar to Weka, you may also try SCaVis. You can create large data containers using Python language (or Java, Groovy, Rubu - they are all supported by SCaVis). I think if you do not want to create in-memory container, try to use PFile object that can scan your data line-by-line without loading all data to the memory