Machine Learning using Python

Question

I am considering using Python libraries for doing my Machine Learning experiments. Thus far, I had been relying on WEKA, but have been pretty dissatisfied on the whole. This is primarily because I have found WEKA to be not so well supported (very few examples, documentation is sparse and community support is less than desirable in my experience), and have found myself in sticky situations with no help forthcoming. Another reason I am contemplating this move is because I am really liking Python (I am new to Python), and don't want to go back to coding in Java.

So my question is, what are the more

1. comprehensive
2. scalable (100k features, 10k examples) and
3. well supported libraries for doing ML in Python out there?

I am particularly interested in doing text classification, and so would like to use a library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.), and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

Based on the past e-mail threads here and elsewhere, I have been looking at PyML, scikits-learn and Orange so far. How have people's experiences been with respect to the above 3 metrics that I mention?

Any other suggestions?

Answer 1

About the scikit-learn option: 100k (sparse) features and 10k samples is reasonably small enough to fit in memory hence perfectly doable with scikit-learn (same size as the 20 newsgroups dataset).

Here is a tutorial I gave at PyCon 2011 with a chapter on text classification with exercises and solutions:

• https://github.com/scikit-learn/scikit-learn (online HTML version)

• https://github.com/downloads/scikit-learn/scikit-learn-tutorial/scikit_learn_tutorial.pdf (PDF version)

• https://github.com/scikit-learn/scikit-learn-tutorial (source code + exercises)

I also gave a talk on the topic which is an updated version of the version I gave at PyCon FR. Here are the slides (and the embedded video in the comments):

• http://www.slideshare.net/ogrisel/statistical-machine-learning-for-text-classification-with-scikitlearn-and-nltk

As for feature selection, have a look at this answer on quora where all the examples are based on the scikit-learn documentation:

• http://www.quora.com/What-are-some-feature-selection-methods/answer/Olivier-Grisel

We don't have collocation feature extraction in scikit-learn yet. Use nltk and nltk-trainer to do this in the mean time:

• https://github.com/japerk/nltk-trainer

Answer 2

In terms of working with text, have a look at NLTK. Very, very well supported & documented (there's even a book online, or in paper if you prefer) and will do the preprocesing you require. You might find Gensim useful as well; the emphasis is on vector space modeling and it's got scalable implementations of LSI and LDA (pLSI too I think) if those are of interest. It will also do selection by tf-idf - I'm not sure that NLTK does. I've used pieces of these on corpora of ~50k without much difficulty.

NLTK: http://www.nltk.org/

Gensim: http://nlp.fi.muni.cz/projekty/gensim/

Unfortunately, as to the main thrust of your question I'm not familiar with the specific libraries you reference (although I've used bits of scikits-learn before).

Answer 3

Python has a wide range of ML libraries (check out mloss.org as well). However, I always have the feeling that it's more of use for ml researchers than for ml practitioners.

Numpy/SciPy and matplotlib are excellent tools for scientific work with Python. If you are not afraid to hack in most of the math formulas yourself, you will not be disappointed. Also, it is very easy to use the GPU with cudamat or gnumpy - experiments that took days before are now completed in hours or even minutes.

The latest kid on the block is probably Theano. It is a symbolic language for mathematical expressions that comes with opmitimzations, GPU implementations and the über-feature automatic differentiation which is nothing short of awesome for gradient based methods.

Also, as far as I know the NLTK mentioned by JMS is basically the number one open source natural language library out there.

Python is the right tool for machine learning.

Answer 4

Let me suggest Orange

comprehensive

Yes

scalable (100k features, 10k examples)

Yes

well supported libraries for doing ML in Python out there?

Yes

library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.),

All of these work out of box in Orange

and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

I have never used Orange for text processing, though

Answer 5

Not sure if this is particularly useful, but there's a guide for programmers to learn statistics in Python available online. http://www.greenteapress.com/thinkstats/

It seems pretty good from my brief scan, and it appears to talk about some machine learning methods, so it might be a good place to start.

Answer 6

Check out libsvm.

Answer 7

SHOGUN (将軍) is a large scale machine learning toolbox, which seems promising.

Answer 8

open source python ml library PySpark MLlib https://spark.apache.org/docs/0.9.0/mllib-guide.html

proprietary ml library with free trial GraphLab Create https://web.archive.org/web/20150109004658/https://dato.com/products/create/

Answer 9

As @ogrisel highlighted, scikit-learn is one of the best machine learning packages out there for Python. It is well suited for data-sets as small as 100k (sparse) features and 10k samples, and even for marginally bigger data-sets that may contains over 200k rows. Basically, any dataset that fits in the memory.

But, if you are looking for a highly scalable Python Machine Learning framework, I'd highly recommend Pyspark MLlib. Since datasets these days can grow big exponentially (given the big data and deep learning wave), you would often need a platform that can scale well and run fast not just on the model training phase, but also during the feature engineering phase (feature transformation, feature selection). Let's look at all the three metrics for Spark Mllib platform that you are interested in:

1. Scalability: If your dataset can fit in the memory, scikit-learn should be your choice. If it's too big to fit in the memory, Spark is the way to go. The important thing to note here is that Spark works faster only in a distributed setting.

2. Comprehensiveness: Sklearn is far richer in terms of decent implementations of a large number of commonly used algorithms as compared to spark mllib. The support for data manipulation and transformation is also more richer in scikit-learn. Spark Mllib has sufficient data transformation modules that does the trick majority of the times. So, in case you end up with spark mllib for scalability concerns, you will still be able to get the job done. It has all the support for correlation analysis, feature extraction (tf-idf, word2vec, CountVectorizer), feature transformation (Tokenizer, StopWordsRemover, nn-gram, Binarizer, PCA etc). For a detailed list see the link below:

Extracting, transforming and selecting features in Spark mllib

3. Classification: Spark mllib has all the major algorithms' implementation that you'd be using majority of the times (including algos that work well for text classification). For a detailed overview of what algorithms are available through mllib, see the link below.

Mllib Classification and regression

Bonus: Apache Spark has support for Python, R, Java, and Scala. So, if tomorrow you decide to experiment with a different language (as a personal choice or for professional reasons), you won't have to learn an entirely new framework.

Answer 10

I don't know if your are still looking for some advice (you made this question 5 months ago...). I just started this book and so far is pretty well:

https://www.amazon.com.mx/dp/1491962291/ref=cm_cr_ryp_prd_ttl_sol_3

The author shows code, examples and explains some theory and math "behind the scenes" of ML algorithms. I'm finding this very instructive. Hope this could be the same for you.