I am considering using Python libraries for doing my Machine Learning experiments. Thus far, I had been relying on WEKA, but have been pretty dissatisfied on the whole. This is primarily because I have found WEKA to be not so well supported (very few examples, documentation is sparse and community support is less than desirable in my experience), and have found myself in sticky situations with no help forthcoming. Another reason I am contemplating this move is because I am really liking Python (I am new to Python), and don't want to go back to coding in Java.

So my question is, what are the more

  1. comprehensive
  2. scalable (100k features, 10k examples) and
  3. well supported libraries for doing ML in Python out there?

I am particularly interested in doing text classification, and so would like to use a library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.), and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

Based on the past e-mail threads here and elsewhere, I have been looking at PyML, scikits-learn and Orange so far. How have people's experiences been with respect to the above 3 metrics that I mention?

Any other suggestions?

  • $\begingroup$ Thank you so much everyone for your comments. I think for now I will go with NLTK+scikit-learn and see where things go. $\endgroup$ – Andy Mar 27 '11 at 14:37
  • 1
    $\begingroup$ I can't help proposing you to look at R, which has way bigger ML library and (being made especially for this job) provides a much more homogeneous environment. R's text mining package is presented here. $\endgroup$ – user88 Mar 27 '11 at 16:13
  • $\begingroup$ @mbq : Yes, I am very aware of R, but since I have to do so much "non-statistics" programming before doing the actual text mining, I think I will stick to Python for now. $\endgroup$ – Andy Mar 28 '11 at 0:51
  • 1
    $\begingroup$ You could use both Python and R. One option is to use a language neutral method of storage like a database to keep results in between data processing steps. $\endgroup$ – Faheem Mitha Dec 8 '11 at 7:44
  • $\begingroup$ You can use Weka from your Python code: dimitri-christodoulou.blogspot.com/2012/03/… $\endgroup$ – user9739 Mar 9 '12 at 21:59

10 Answers 10


About the scikit-learn option: 100k (sparse) features and 10k samples is reasonably small enough to fit in memory hence perfectly doable with scikit-learn (same size as the 20 newsgroups dataset).

Here is a tutorial I gave at PyCon 2011 with a chapter on text classification with exercises and solutions:

I also gave a talk on the topic which is an updated version of the version I gave at PyCon FR. Here are the slides (and the embedded video in the comments):

As for feature selection, have a look at this answer on quora where all the examples are based on the scikit-learn documentation:

We don't have collocation feature extraction in scikit-learn yet. Use nltk and nltk-trainer to do this in the mean time:

  • $\begingroup$ Thanks. This seems like a particularly good way forward. I will try it. $\endgroup$ – Andy Mar 27 '11 at 14:35
  • $\begingroup$ @ogrisel, some link rot with the first three links (the video in link four is also missing). Here's a link to the new Scikit domain scikit-learn.org $\endgroup$ – xtian Dec 28 '14 at 0:35

In terms of working with text, have a look at NLTK. Very, very well supported & documented (there's even a book online, or in paper if you prefer) and will do the preprocesing you require. You might find Gensim useful as well; the emphasis is on vector space modeling and it's got scalable implementations of LSI and LDA (pLSI too I think) if those are of interest. It will also do selection by tf-idf - I'm not sure that NLTK does. I've used pieces of these on corpora of ~50k without much difficulty.

NLTK: http://www.nltk.org/

Gensim: http://nlp.fi.muni.cz/projekty/gensim/

Unfortunately, as to the main thrust of your question I'm not familiar with the specific libraries you reference (although I've used bits of scikits-learn before).

  • 3
    $\begingroup$ +1 for NLTK. Olivier Grisel, who is doing Machine Learning research and Natural Language Processing, has some good tutorials and slides around the web. I can think of Statistical Learning and Text Classification with NLTK and scikit-learn (PyCON FR 2010), for example. $\endgroup$ – chl Mar 27 '11 at 9:45
  • $\begingroup$ @chi Thanks for that tutorial, I've been poking around a little and it seems he's got a lot of nice material out there. Also displays a confusion matrix with a heatmap in his slides ;) $\endgroup$ – JMS Mar 27 '11 at 15:28

Python has a wide range of ML libraries (check out mloss.org as well). However, I always have the feeling that it's more of use for ml researchers than for ml practitioners.

Numpy/SciPy and matplotlib are excellent tools for scientific work with Python. If you are not afraid to hack in most of the math formulas yourself, you will not be disappointed. Also, it is very easy to use the GPU with cudamat or gnumpy - experiments that took days before are now completed in hours or even minutes.

The latest kid on the block is probably Theano. It is a symbolic language for mathematical expressions that comes with opmitimzations, GPU implementations and the über-feature automatic differentiation which is nothing short of awesome for gradient based methods.

Also, as far as I know the NLTK mentioned by JMS is basically the number one open source natural language library out there.

Python is the right tool for machine learning.


Let me suggest Orange



scalable (100k features, 10k examples)


well supported libraries for doing ML in Python out there?


library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.),

All of these work out of box in Orange

and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

I have never used Orange for text processing, though

  • $\begingroup$ +1 for Orange. They have great documentation that shows how to get up to speed quickly, and you can use either their visual-programming GUI or call libraries from Python. $\endgroup$ – Josh Hemann Oct 18 '11 at 15:07

Not sure if this is particularly useful, but there's a guide for programmers to learn statistics in Python available online. http://www.greenteapress.com/thinkstats/

It seems pretty good from my brief scan, and it appears to talk about some machine learning methods, so it might be a good place to start.


Check out libsvm.

  • 4
    $\begingroup$ libsvm does not scale well to large numbers of features and samples. Better use liblinear by the same authors. linear models are usually good enough in high dimensional space. $\endgroup$ – ogrisel Mar 27 '11 at 11:22
  • $\begingroup$ Isn't libsvm the svm engine for both scikits-learn and Orange? Or do they really fall back to liblinear in the linear case? $\endgroup$ – John Robertson May 24 '12 at 22:01

SHOGUN (将軍) is a large scale machine learning toolbox, which seems promising.


open source python ml library PySpark MLlib https://spark.apache.org/docs/0.9.0/mllib-guide.html

proprietary ml library with free trial GraphLab Create https://dato.com/products/create/


As @ogrisel highlighted, scikit-learn is one of the best machine learning packages out there for Python. It is well suited for data-sets as small as 100k (sparse) features and 10k samples, and even for marginally bigger data-sets that may contains over 200k rows. Basically, any dataset that fits in the memory.

But, if you are looking for a highly scalable Python Machine Learning framework, I'd highly recommend Pyspark MLlib. Since datasets these days can grow big exponentially (given the big data and deep learning wave), you would often need a platform that can scale well and run fast not just on the model training phase, but also during the feature engineering phase (feature transformation, feature selection). Let's look at all the three metrics for Spark Mllib platform that you are interested in:

  1. Scalability: If your dataset can fit in the memory, scikit-learn should be your choice. If it's too big to fit in the memory, Spark is the way to go. The important thing to note here is that Spark works faster only in a distributed setting.

  2. Comprehensiveness: Sklearn is far richer in terms of decent implementations of a large number of commonly used algorithms as compared to spark mllib. The support for data manipulation and transformation is also more richer in scikit-learn. Spark Mllib has sufficient data transformation modules that does the trick majority of the times. So, in case you end up with spark mllib for scalability concerns, you will still be able to get the job done. It has all the support for correlation analysis, feature extraction (tf-idf, word2vec, CountVectorizer), feature transformation (Tokenizer, StopWordsRemover, nn-gram, Binarizer, PCA etc). For a detailed list see the link below:

Extracting, transforming and selecting features in Spark mllib

  1. Classification: Spark mllib has all the major algorithms' implementation that you'd be using majority of the times (including algos that work well for text classification). For a detailed overview of what algorithms are available through mllib, see the link below.

Mllib Classification and regression

Bonus: Apache Spark has support for Python, R, Java, and Scala. So, if tomorrow you decide to experiment with a different language (as a personal choice or for professional reasons), you won't have to learn an entirely new framework.


I don't know if your are still looking for some advice (you made this question 5 months ago...). I just started this book and so far is pretty well:


The author shows code, examples and explains some theory and math "behind the scenes" of ML algorithms. I'm finding this very instructive. Hope this could be the same for you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.