Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am considering using Python libraries for doing my Machine Learning experiments. Thus far, I had been relying on WEKA but have been pretty dissatisfied on the whole. This is primarily because I have found WEKA to be not so well supported (very few examples, documentation is sparse and community support is less than desirable in my experience), and have found myself in sticky situations with no help forthcoming. Another reason I am contemplating this move is because I am really liking Python (I am new to Python), and don't want to go back to coding in Java.

So my question is, what are the more (1) comprehensive (2) scalable (100k features, 10k examples) and (3) well supported libraries for doing ML in Python out there? I am particularly interested in doing text classification, and so would like to use a library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.), and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

Based on the past e-mail threads here and elsewhere, I have been looking at PyML, scikits-learn and Orange so far. How have people's experiences been with respect to the above 3 metrics that I mention?

Any other suggestions?

Thanks in advance. Andy

share|improve this question
Thank you so much everyone for your comments. I think for now I will go with NLTK+scikit-learn and see where things go. – Andy Mar 27 '11 at 14:37
I can't help proposing you to look at R, which has way bigger ML library and (being made especially for this job) provides a much more homogeneous environment. R's text mining package is presented here. – mbq Mar 27 '11 at 16:13
@mbq : Yes, I am very aware of R, but since I have to do so much "non-statistics" programming before doing the actual text mining, I think I will stick to Python for now. – Andy Mar 28 '11 at 0:51
You could use both Python and R. One option is to use a language neutral method of storage like a database to keep results in between data processing steps. – Faheem Mitha Dec 8 '11 at 7:44
You can use Weka from your Python code:… – user9739 Mar 9 '12 at 21:59
up vote 36 down vote accepted

About the scikit-learn option: 100k (sparse) features and 10k samples is reasonably small enough to fit in memory hence perfectly doable with scikit-learn (same size as the 20 newsgroups dataset).

Here is a tutorial I gave at PyCon 2011 with a chapter on text classification with exercises and solutions:

I also gave a talk on the topic which is an updated version of the version I gave at PyCon FR. Here are the slides (and the embedded video in the comments):

As for feature selection, have a look at this answer on quora where all the examples are based on the scikit-learn documentation:

We don't have collocation feature extraction in scikit-learn yet. Use nltk and nltk-trainer to do this in the mean time:

share|improve this answer
Thanks. This seems like a particularly good way forward. I will try it. – Andy Mar 27 '11 at 14:35
@ogrisel, some link rot with the first three links (the video in link four is also missing). Here's a link to the new Scikit domain – xtian Dec 28 '14 at 0:35

In terms of working with text, have a look at NLTK. Very, very well supported & documented (there's even a book online, or in paper if you prefer) and will do the preprocesing you require. You might find Gensim useful as well; the emphasis is on vector space modeling and it's got scalable implementations of LSI and LDA (pLSI too I think) if those are of interest. It will also do selection by tf-idf - I'm not sure that NLTK does. I've used pieces of these on corpora of ~50k without much difficulty.



Unfortunately, as to the main thrust of your question I'm not familiar with the specific libraries you reference (although I've used bits of scikits-learn before).

share|improve this answer
+1 for NLTK. Olivier Grisel, who is doing Machine Learning research and Natural Language Processing, has some good tutorials and slides around the web. I can think of Statistical Learning and Text Classification with NLTK and scikit-learn (PyCON FR 2010), for example. – chl Mar 27 '11 at 9:45
@chi Thanks for that tutorial, I've been poking around a little and it seems he's got a lot of nice material out there. Also displays a confusion matrix with a heatmap in his slides ;) – JMS Mar 27 '11 at 15:28

Python has a wide range of ML libraries (check out as well). However, I always have the feeling that it's more of use for ml researchers than for ml practitioners.

Numpy/SciPy and matplotlib are excellent tools for scientific work with Python. If you are not afraid to hack in most of the math formulas yourself, you will not be disappointed. Also, it is very easy to use the GPU with cudamat or gnumpy - experiments that took days before are now completed in hours or even minutes.

The latest kid on the block is probably Theano. It is a symbolic language for mathematical expressions that comes with opmitimzations, GPU implementations and the über-feature automatic differentiation which is nothing short of awesome for gradient based methods.

Also, as far as I know the NLTK mentioned by JMS is basically the number one open source natural language library out there.

Python is the right tool for machine learning.

share|improve this answer

Let me suggest Orange



scalable (100k features, 10k examples)


well supported libraries for doing ML in Python out there?


library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.),

All of these work out of box in Orange

and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

I have never used Orange for text processing, though

share|improve this answer
+1 for Orange. They have great documentation that shows how to get up to speed quickly, and you can use either their visual-programming GUI or call libraries from Python. – Josh Hemann Oct 18 '11 at 15:07

Not sure if this is particularly useful, but there's a guide for programmers to learn statistics in Python available online.

It seems pretty good from my brief scan, and it appears to talk about some machine learning methods, so it might be a good place to start.

share|improve this answer
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – gung Jun 6 '14 at 23:09

Check out libsvm.

share|improve this answer
libsvm does not scale well to large numbers of features and samples. Better use liblinear by the same authors. linear models are usually good enough in high dimensional space. – ogrisel Mar 27 '11 at 11:22
@ogrisel: good to know, thanks! – Mike Lawrence Mar 27 '11 at 12:58
Isn't libsvm the svm engine for both scikits-learn and Orange? Or do they really fall back to liblinear in the linear case? – John Robertson May 24 '12 at 22:01

SHOGUN (将軍) is a large scale machine learning toolbox, which seems promising.

share|improve this answer

open source python ml library PySpark MLlib

proprietary ml library with free trial GraphLab Create

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.