Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am considering using Python libraries for doing my Machine Learning experiments. Thus far, I had been relying on WEKA but have been pretty dissatisfied on the whole. This is primarily because I have found WEKA to be not so well supported (very few examples, documentation is sparse and community support is less than desirable in my experience), and have found myself in sticky situations with no help forthcoming. Another reason I am contemplating this move is because I am really liking Python (I am new to Python), and don't want to go back to coding in Java.

So my question is, what are the more (1) comprehensive (2) scalable (100k features, 10k examples) and (3) well supported libraries for doing ML in Python out there? I am particularly interested in doing text classification, and so would like to use a library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.), and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

Based on the past e-mail threads here and elsewhere, I have been looking at PyML, scikits-learn and Orange so far. How have people's experiences been with respect to the above 3 metrics that I mention?

Any other suggestions?

Thanks in advance. Andy

share|improve this question
    
Thank you so much everyone for your comments. I think for now I will go with NLTK+scikit-learn and see where things go. –  Andy Mar 27 '11 at 14:37
1  
I can't help proposing you to look at R, which has way bigger ML library and (being made especially for this job) provides a much more homogeneous environment. R's text mining package is presented here. –  mbq Mar 27 '11 at 16:13
    
@mbq : Yes, I am very aware of R, but since I have to do so much "non-statistics" programming before doing the actual text mining, I think I will stick to Python for now. –  Andy Mar 28 '11 at 0:51
1  
You could use both Python and R. One option is to use a language neutral method of storage like a database to keep results in between data processing steps. –  Faheem Mitha Dec 8 '11 at 7:44
add comment

8 Answers

up vote 25 down vote accepted

About the scikit-learn option: 100k (sparse) features and 10k samples is reasonably small enough to fit in memory hence perfectly doable with scikit-learn (same size as the 20 newsgroups dataset).

Here is a tutorial I gave at PyCon 2011 with a chapter on text classification with exercises and solutions:

I also gave a talk on the topic which is an updated version of the version I gave at PyCon FR. Here are the slides (and the embedded video in the comments):

As for feature selection, have a look at this answer on quora where all the examples are based on the scikit-learn documentation:

We don't have collocation feature extraction in scikit-learn yet. Use nltk and nltk-trainer to do this in the mean time:

share|improve this answer
    
Thanks. This seems like a particularly good way forward. I will try it. –  Andy Mar 27 '11 at 14:35
add comment

In terms of working with text, have a look at NLTK. Very, very well supported & documented (there's even a book online, or in paper if you prefer) and will do the preprocesing you require. You might find Gensim useful as well; the emphasis is on vector space modeling and it's got scalable implementations of LSI and LDA (pLSI too I think) if those are of interest. It will also do selection by tf-idf - I'm not sure that NLTK does. I've used pieces of these on corpora of ~50k without much difficulty.

NLTK: http://www.nltk.org/

Gensim: http://nlp.fi.muni.cz/projekty/gensim/

Unfortunately, as to the main thrust of your question I'm not familiar with the specific libraries you reference (although I've used bits of scikits-learn before).

share|improve this answer
3  
+1 for NLTK. Olivier Grisel, who is doing Machine Learning research and Natural Language Processing, has some good tutorials and slides around the web. I can think of Statistical Learning and Text Classification with NLTK and scikit-learn (PyCON FR 2010), for example. –  chl Mar 27 '11 at 9:45
    
@chi Thanks for that tutorial, I've been poking around a little and it seems he's got a lot of nice material out there. Also displays a confusion matrix with a heatmap in his slides ;) –  JMS Mar 27 '11 at 15:28
add comment

Let me suggest Orange

comprehensive

Yes

scalable (100k features, 10k examples)

Yes

well supported libraries for doing ML in Python out there?

Yes

library that has a good collection of classifiers, feature selection methods (Information Gain, Chi-Sqaured etc.),

All of these work out of box in Orange

and text pre-processing capabilities (stemming, stopword removal, tf-idf etc.).

I have never used Orange for text processing, though

share|improve this answer
    
+1 for Orange. They have great documentation that shows how to get up to speed quickly, and you can use either their visual-programming GUI or call libraries from Python. –  Josh Hemann Oct 18 '11 at 15:07
add comment

Python has a wide range of ML libraries (check out mloss.org as well). However, I always have the feeling that it's more of use for ml researchers than for ml practitioners.

Numpy/SciPy and matplotlib are excellent tools for scientific work with Python. If you are not afraid to hack in most of the math formulas yourself, you will not be disappointed. Also, it is very easy to use the GPU with cudamat or gnumpy - experiments that took days before are now completed in hours or even minutes.

The latest kid on the block is probably Theano. It is a symbolic language for mathematical expressions that comes with opmitimzations, GPU implementations and the über-feature automatic differentiation which is nothing short of awesome for gradient based methods.

Also, as far as I know the NLTK mentioned by JMS is basically the number one open source natural language library out there.

Python is the right tool for machine learning.

share|improve this answer
add comment

Not sure if this is particularly useful, but there's a guide for programmers to learn statistics in Python available online. http://www.greenteapress.com/thinkstats/

It seems pretty good from my brief scan, and it appears to talk about some machine learning methods, so it might be a good place to start.

share|improve this answer
add comment

Check out libsvm.

share|improve this answer
4  
libsvm does not scale well to large numbers of features and samples. Better use liblinear by the same authors. linear models are usually good enough in high dimensional space. –  ogrisel Mar 27 '11 at 11:22
    
@ogrisel: good to know, thanks! –  Mike Lawrence Mar 27 '11 at 12:58
    
Isn't libsvm the svm engine for both scikits-learn and Orange? Or do they really fall back to liblinear in the linear case? –  John Robertson May 24 '12 at 22:01
add comment

SHOGUN (将軍) is a large scale machine learning toolbox, which seems promising.

share|improve this answer
add comment

You can use Weka from your Python code: http://dimitri-christodoulou.blogspot.com/2012/03/use-weka-in-your-python-code.html

share|improve this answer
add comment

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.