Boosted decision trees in python?

Is there a a good python library for training boosted decision trees ?

-
Rpy, of course ;-) –  mbq Sep 6 '10 at 19:31
I agree with mbq. Is there a very good reason why you have to do that in Python? Otherwise I'd use the workhorse R as a back-end as well. –  Joris Meys Sep 7 '10 at 13:54
the only reason being that I have used R only very few times a year or so ago and python I'm using every day... –  Andre Holzner Sep 8 '10 at 6:03
Rpy is a really nasty dependency. R has a huge set of features, and thus it is nice to be able to dig in them using Rpy, but if you have to share that work, you might be in trouble, even if it is across different computers of a same lab, if your lab is in a heterogeneous computing environment. This is due to the fact that Rpy depends on having the right minor versions of Python, numpy and R. For instance, it keeps being broken in the major Linux distributions. –  Gael Varoquaux Feb 5 '11 at 10:59
@mbq, @Joris Meys, @Andre Holzner or @Gael, I am running into the same problem. Could any of you point me to those R boosted decision trees packages? (is it just rpart or are there others?) Thanks! –  Amelio Vazquez-Reina Apr 25 '11 at 15:35

My first look would be at Orange, which is a fully-featured app for ML, with a backend in Python. See e.g. orngEnsemble.

Other promising projects are mlpy and the scikit.learn.

I know that PyCV include several boosting procedures, but apparently not for CART. Take also a look at MLboost

-
You are a great resource for new softwares. I had no idea about Orange! –  suncoolsu Nov 7 '10 at 18:28
I am a developer of the scikit-learn. I must say that it does not including boosting or decision trees yet. Some people are working on it, so I hope it will be included in 6 months. –  Gael Varoquaux Feb 5 '11 at 11:02
@Gael That's good to know! I wish you nice coding sprints :) –  chl Feb 5 '11 at 11:12

You can use R decision tree library using Rpy(http://rpy.sourceforge.net/). Also check the article "building decision trees using python"(http://onlamp.com/pub/a/python/2...).

there is also

http://opencv.willowgarage.com/documentation/index.html

http://research.engineering.wustl.edu/~amohan/

-

I had good success with the tree-based learners in Milk: Machine Learning Toolkit for Python. It seems to be under active development, but the documentation was a bit sparse when I was using it. The test suite (github.com/luispedro/milk/blob/master/tests/test_adaboost.py) contains a "boosted stump" though, which could get you going pretty quickly:

import numpy as np
import milk.supervised.tree

def test_learner():
from milksets import wine
features, labels = wine.load()
features = features[labels < 2]
labels = labels[labels < 2] == 0
labels = labels.astype(int)
model = learner.train(features, labels)
train_out = np.array(map(model.apply, features))
assert (train_out == labels).mean() > .9

-
I develop milk. If either of you run into any problems, please let me know by email (lpc at cmu dot edu). Bug reports generally get fixed in under 24 hours. –  luispedro Jan 5 '11 at 16:25
In the meanwhile, I've added a bit more documentation on adaboost: packages.python.org/milk/adaboost.html so the above comment might be less valid than it was earlier. –  luispedro Feb 23 '11 at 21:41

The scikit-learn now has good regression (and classification) trees and random forests implementations. However, boosted tree still isn't included. People are working on it, but it takes a while to get an efficient implementation.

Disclaimer: I am a scikit-learn developer.

-

JBoost is an awesome library. It is definitely not written in Python, however It is somewhat language agnostic, because it can be executed from the command line and such so it can be "driven" from Python. I've used it in the past and liked it a lot, particularly the visualization stuff.

-

I have the same issue right now: I code in Python daily, use R once in a while, and need a good boosted regression tree algorithm. While there are lots of great Python packages for advanced analytics, my searching has not found a good offering for this particular algorithm. So, the route I think I'll be taking in coming weeks is to use the GBM package in R. There is a good paper showing practical issues with using it that can be found here. Importantly, the GBM package was basically used "off the shelf" to win the 2009 KDD Cup. So, I'll probably do all of my pre and post modeling in Python and use RPy to go back and forth with R/GBM.

-

I have experienced the similar situation with you, I find Orange is hard to tune (maybe it is my problem). In the end, I used Peter Norivig's code for his famous book, in there he provided a well written code framework for tree, all you need is to add boosting in it. This way, you can code anything you like.

-