Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there a a good python library for training boosted decision trees ?

share|improve this question
Rpy, of course ;-) – mbq Sep 6 '10 at 19:31
I agree with mbq. Is there a very good reason why you have to do that in Python? Otherwise I'd use the workhorse R as a back-end as well. – Joris Meys Sep 7 '10 at 13:54
the only reason being that I have used R only very few times a year or so ago and python I'm using every day... – Andre Holzner Sep 8 '10 at 6:03
Rpy is a really nasty dependency. R has a huge set of features, and thus it is nice to be able to dig in them using Rpy, but if you have to share that work, you might be in trouble, even if it is across different computers of a same lab, if your lab is in a heterogeneous computing environment. This is due to the fact that Rpy depends on having the right minor versions of Python, numpy and R. For instance, it keeps being broken in the major Linux distributions. – Gael Varoquaux Feb 5 '11 at 10:59
@mbq, @Joris Meys, @Andre Holzner or @Gael, I am running into the same problem. Could any of you point me to those R boosted decision trees packages? (is it just rpart or are there others?) Thanks! – Amelio Vazquez-Reina Apr 25 '11 at 15:35
up vote 11 down vote accepted

My first look would be at Orange, which is a fully-featured app for ML, with a backend in Python. See e.g. orngEnsemble.

Other promising projects are mlpy and the scikit.learn.

I know that PyCV include several boosting procedures, but apparently not for CART. Take also a look at MLboost

share|improve this answer

You can use R decision tree library using Rpy( Also check the article "building decision trees using python"(

there is also

share|improve this answer

I had good success with the tree-based learners in Milk: Machine Learning Toolkit for Python. It seems to be under active development, but the documentation was a bit sparse when I was using it. The test suite ( contains a "boosted stump" though, which could get you going pretty quickly:

import numpy as np
import milk.supervised.tree
import milk.supervised.adaboost

def test_learner():
    from milksets import wine
    learner = milk.supervised.adaboost.boost_learner(milk.supervised.tree.stump_learner())
    features, labels = wine.load()
    features = features[labels < 2]
    labels = labels[labels < 2] == 0
    labels = labels.astype(int)
    model = learner.train(features, labels)
    train_out = np.array(map(model.apply, features))
    assert (train_out == labels).mean() > .9
share|improve this answer
I develop milk. If either of you run into any problems, please let me know by email (lpc at cmu dot edu). Bug reports generally get fixed in under 24 hours. – luispedro Jan 5 '11 at 16:25
In the meanwhile, I've added a bit more documentation on adaboost: so the above comment might be less valid than it was earlier. – luispedro Feb 23 '11 at 21:41

The scikit-learn now has good regression (and classification) trees and random forests implementations. However, boosted tree still isn't included. People are working on it, but it takes a while to get an efficient implementation.

Disclaimer: I am a scikit-learn developer.

share|improve this answer

JBoost is an awesome library. It is definitely not written in Python, however It is somewhat language agnostic, because it can be executed from the command line and such so it can be "driven" from Python. I've used it in the past and liked it a lot, particularly the visualization stuff.

share|improve this answer

I have the same issue right now: I code in Python daily, use R once in a while, and need a good boosted regression tree algorithm. While there are lots of great Python packages for advanced analytics, my searching has not found a good offering for this particular algorithm. So, the route I think I'll be taking in coming weeks is to use the GBM package in R. There is a good paper showing practical issues with using it that can be found here. Importantly, the GBM package was basically used "off the shelf" to win the 2009 KDD Cup. So, I'll probably do all of my pre and post modeling in Python and use RPy to go back and forth with R/GBM.

share|improve this answer

I have experienced the similar situation with you, I find Orange is hard to tune (maybe it is my problem). In the end, I used Peter Norivig's code for his famous book, in there he provided a well written code framework for tree, all you need is to add boosting in it. This way, you can code anything you like.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.