Boosted decision trees in python?

Question

Is there a a good python library for training boosted decision trees ?

Answer 1

My first look would be at Orange, which is a fully-featured app for ML, with a backend in Python. See e.g. orngEnsemble.

Other promising projects are mlpy and the scikit.learn.

I know that PyCV include several boosting procedures, but apparently not for CART. Take also a look at MLboost

Answer 2

You can use R decision tree library using Rpy(http://rpy.sourceforge.net/). Also check the article "building decision trees using python"(http://onlamp.com/pub/a/python/2...).

there is also

http://opencv.willowgarage.com/documentation/index.html

http://research.engineering.wustl.edu/~amohan/

Answer 3

I had good success with the tree-based learners in Milk: Machine Learning Toolkit for Python. It seems to be under active development, but the documentation was a bit sparse when I was using it. The test suite (github.com/luispedro/milk/blob/master/tests/test_adaboost.py) contains a "boosted stump" though, which could get you going pretty quickly:

import numpy as np
import milk.supervised.tree
import milk.supervised.adaboost

def test_learner():
    from milksets import wine
    learner = milk.supervised.adaboost.boost_learner(milk.supervised.tree.stump_learner())
    features, labels = wine.load()
    features = features[labels < 2]
    labels = labels[labels < 2] == 0
    labels = labels.astype(int)
    model = learner.train(features, labels)
    train_out = np.array(map(model.apply, features))
    assert (train_out == labels).mean() > .9

Answer 4

The scikit-learn now has good regression (and classification) trees and random forests implementations. However, boosted tree still isn't included. People are working on it, but it takes a while to get an efficient implementation.

Disclaimer: I am a scikit-learn developer.

Answer 5

JBoost is an awesome library. It is definitely not written in Python, however It is somewhat language agnostic, because it can be executed from the command line and such so it can be "driven" from Python. I've used it in the past and liked it a lot, particularly the visualization stuff.

Answer 6

I have the same issue right now: I code in Python daily, use R once in a while, and need a good boosted regression tree algorithm. While there are lots of great Python packages for advanced analytics, my searching has not found a good offering for this particular algorithm. So, the route I think I'll be taking in coming weeks is to use the GBM package in R. There is a good paper showing practical issues with using it that can be found here. Importantly, the GBM package was basically used "off the shelf" to win the 2009 KDD Cup. So, I'll probably do all of my pre and post modeling in Python and use RPy to go back and forth with R/GBM.

Answer 7

I have experienced the similar situation with you, I find Orange is hard to tune (maybe it is my problem). In the end, I used Peter Norivig's code for his famous book, in there he provided a well written code framework for tree, all you need is to add boosting in it. This way, you can code anything you like.