# Boosted decision trees in python? [closed]

Is there a a good python library for training boosted decision trees ?

## closed as off-topic by mkt, user20160, Michael Chernick, Peter Flom♦Sep 30 at 11:15

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – mkt, user20160, Michael Chernick, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

• Rpy, of course ;-) – user88 Sep 6 '10 at 19:31
• I agree with mbq. Is there a very good reason why you have to do that in Python? Otherwise I'd use the workhorse R as a back-end as well. – Joris Meys Sep 7 '10 at 13:54
• the only reason being that I have used R only very few times a year or so ago and python I'm using every day... – Andre Holzner Sep 8 '10 at 6:03
• Rpy is a really nasty dependency. R has a huge set of features, and thus it is nice to be able to dig in them using Rpy, but if you have to share that work, you might be in trouble, even if it is across different computers of a same lab, if your lab is in a heterogeneous computing environment. This is due to the fact that Rpy depends on having the right minor versions of Python, numpy and R. For instance, it keeps being broken in the major Linux distributions. – Gael Varoquaux Feb 5 '11 at 10:59
• Answers here seem outdated - they seem to come from the time when scikit.learn was at its beginning. I think readers and the site would benefit if somebody knowledgeable put an updated answer. – Pere Oct 14 '16 at 20:06

## 9 Answers

Updated Answer

The landscape has changed a lot and the answer is clear nowadays:

• scikit-learn is the library in python and has several great algorithms for boosted decision trees
• the "best" boosted decision tree in python is the XGBoost implementation.

Update 1

• Meanwhile, LightGBM, though still quite "new", seems to be equally good or even better then XGBoost

My first look would be at Orange, which is a fully-featured app for ML, with a backend in Python. See e.g. orngEnsemble.

Other promising projects are mlpy and the scikit.learn.

I know that PyCV include several boosting procedures, but apparently not for CART. Take also a look at MLboost

You can use R decision tree library using Rpy(http://rpy.sourceforge.net/). Also check the article "building decision trees using python"(http://onlamp.com/pub/a/python/2...).

there is also

http://opencv.willowgarage.com/documentation/index.html

http://research.engineering.wustl.edu/~amohan/

I had good success with the tree-based learners in Milk: Machine Learning Toolkit for Python. It seems to be under active development, but the documentation was a bit sparse when I was using it. The test suite (github.com/luispedro/milk/blob/master/tests/test_adaboost.py) contains a "boosted stump" though, which could get you going pretty quickly:

import numpy as np
import milk.supervised.tree
import milk.supervised.adaboost

def test_learner():
from milksets import wine
learner = milk.supervised.adaboost.boost_learner(milk.supervised.tree.stump_learner())
features, labels = wine.load()
features = features[labels < 2]
labels = labels[labels < 2] == 0
labels = labels.astype(int)
model = learner.train(features, labels)
train_out = np.array(map(model.apply, features))
assert (train_out == labels).mean() > .9

• I develop milk. If either of you run into any problems, please let me know by email (lpc at cmu dot edu). Bug reports generally get fixed in under 24 hours. – luispedro Jan 5 '11 at 16:25
• In the meanwhile, I've added a bit more documentation on adaboost: packages.python.org/milk/adaboost.html so the above comment might be less valid than it was earlier. – luispedro Feb 23 '11 at 21:41

The scikit-learn now has good regression (and classification) trees and random forests implementations. However, boosted tree still isn't included. People are working on it, but it takes a while to get an efficient implementation.

Disclaimer: I am a scikit-learn developer.

JBoost is an awesome library. It is definitely not written in Python, however It is somewhat language agnostic, because it can be executed from the command line and such so it can be "driven" from Python. I've used it in the past and liked it a lot, particularly the visualization stuff.

I have the same issue right now: I code in Python daily, use R once in a while, and need a good boosted regression tree algorithm. While there are lots of great Python packages for advanced analytics, my searching has not found a good offering for this particular algorithm. So, the route I think I'll be taking in coming weeks is to use the GBM package in R. There is a good paper showing practical issues with using it that can be found here. Importantly, the GBM package was basically used "off the shelf" to win the 2009 KDD Cup. So, I'll probably do all of my pre and post modeling in Python and use RPy to go back and forth with R/GBM.

I have experienced the similar situation with you, I find Orange is hard to tune (maybe it is my problem). In the end, I used Peter Norivig's code for his famous book, in there he provided a well written code framework for tree, all you need is to add boosting in it. This way, you can code anything you like.

Decision Trees - Ada Boosting

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import accuracy_score


Decision Trees with No Boosting

clf_entropy_no_ada = DecisionTreeClassifier(criterion = "entropy", random_state = 100,
max_depth=5, min_samples_leaf=5)
clf_entropy_no_ada.fit(X_train, y_train)


Decision Trees with Ada Boosting

clf_entropy_ada = AdaBoostClassifier(base_estimator= clf_entropy_no_ada,n_estimators=400,learning_rate=1)
clf_entropy_ada.fit(X_train, y_train)


Fitting Models and calculating Accuracy

y_predict_no_ada = clf_entropy_no_ada.predict(X_test)
print ("Accuracy is ", accuracy_score(y_test,y_predict_no_ada)*100)

y_predict_ada = clf_entropy_ada.predict(X_test)
print ("Accuracy is ", accuracy_score(y_test,y_predict_ada)*100)