Is there a a good python library for training boosted decision trees ?
closed as off-topic by mkt, user20160, Michael Chernick, Peter Flom♦ Sep 30 at 11:15
This question appears to be off-topic. The users who voted to close gave this specific reason:
- "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – mkt, user20160, Michael Chernick, Peter Flom
The landscape has changed a lot and the answer is clear nowadays:
- scikit-learn is the library in python and has several great algorithms for boosted decision trees
- the "best" boosted decision tree in python is the XGBoost implementation.
You can use R decision tree library using Rpy(http://rpy.sourceforge.net/). Also check the article "building decision trees using python"(http://onlamp.com/pub/a/python/2...).
there is also
I had good success with the tree-based learners in Milk: Machine Learning Toolkit for Python. It seems to be under active development, but the documentation was a bit sparse when I was using it. The test suite (github.com/luispedro/milk/blob/master/tests/test_adaboost.py) contains a "boosted stump" though, which could get you going pretty quickly:
import numpy as np import milk.supervised.tree import milk.supervised.adaboost def test_learner(): from milksets import wine learner = milk.supervised.adaboost.boost_learner(milk.supervised.tree.stump_learner()) features, labels = wine.load() features = features[labels < 2] labels = labels[labels < 2] == 0 labels = labels.astype(int) model = learner.train(features, labels) train_out = np.array(map(model.apply, features)) assert (train_out == labels).mean() > .9
The scikit-learn now has good regression (and classification) trees and random forests implementations. However, boosted tree still isn't included. People are working on it, but it takes a while to get an efficient implementation.
Disclaimer: I am a scikit-learn developer.
JBoost is an awesome library. It is definitely not written in Python, however It is somewhat language agnostic, because it can be executed from the command line and such so it can be "driven" from Python. I've used it in the past and liked it a lot, particularly the visualization stuff.
I have the same issue right now: I code in Python daily, use R once in a while, and need a good boosted regression tree algorithm. While there are lots of great Python packages for advanced analytics, my searching has not found a good offering for this particular algorithm. So, the route I think I'll be taking in coming weeks is to use the GBM package in R. There is a good paper showing practical issues with using it that can be found here. Importantly, the GBM package was basically used "off the shelf" to win the 2009 KDD Cup. So, I'll probably do all of my pre and post modeling in Python and use RPy to go back and forth with R/GBM.
I have experienced the similar situation with you, I find Orange is hard to tune (maybe it is my problem). In the end, I used Peter Norivig's code for his famous book, in there he provided a well written code framework for tree, all you need is to add boosting in it. This way, you can code anything you like.
Decision Trees - Ada Boosting
from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import AdaBoostClassifier from sklearn.metrics import accuracy_score
Decision Trees with No Boosting
clf_entropy_no_ada = DecisionTreeClassifier(criterion = "entropy", random_state = 100, max_depth=5, min_samples_leaf=5) clf_entropy_no_ada.fit(X_train, y_train)
Decision Trees with Ada Boosting
clf_entropy_ada = AdaBoostClassifier(base_estimator= clf_entropy_no_ada,n_estimators=400,learning_rate=1) clf_entropy_ada.fit(X_train, y_train)
Fitting Models and calculating Accuracy
y_predict_no_ada = clf_entropy_no_ada.predict(X_test) print ("Accuracy is ", accuracy_score(y_test,y_predict_no_ada)*100) y_predict_ada = clf_entropy_ada.predict(X_test) print ("Accuracy is ", accuracy_score(y_test,y_predict_ada)*100)