I'm studying the Random Forests and I made a little example to validate my knowledge. I create two classifiers, one with the DecisionTreeClassifier and another with RandomForestClassifier. After I used a cross-validation with the iris dataset and calculate the score for each one, how you could see in the code below. But when I analyze the values of the print, the results differs, what is a little strange for me. In theory if I have a Random Forest with only 1 estimator, that is not equivalent to a simple Tree ? So i don't need to have the same score?

from sklearn import tree, cross_validation
from sklearn.datasets import load_iris

max_depth = 1
n_estimators = 1

X, y = iris.data[:, :2], iris.target  #take only the first two column values
X, y = shuffle(X, y, random_state=42)

# Standardize
mean, std = X.mean(axis=0), X.std(axis=0)
X = (X - mean) / std

kf_total = cross_validation.KFold(len(X), n_folds=2)

clf1 = tree.DecisionTreeClassifier(max_depth = max_depth)
clf2 = RandomForestClassifier(n_estimators=n_estimators)

scoreForest = cross_validation.cross_val_score(clf1, X, y, cv=kf_total, n_jobs = 1)
scoreTree = cross_validation.cross_val_score(clf2, X, y, cv=kf_total, n_jobs = 1)

print 'forest', scoreForest , 'tree', scoreTree

My print result: forest [ 0.64 0.61333333] tree [ 0.69333333 0.66666667]


Random forest is not not just a bunch of trees -- each of them is built on a different resample of objects (i.e. it is a bagging ensemble) and optimisation of splits is clipped to $m$ randomly selected attributes. Both procedures are applied mainly to reduce the correlation of individual trees and thus make voting work, and in context of a single tree they actually hurt its accuracy.

Also, you have set max_depth of a decision tree to 1, which makes it a dumb single-split model rather than a decision tree.

This way you got unrealistically bad accuracies in both cases.

  • 1
    $\begingroup$ Interesting. Increasing the max_depth, I noticed that the values are more accurate $\endgroup$
    – Mike
    Apr 13 '15 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.