I am currently working on a supervised learning project with sklearn. According to my experiments I observe DecisionTreeClassifier(DTC) performs better than RandomForestClassifier(RFC), both in term of training and testing error (for several test sets actually). Given the fact that RFC is an ensemble of DTCs, would it be acceptable to report that DTC is a better method for a given dataset over RFC? Or is it always that case that you can get a better RFC, but I have not tuned it enough to outperform DTC?

Example of tunegrids I 've tried with grid search cv, where DTC outperforms RFC when the best found specifications are tested on independent test sets:

DTC: [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,10,20,30,50],"class_weight":["balanced"]}]

RFC: [{"criterion":["gini", "entropy"],"n_estimators":[5,10,20,30,40,50],"max_depth":[1,2,3,4,5,10,20,30,50],"class_weight":["balanced"]}]

In short, is RFC always a better choice (or at least as good as) the DTC?

  • 1
    $\begingroup$ According to the no free lunch theorem, no. $\endgroup$ Feb 24 at 17:47
  • $\begingroup$ Don't cross validate over n_estimators: The more the better. I usually just set this to 1000. As Arya mentions, the No Free Lunch Theorem says that no algorithm uniformly dominates another. I also am dubious that a decision tree would do better, as they are high variance estimators and should thus have larger error. Can you post the method in which you validate these models? Because you are doing grid search, I think the best method of validation would be nested cross validation. $\endgroup$ Feb 24 at 17:55
  • $\begingroup$ Well, the method used for validation is scikit-learn's GridSearchCV with scoring="f1_micro" $\endgroup$
    – Enk9456
    Feb 24 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.