2
$\begingroup$

I am training a Decision Tree on a dataset of around 580.000 data points.

I took the following steps:

  • Split the dataset in training (75%) and validation (25%) set.

  • Determined the best depth for the Decision Tree by creating trees with depth ranging between 1 and 100 and taking the one with the best score.

  • Used this optimal depth to create trees with different training set sizes (steps of 5000). I took the error of each tree w.r.t. the training set and the validation set to plot the learning curves.

I got the following learning curves:

learning curves

I am not quite sure how to interpret these curves. Both the error on the training and validation set are low. Since the difference between both curves is not that big, I'm assuming my model does not overfit (a lot).

Is this a correct interpretation? Anything I can do to improve on this?

$\endgroup$
5
  • $\begingroup$ You are overfitting slightly, but that's nothing to worry about. $\endgroup$ Apr 11, 2015 at 14:09
  • 2
    $\begingroup$ Please keep in mind that the balanced tree of depth 100 would have around 2**100 = 1.27e30 leaves. This is much larger than number of points in your data set. Therefore such depth makes no sense to me. I am interested - what is your optimal depth? $\endgroup$
    – lanenok
    Apr 11, 2015 at 16:59
  • $\begingroup$ I did not think about it that way. It gave me 58 as optimal depth. So you suggest to only use values i for which 2**i is smaller than the number of points in the dataset (which does make sense when you think about it). $\endgroup$
    – JNevens
    Apr 11, 2015 at 18:03
  • 1
    $\begingroup$ Your tree is not necessarily balanced, so there is no strict rule. But 58 seems suspicious to me. How many features do you have? Try to visualize your tree. $\endgroup$
    – lanenok
    Apr 11, 2015 at 19:09
  • $\begingroup$ There are 55 features. $\endgroup$
    – JNevens
    Apr 11, 2015 at 19:10

1 Answer 1

4
$\begingroup$

For the plot as it is there is little to worry about. Your CV (and it better be cross-validation error, than a single train/test split) error is decreasing at decreasing rate (and eventually saturates) as the number of your training instances grows. This is normal. Yes, you overfit, but not a lot.

There is a lot to worry about before you obtain this plot.

Please keep in mind that the balanced tree of depth 100 would have around 2**100 = 1.27e30 leaves. This is much larger than number of points in your data set. Therefore such depth makes no sense to me. Since your tree is not always balanced, there is no strict rule. But the optimal depth 58 also seems suspicious to me. Check how many leaves you have. It should be much less than 580K.

The way to improve depends on your purpose. If your primary goal is understanding (looks like so, since you are using a decision tree)

  • have a look at feature importance: probably only few of 55 make a real difference; with python scikit-learn use "clf.feature_importances_" after you trained your classifier

  • build a tree of human-understandable depth (3 or 4, at most 5 or 6) and visualize it

If you are hunting for accuracy, try other methods. Based on your initial choice of a tree:

  • for a single tree GridSearchCV for mean_samples_split or min_samples_leaf combined with max_depth or max_leaf_nodes; of course, you can include other parameters too

  • a simplest way: with 55 features you are good for random forest

  • bagging and boosting, read about Ensemble Methods, for example here with python or find info yourself for a language of your choice

$\endgroup$
2
  • $\begingroup$ I am indeed using the Scikit Learn package in Python. This is a more technical question: can I derive from the DecisionTreeClassifier object which features are used at the top of the tree? Or any other way to derive the more important features? This way, I can e.g. select 10 features and see if my accuracy increases. $\endgroup$
    – JNevens
    Apr 12, 2015 at 7:51
  • $\begingroup$ clf.feature_importances_ for feature importances. I also extended my answer. $\endgroup$
    – lanenok
    Apr 12, 2015 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.