4
$\begingroup$

I created the following learning curve in order to diagnose my Random Forest model.

enter image description here

As I can see the curve indicates high variance and 'underfitting' (not overfitting), because cross-validation error is much lower than training error.

What are the general recommendations to improve a given model? For example, I know that increasing the number of training examples might help resolve high variance issue (for the overfitting case). Is it true for my case? What if I don't have more training data?

$\endgroup$
  • $\begingroup$ Assuming that score is positive (so an accuracy-like measure rather than an error measure), the direction of the learning curve (improving cross validation results with more training cases) makes more sense. And then the situation would be overfitting. $\endgroup$ – cbeleites Jan 27 '16 at 19:43
  • $\begingroup$ @cbeleites: Correct me if I misunderstood your comment. I think that error should increase (till certain point) with the increasing number of training examples. If you have just a single training and validation example, your error will be smaller if compared to 10 examples. $\endgroup$ – Klausos Jan 27 '16 at 20:46
  • $\begingroup$ What are the random forest model training parameters? From the graph it appears to be over training: high training score, low validation score. You can reduce the numbers of tree in your random forest or reduce the depth of the trees to more generalize your model, among other things. $\endgroup$ – Matt L. Jan 27 '16 at 21:41
  • $\begingroup$ @Klausos: error should decrease with more training cases. Particularly, if you plot an appropriate confidence interval for the measured error. Error is expressed relative to the number of tested cases (for regression as well as for classification). The 1 test case error is divided by 1, the 10 test cases error by 10: mean squared error, root mean squared error, % correctly classified test cases, etc.. $\endgroup$ – cbeleites Jan 28 '16 at 9:37
5
$\begingroup$

I suspect you have trained a series of RF regression models and have plotted explained variance(not error) against training set size. Explained variance is the opposite than a error. The value would be between 0 and 1.

Secondly it does not make much sense to diagnose training explained accuracy for a random forest. Samples take the same paths through the trees when training and predicting, so of course a near perfect fit is obtained. That is why out-of-bag training accuracy/error is used.

The cross-validated score increases a little because more samples both lowers bias(deeper trees + denser sampling from data structure) and lowers variance(decreased tree correlation + less sample error).

So everything looks ok and you probably neither have overly over- nor underfitting. I would prefer to (a) simply plot OOB-CV against different settings of hyperparameters or (b) wrap the model in a repeated nested-CV grid search, if you wanna be really thorough. You will probably find the default parameters are close to optimal.

$\endgroup$
  • $\begingroup$ I agree mostly, but the learning curve clearly exhibits some overfitting, which is the nature of random forests. You might want to trade some feature selection and see if that helps. It would certainly decrease the different between your training and cv scores, but would it increase cv? There is only one way to find out. $\endgroup$ – JPN May 25 '16 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.