4
$\begingroup$

I am using a random forest for a 2 class classification problem. But eventually using probability of class "1" returned by the model for my task and not the label. I get AUC of about 70%

Then I compare the probability with the real world value and measure the difference (residual). Then I build a regression random forest model to predict the residual given the same features! This seems to be a weird idea but I tried it. Then I correct the probabilities returned by the first model with the output of the 2nd model and this improved performance in the "test set" Significantly! The 2nd model explains 85% of the variability.

What does this mean? Why is the first model not accurate enough? Even those the same features are used in the 2nd model, it improves performance.

Somehow, the model the predict the residual of the classification model has a higher performance compared to the classification model itself. And both models use the same features.

Improve this question
$\endgroup$
4
  • $\begingroup$ Well, does it mean that for the first run, the random forest is used for a classification problem while for the second run for a regression problem? $\endgroup$
    – Karel Macek
    Jun 25, 2015 at 21:23
  • $\begingroup$ Yes, and the 2nd one predicts the residual of the first. $\endgroup$
    – gnjago
    Jun 25, 2015 at 23:53
  • 1
    $\begingroup$ You could add a ~20 line code example. It always good to discuss concretely where any over-fitting may occour $\endgroup$
    – Soren Havelund Welling
    Jun 27, 2015 at 16:50
  • $\begingroup$ Depending on the details, this is either equivalent or highly related to gradient boosting, using two random forest base models. So, there's precedent for, and principle behind this approach (but, boosting typically uses a much greater number of simpler base models, e.g. single, shallow trees). What's happening is that the second model is compensating for the deficiencies of the first, which might be caused by either overfitting or underfitting. $\endgroup$
    – user20160
    Sep 18, 2018 at 5:23

2 Answers 2

Reset to default
4
$\begingroup$

It means you're over-fitting the training data without assessing the generalization error using e.g. cross-validation.

You can avoid using cross validation when using random forests because of the way it estimates the out-of-bag (OOB) error as it goes. However, once you use those OOB prediction residuals as the inputs to the second random forest model in your pipeline this is no longer true. In order to get an estimate of the generalization error you need to think about this two stage process as components of a new model that needs to be assessed via cross validation. Build your model on a training sample then assess its accuracy on a test set and I guarantee your results will not look as good.

Another way to look at this is see what happens is instead of stopping at 2 random forest models you now built a third model to predict the residuals of the second. You'll get an even better result. Chain enough of these together and you'll predict the test set perfectly.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I am building the first model and the second model using the training set. It is just that the first model uses the class label as y and the second model uses the output of the first model in "probability" as the y. But testing is done with a test set. And the train and test set is different. $\endgroup$
    – gnjago
    Jun 25, 2015 at 23:51
  • $\begingroup$ What result do you get if you just use a random forest regression model instead of the classifier and then regressor? $\endgroup$
    – Bogdanovist
    Jun 26, 2015 at 0:21
1
$\begingroup$

What you did is very smart.

While both models have the same features as input, they don't have the same target values.

One possible explanation (just a wild guess): Maybe, your second model is able to learn about the sparse and dense regions of your data-set. In sparse regions, predictions are inaccurate; while in dense regions they are good.

Just for fun, you should try to push further your edge: train a third model on the output of your second model, to see if you can correct the output of the second model. If that still improves performance, build a fourth model, etc. Then, tell us how many models allow to have a significant performance improvement. I guess, a total of three models could work.

I trust you to test the performance on a test set, i.e. a data-set that was never used during training of any model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.