2
$\begingroup$

I have a training set and a test set.
Let's assume the following:

  1. I train random forest on the training set
  2. I make prediction on training set and test set
  3. Then I add those prediction as features back into the training set and the test set
  4. Now I train another random forest but on the training set from step 3
  5. Finally I make prediction on the test set from step 3 with the model from step 4.

Would it count as data leakage when I use prediction from step 2 as feature to the model in step 4-5.

$\endgroup$

1 Answer 1

1
$\begingroup$

No, it’s not. It’s a special case of stacking in general and used widely in practice, Especially with different methods one after another.

In typical stacking you use several models and a second level on top of these as a meta model. The meta model uses predictions from 1st level models and learns on top of them. You've just one model in your first level and a combiner in the second level with additional features (i.e. base features again). This is similar (not the same) to adding layers to neural networks.

I didn't encounter your specific case (i.e. RF after RF), but I recall several other problems first using a typical baseline method and then fit another model on predictions/residuals using extra features to boost it.

TL;DR This is not data leakage.

$\endgroup$
2
  • $\begingroup$ Do you have any reference to this being a special case of stacking. I would like to read something about this or see the results of this being used. $\endgroup$ Nov 22, 2019 at 16:15
  • $\begingroup$ I've added some more comments. $\endgroup$
    – gunes
    Nov 22, 2019 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.