0
$\begingroup$

I would like to create predictions for all outcomes in my sample of data. Typically with a random forest you would split the data in order to train on one part and predict on the other part.

Is there any possible way to train the forest and predict on to that same data without the predictions being biased?

I have been using SAS HP procedures and python Scikit-learn RandomForestClassifier, but have had little luck.

$\endgroup$
1
$\begingroup$

The most straightforward answer is No! You can't train and predict on the same data without biasing. Nevertheless you can use a technique called K-fold cross validation which involves partitioning a sample of data into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are combined (e.g. averaged) over the rounds to estimate a final predictive model.

4-Fold cross validation, example from Wikipedia

You may also like to check this Udacity video on K-Fold Cross Validation.

$\endgroup$
  • $\begingroup$ Furthermore, you can check out nested cross-validation. In bioinformatics, there's for example a technique for so-called repeated double cross-validation to do what you want to do: sort of unbiased predictions of all of your data. That's however done for PLS instead of RF, but you can always custom design your own nested cross-validation and put the RF core in the inner loop. $\endgroup$ – CarlBrunius Sep 22 '17 at 19:30
  • $\begingroup$ Are k-fold cross-validation predictions less biased than OOB predictions that you get basically for free? $\endgroup$ – Michael M Sep 22 '17 at 19:35
  • $\begingroup$ @MichaelM, it has smaller variance and greater bias than CV, please take a look at the plots in 'Question 5' here. The author also points to other reasons that CV is a more safe strategy than relying only on OOB. $\endgroup$ – xboard Sep 22 '17 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.