Given the following from Breiman (2001) section 3.1:
"But unlike cross-validation, where bias is present but its extent unknown, the out-of-bag estimates are unbiased."
Even though it has been said that "the algorithm learned the training data" which was created from a random split to create training and test, if the test data is identical to the training data, the diagnostics (lift curve, tpr vs fpr, auc, etc) are overstated vs diagnostics for test data not identical to train data.
Though trivial, the out-of-bag unbiased estimates used to create the rF model are unable to influence the rF model with a weight (optimization) scheme that preempts the above mentioned overstated diagnostics even though the train and test were randomly created from the same parent file.
Is it the case that a less than sufficient random number generation process caused this?
What percent of records must be different in train vs test to prevent the above mentioned misleading diagnostics? Would just one different record of 10,000 work? Would some percent of different records of N work? Is there a documented threshold to transition from this trivial bias to "the out-of-bag estimates are unbiased" (i.e. unbiased to use as guide to create the rF model to be applied to the true population and not just a subset of the true population)?
I substituted 100 of 320,002 records for the train file & treated this file like a test file to generate predictions and it made no difference (the area under curve maxed out to .99 as though the 100 records were not there).
I substituted 160,001 of 320,002 records (half) for the train file & treated this file like a test file to generate predictions and it made a difference but the area under curve (.82) was still much larger than the area under the curve for a test file with totally different records from the train file (.62).
I need to run more permutations to nail this but it looks like rF is biased if you ultimately score records that are identical to the records used to develop the model. That is, if score targets have a combination of explanatory variables with values equal to the a combination of values in the train file used to develop the rF model, then based on this analysis, the scored output could be bias because the rF process implies there will be no variability for these exact matches. We know that in the real world there will be variability.
For example, what if we are trying to predict who will buy our product online and our capture of who recently created server logs or cookies or abandoned carts or other tracking info is one of our strong explanatory variables. Not all people who create this info ultimately buy online. They might go to a store or do not buy at all. There is variability in the real world. For exact variable value matches, the rF scheme assumes there is no variability for these records. Experience has taught us this is not true and even though there is a high likelihood exact variable value matches will buy (or be consistent with the training file), not all will buy (or be consistent with the training file). Thus, the algorithm consistently errors in the wrong direction by over predicting and this is the definition of bias. In particular for clarity, exact variable value matches could exist for potential customers yet some of these potential customers will never be customers. This is not simply an over-prediction, this is a bias because after many trials the same prediction results.
Maybe this means that we need to develop rF models on samples as small as possible so that the ultimate target score universe is as large as possible. I think a see a bias.
Am I missing something? PLEASE COMMENT.