2
$\begingroup$

I am training a random forest model with ~100 features (call them X1 through X100).

Then I add a new feature X101 = A * X100, where A is a positive constant. X101 is perfectly collinear with X100 and in principle adds no new information. However, training/testing the model with this new set of "independent" variables results in an apparent improvement in the model predictions, in the sense that the new model has a small but significant increase in AUC.

Under what circumstances could this happen?

$\endgroup$
2
$\begingroup$

If the original variable X100 is highly informative, then the model can improve in quality when both X100 and X101 are present because X100 or its cousin X101 can be sampled at each split, improving the probability that the feature appears in the model. That is, the model could sample X100, X101 or both when constructing a new split, so the probability that the information in X100 is present when considering a split is higher than for other features. Under the assumption that X100 is useful for predicting the outcome, this means the model gets an artificial boost.

This is a well-known phenomenon and is related to the reason that random forest importances aren't completely trustworthy -- in this case, X100 and X101 will "share" the variable importance metric, driving both of their rankings down.

$\endgroup$
  • $\begingroup$ Very interesting -- I guess I'm essentially forcing the model to use a useful feature in more of the trees. Would you expect this boost to disappear in some limit? E.g. a huge number of trees, or a very large max tree depth? $\endgroup$ – abeboparebop May 9 '17 at 16:24
  • $\begingroup$ Anything is possible. Perhaps this feature is only useful conditional upon some other splits, or some particular subsetting of the data. In either case, the increased probability that (some form of) X100 is sampled will persist across all trees and all nodes -- whether or not X100 is selected as a split depends on the data. Deeper trees may eventually encounter diminishing (or not) returns for this reason. Trees are iid so adding more of them will just decrease sampling variance of the trees' average vote. $\endgroup$ – Sycorax says Reinstate Monica May 9 '17 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.