3
$\begingroup$

I am training a random forest model with ~100 features (call them X1 through X100).

Then I add a new feature X101 = A * X100, where A is a positive constant. X101 is perfectly collinear with X100 and in principle adds no new information. However, training/testing the model with this new set of "independent" variables results in an apparent improvement in the model predictions, in the sense that the new model has a small but significant increase in AUC.

Under what circumstances could this happen?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

If the original variable X100 is highly informative, then the model can improve in quality when both X100 and X101 are present because X100 or its cousin X101 can be sampled at each split, improving the probability that the feature appears in the model. That is, the model could sample X100, X101 or both when constructing a new split, so the probability that the information in X100 is present when considering a split is higher than for other features. Under the assumption that X100 is useful for predicting the outcome, this means the model gets an artificial boost.

This is a well-known phenomenon and is related to the reason that random forest importances aren't completely trustworthy -- in this case, X100 and X101 will "share" the variable importance metric, driving both of their rankings down.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Very interesting -- I guess I'm essentially forcing the model to use a useful feature in more of the trees. Would you expect this boost to disappear in some limit? E.g. a huge number of trees, or a very large max tree depth? $\endgroup$
    – abeboparebop
    Commented May 9, 2017 at 16:24
  • $\begingroup$ Anything is possible. Perhaps this feature is only useful conditional upon some other splits, or some particular subsetting of the data. In either case, the increased probability that (some form of) X100 is sampled will persist across all trees and all nodes -- whether or not X100 is selected as a split depends on the data. Deeper trees may eventually encounter diminishing (or not) returns for this reason. Trees are iid so adding more of them will just decrease sampling variance of the trees' average vote. $\endgroup$
    – Sycorax ♦
    Commented May 9, 2017 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.