6
$\begingroup$

I have a binary response variable I am seeking to predict using Random Forest. I have a sizable dataset of 150k rows, I have about 200 independent variables or features to use to model the outcome.

Many of my features are continuous numerical quantities, say, taking values from 0 to 100, including decimal values therein. However, in 5% to 10% of cases, there will be special values which indicate some reason for why this variable wasn't populated for the row in question, these values are coded in the dataset as "-9", "-8", "-7" currently, even though they are really categorical values, not numerical quantities with a relation to the measure of the bulk of the data in the feature.

Can someone with Random Forest experience advise how I should prepare these data for use in a ML context, specifically with Random Forest? My instinct is to split features for which this is the case into two new feature, one with the numerical values, and NA values where there was previously a special value, and another feature with NAs for the cases where the original value was numeric, and strings for the special values, coding this as a "factor" variable (I am using R). The obvious problem with this approach is that Random Forest won't "know" that these two features are linked, and will not include them together necessarily when building trees. Maybe this isn't such a big problem given that the trees are intentionally "weak learners" that don't try to do everything alone in one tree.

Thoughts on the same feature engineering issue for use in other ML paradigms commonly used for binary classification are also valued, as this will likely end in an ensemble solution.

If it is useful, I am using H2O for this work.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

My instinct is to split features for which this is the case into two new feature, one with the numerical values, and NA values where there was previously a special value, and another feature with NAs for the cases where the original value was numeric, and strings for the special values, coding this as a "factor" variable (I am using R).

This is exactly the correct way to go about this. Do this.

The obvious problem with this approach is that Random Forest won't "know" that these two features are linked, and will not include them together necessarily when building trees.

This should not matter. If the difference is important in your classification task, then your RF should learn it automatically. RFs are pretty good at learning interactions. And if it isn't important, then it isn't important.

Maybe this isn't such a big problem given that the trees are intentionally "weak learners" that don't try to do everything alone in one tree.

That one, too.

If you have any influence at all on the "upstream" data acquisition, try to help people to understand that encoding special values by using "obviously" invalid numericals is not good practice. Too much can go wrong when you reverse this, and there is really no good reason to do this unless you have a very specific use case.

Improve this answer
$\endgroup$
3
  • $\begingroup$ +1 to your answer and to the OP. Let me stress that it is definitely important to make the learner aware of the sparsity pattern in the data. That said, I think that if the NA's in the variable in question are replace by a value outside the range of the current value (so say in a feature that takes values in $U[0,1]$, NA are encoded as 1.5), then theoretically a RF (or a tree learner in general) should be able to distinguish between the two. Once again, what you and the OP suggest is correct, but sometimes I am thinking that this advice stems from using linear models and not tree ones. $\endgroup$
    – usεr11852
    Commented Aug 4, 2018 at 10:00
  • 1
    $\begingroup$ @usεr11852: I see your point, and I agree that my advice applies more to linear models than to RFs. Then again, if we know that certain values are qualitatively different and should not be interpreted in a linear way, it's still better to explicitly tell the RF, not hope for it to find out by itself. $\endgroup$
    – Stephan Kolassa
    Commented Aug 4, 2018 at 11:52
  • $\begingroup$ Agreed. +1 :) (Side-note: Certain ensemble learner implementations (e.g. XGBoost) have in-built capabilities of handling missing values. I have not tested how the "dual" encoding behaves in that case but it is some food for thought too.) $\endgroup$
    – usεr11852
    Commented Aug 4, 2018 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.