Suppose I have columns like this

 date_ago happened_or_not
0  3.0  1
1  1.0  1
2  NaN  0
3  NaN  0
4  3.0  1
5  5.0  1
6  NaN  0
7  NaN  0
8  2.0  1

Now the first column contains null values when they should be null, how do I treat this column and eventually pass it into RandomForestClassifier? Any idea is appreciated!

I tried to search for this but everything is concerning when the data is actually missing, where as in this case it's not missing but rather just null and unable to pass into RandomForestClassifier.

  • $\begingroup$ You are already on the right track,imo. I'd just invert0 and 1 in happend_or_not, and set to 0 the NaN in date_ago. $\endgroup$ – RLave Oct 22 '18 at 14:56
  • $\begingroup$ Thanks, I can understand the reasoning, but why invert 0 and 1 in happened_or_not? Is there a particular reason? $\endgroup$ – Rocky Li Oct 22 '18 at 15:00
  • 1
    $\begingroup$ This is an interesting question, I really don't know if the forest would treat it differently..Now that I think about it, it shouldn't make a difference, the important part is to remove the NaN values. $\endgroup$ – RLave Oct 22 '18 at 15:03
  • $\begingroup$ You have several options. 1) Remove those rows with NaNs. 2) Fill the NaN values by some specific number. 3) Fill the NaN values by some random representative values from the rest of the dataset. $\endgroup$ – sega_sai Oct 22 '18 at 15:04
  • $\begingroup$ @RLave And also to my concern is that if I replace all NaN with 0, does that constitute a relationship between other entries such that the event that it does not happen is now less than if it happened before. Where as logically this relationship does not exist. $\endgroup$ – Rocky Li Oct 22 '18 at 15:48

You have to go back to the algorithm behind the random forest to have an insight of what you should do.

If the variable happened_or_not is relevant, you would expect your classifier to produce trees with logic formulaes such as if event happened AND date_ago > 2.5 THEN probability of other event = 0.5 and other branch where the condition would be if NOT event happened THEN probability of other event = 0.2

Now, the question you ask can be translated into "how can I encode, with these two variables, the fact that an event happened ?"

What you could do, in this case, is to replace the NaNs in date_ago with a negative number (say -1), and get rid of the other column.

It will not cause any confusion to the random forest algorithm, now the rule if date_ago < 0 will simply be equivalent to NOT event_happened.

Note that this work only if date_ago is positive. Otherwise, you could just replace the NaNs by any value smaller than the smallest element (or larger than the largest element).

  • $\begingroup$ Thanks, but if I replace NaN with a -1, Does that infer that if it doesn't exist the value should be counted as smaller than if it existed? i.e. inferring a negative relationship for the event of not showing up? Imagine if a replaced it with 10000 instead, which would now infer a positive relationship. Does this make a difference to Random Forest Classifier? $\endgroup$ – Rocky Li Oct 22 '18 at 15:43
  • $\begingroup$ No. A random forest classifier will estimate various indicator functions, therefore, they would not "mind" that a huge discontinuity appears with this extra point. A linear regression, on the other hand, would be badly affected by this recipy $\endgroup$ – RUser4512 Oct 22 '18 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.