While working on a neural network for classification problem I'm dealing with huge number of possible features and information gain seems like a good way to narrow them down (there are hundreds of millions of possible features). However, I also encounter some features that only class A seems to have, and none of the members of class B have that feature. This presents a problem for Information Gain calculation because it calculates entropy and logarithm of 0 is not a nice thing for computers :). At the moment, every time I encounter feature that is only contained within 1 class, I just try to "smooth" it out by assuming 1 sample appeared that belongs to that class.

Is there a better way to deal with missing values when calculating Information Gain?

  • $\begingroup$ What is your ultimate goal? Am I assuming correctly that you are facing a two-class problem? $\endgroup$ – geekoverdose Jun 21 '16 at 23:58
  • $\begingroup$ @geekoverdose - you are assuming correctly. I'm facing a two class problem. $\endgroup$ – Drag0 Jun 23 '16 at 9:08
  • $\begingroup$ Is your goal predicting those classes - or something else? If there are features that only one class has those will be very helpful in class prediction... $\endgroup$ – geekoverdose Jun 23 '16 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.