3
$\begingroup$

I'm making a random forest classifier. In every tutorial, there is a very simple example of how to calculate entropy with Boolean attributes.
In my problem, I have attribute values that are calculated by tf-idf schema, and values are real numbers.
Is there some clever way of applying an information gain function so it will calculate IG with real-number weights? Or should I use discretization like:

0 = 0  
(-0 - 0.1> = 1  
(-0.1 - 0.2> = 2

etc.?

EDIT
I have functions:

$$ IG(X) = E(C) - E(C,A), $$

$$ E(C) = \sum\limits_{i=1}^C-P(c_i)\log(P(c_i)), $$

and
$$ E(C,A) = \sum\limits_{a\in A}P(a)E(a). $$

But I have an infinite number of possible values of $A$ and I think I should perform discretization of these values. Do you agree?

Improve this question
$\endgroup$
8
  • $\begingroup$ I think this is a duplicate question. There are plenty of questions like that on CV. See this question here: stats.stackexchange.com/questions/95839/… $\endgroup$
    – rapaio
    Jun 20, 2014 at 10:29
  • $\begingroup$ My question is how can i use real-number attributes for calculating Information Gain. Normally, my attribute would have known number of possible values, here, i have a real numbers, and probably, ther won't be two documents with attribte a1 with same values. $\endgroup$
    – kam
    Jun 20, 2014 at 11:01
  • $\begingroup$ My answer from the posted question explains how can you compute InfoGain for numerical attributes, like yours. $\endgroup$
    – rapaio
    Jun 20, 2014 at 11:03
  • $\begingroup$ Ok, so if you have a classifier, you must have a target attribute, an attribute which has to be predicted. Then you have your numerical attribute, computed from tf-idf. You use both of them to compute InfoGain or any entropy related value. Not only the numerical value. Sort by numerical, consider each split value on numerical, then do the counts on target attribute for each split value. This is explained in my answer $\endgroup$
    – rapaio
    Jun 20, 2014 at 11:09
  • $\begingroup$ This is the last try. The formula you use works if A is categorical/nominal variable. Thus a in A, means all values for variable A. If A is numeric, like it is your case, than you split A in two groups, one which have values less than a threshold, and the other group. Then you compute entropies for those groups: A_left and A_right for each possible threshold. Then your IG = E(C) - E(C,A_left) - E(C, A_right) $\endgroup$
    – rapaio
    Jun 20, 2014 at 11:47

1 Answer 1

Reset to default
0
$\begingroup$

Yes, you want to descretize your data. In fact it's generally good practice to do this in Machine Learning as it opens things up for more general algorithms.

The way to do it is an optimization problem, I guess you can think of it as a clustering problem too, you want to choose a bucketing of your real values such that the resulting categorical variables maximize the pairwise distance between the variables while ensuring the internal distance is minimal.

Now smoothing for the probability estimation will be very important here otherwise each bucket may end up with a tiny number of data points. Make sure you smooth aggressively towards 1/J for buckets which have a small number of examples in them. How to do this is unfortunately somewhat underdocumented in the literature and only Laplacian Smoothing (which is terrible, and has no justification at all) appears to be well known.

As a rule of thumb / rough hack, you could insist each bucket has some minimum number of data points, or you could use something like a p value.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Is smoothing really necessary? I mean, i got weights saved as real values. So I may create weights for interval. If weight of tf-idf is less than 0.1, then it has value 1, if less than 0.2 - value 2. It is more like rounding up than discretization. Will it work or should i just go back to Boolean model? $\endgroup$
    – kam
    Jun 20, 2014 at 13:34
  • $\begingroup$ Smoothing is not necessary as long as you have a large number of examples for each bucket. $\endgroup$
    – samthebest
    Jun 20, 2014 at 14:44
  • $\begingroup$ See stats.stackexchange.com/questions/104402/… and many other posts here $\endgroup$
    – kjetil b halvorsen ♦
    Jun 18, 2022 at 0:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.