# Information gain with numerical data

I'm making a random forest classifier. In every tutorial, there is a very simple example of how to calculate entropy with Boolean attributes.
In my problem I have attribute values that are calculated by tf-idf schema, and values are real numbers.
Is there some clever way of applying an information gain function so it will calculate IG with real-number weights, or should I use discretization like:

0 = 0
(-0 - 0.1> = 1
(-0.1 - 0.2> = 2


etc.

EDIT
I have function:

$$IG(X) = E(C) - E(C,A)$$

$$E(C) = \sum\limits_{i=1}^C-P(c_i) * log(P(c_i))$$

and
$$E(C,A) = \sum\limits_{a\in A}P(a) * E(a)$$

The problem is iI have infinite number of possible values of $$A$$ and i think, that I should perform dicretization of these values, shouldn't I?

• I think this is a duplicate question. There are plenty of questions like that on CV. See this question here: stats.stackexchange.com/questions/95839/… – rapaio Jun 20 '14 at 10:29
• My question is how can i use real-number attributes for calculating Information Gain. Normally, my attribute would have known number of possible values, here, i have a real numbers, and probably, ther won't be two documents with attribte a1 with same values. – kam Jun 20 '14 at 11:01
• My answer from the posted question explains how can you compute InfoGain for numerical attributes, like yours. – rapaio Jun 20 '14 at 11:03
• Ok, so if you have a classifier, you must have a target attribute, an attribute which has to be predicted. Then you have your numerical attribute, computed from tf-idf. You use both of them to compute InfoGain or any entropy related value. Not only the numerical value. Sort by numerical, consider each split value on numerical, then do the counts on target attribute for each split value. This is explained in my answer – rapaio Jun 20 '14 at 11:09
• This is the last try. The formula you use works if A is categorical/nominal variable. Thus a in A, means all values for variable A. If A is numeric, like it is your case, than you split A in two groups, one which have values less than a threshold, and the other group. Then you compute entropies for those groups: A_left and A_right for each possible threshold. Then your IG = E(C) - E(C,A_left) - E(C, A_right) – rapaio Jun 20 '14 at 11:47