I am building a decision tree from scratch. I have been using entropy so far (calculated this way):
def calculate_entropy(data):
label_column = data[:,-1]
_,counts = np.unique(label_column,return_counts=True)
probabilities = counts/counts.sum()
entropy = sum(probabilities*(-np.log2(probabilities)))
return entropy
Now I want to implement a function that calculates the Gini index of an array. From here I get the following function that calculates:
def gini_coeff(x):
# requires all values in x to be zero or positive numbers,
# otherwise results are undefined
n = len(x)
s = x.sum()
r = argsort(argsort(-x)) # calculates zero-based ranks
return 1 - (2.0 * (r*x).sum() + s)/(n*s)
That for
x = [1,1,2]
gini_coeff(np.array(x))
0.16666666666666663
But from this other source I see that the Gini impurity can be calculated
$ G = \sum_i^n \cdot p(i) \cdot (1-p(i))$
That I implemented this way
def calculate_gini(data):
label_column = data
_,counts = np.unique(label_column,return_counts=True)
probabilities = counts/counts.sum()
return sum(probabilities*(1-probabilities))
calculate_gini(x)
0.4444444444444444
I believe that I am making here a conceptual problem. What is it? What are the differences? Which one should I use for a decision tree?