Kaggle's competition Porto Seguro's Safe Driver Prediction uses Normalized Gini Score as evaluation metric and this got me curious about the reasons for this choice. What are the advantages of using normalized gini score instead of the most usual metrics, like AUC, for evaluation?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ The Kaggle website used to have this answer: "There is a maximum achievable area for a "perfect" model since not all of the positive examples occur immediately. We use the normalized Gini coefficient by dividing the Gini coefficient of your model by the Gini coefficient of the perfect model." but it is not available anymore. webcache.googleusercontent.com/… $\endgroup$– Sextus EmpiricusCommented Oct 10, 2017 at 1:01
-
1$\begingroup$ So, gini is just auc on a different scale. Or are auc and gini applied to different curves? That is not clear to me as non-expert in machine-learning. The question is not very clear about this. $\endgroup$– Sextus EmpiricusCommented Oct 10, 2017 at 1:10
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
4
I believe that the Gini score is merely a reformulation of the AUC: $$gini = 2 \times AUC -1$$ As for why use this instead of the commonly used AUC, the only reason I can think of is that a random prediction will yield a Gini score of 0 as opposed to the AUC which will be 0.5.
-
7$\begingroup$ Besides that using the gini coefficient sets the performance of a random classifier to a score of 0... the normalization "improves" the other end of the scale and makes that the score of a perfect classifier is equal to 1 rather than a maximum achievable AUC<1. The improvement being only relative depending on whether you think a more intuitive scale is good or not. Although beyond this easier interpretation you might argue that it (the normalization) also improves generalization and comparison of different data-sets. $\endgroup$ Commented Oct 10, 2017 at 21:16
-
$\begingroup$ Why should macimum achievable AUC be less then 1, also I don't see how is gini setting it to 1? $\endgroup$– rep_hoCommented Oct 11, 2017 at 14:21
-
$\begingroup$ It depends on what kind of curve they calculate the gini coefficient. Maybe they use something different than a ROC curve (whose max AUC would indeed be 1). Given the words on the kaggle website it seems plausible that the max AUC is not 1: > " We then move from left to right, asking "In the leftmost x% of the data, how much of the cumulative response have you accumulated?" $\endgroup$ Commented Oct 12, 2017 at 15:44
-
$\begingroup$ example: bayesserver.com/docs/charts/lift-chart $\endgroup$ Commented Oct 12, 2017 at 15:58
$\begingroup$
$\endgroup$
3
The relationship only holds for classification tasks because obviously, you cannot calculate AUC for a continuous target. However, we also use normalized Gini in regression tasks, like predicting insurance losses.
-
$\begingroup$ When you do a grid search is the scoring metric still roc-auc then? $\endgroup$– mathellaCommented Nov 10, 2020 at 8:23
-
$\begingroup$ If it's a classification model, then yes. If its regression, AUC is not defined, so you must use Gini. Gini is usually not a defined metric in most machine learning packages, so you'll have to create your own code. $\endgroup$ Commented Nov 11, 2020 at 18:58
-
1$\begingroup$ It may not be terribly meaningful to refer to the tradeoff between sensitivity & specificity with a continuous outcome variable, but the AUC is equal to the concordance probability, which can certainly be calculated for a model predicting a continuous target (see: How to calculate Area Under the Curve (AUC), or the c-statistic, by hand). $\endgroup$ Commented Jan 8, 2021 at 18:35