I am using doing a binary classification to classify things 0 or 1 using a set of features with LightGBM and XGBoost. Both models give AUC scores roughly in the 0.85s, which seems good. But the $RMSE$ is around 0.32, which is too high, and a negative $R^2$ score of -0.35 on test data which means the features I'm using are terrible at predicting the label.

I think I don't really understand if $RMSE$/$R^2$ is appropriate for binary classifications. Should I just stick with the AUC score or should I be wary of what $RMSE$/$R^2$ says about the model?

  • $\begingroup$ If you are the same person who posted the duplicate, then please visit stats.stackexchange.com/help/merging-accounts to merge your accounts. $\endgroup$
    – whuber
    Nov 1 '19 at 18:28
  • $\begingroup$ I can't merge because I initially asked a question as a guest. $\endgroup$ Nov 1 '19 at 18:47
  • $\begingroup$ I believe you can: that's a regular account and it's associated with the same e-mail address. $\endgroup$
    – whuber
    Nov 1 '19 at 18:55
  • $\begingroup$ @whuber sorry, I should've mentioned that I did not use the same email address. $\endgroup$ Nov 1 '19 at 20:35
  • $\begingroup$ I deleted the old post for you. $\endgroup$
    – whuber
    Nov 1 '19 at 20:47

I think AUC is more acceptable for binary classifiers. I personally prefer Gini, which is simply just a restatement of the AUC. Gini goes between 0 and 1, whereas AUC goes between 0.5 and 1. RMSE is more acceptable when the target variable is continuous. For example, if you were validating a linear model in-sample through k - fold cross validation, RMSE would be a more suitable metric to assess model performance.

Think about it like this. Since you're constructing a $\textbf{binary}$ classifier, you're interested in how well you can separate two groups; the group of 0's and the group of 1's. AUC and Gini measure how well you can separate these two groups. So to me at least, it seems more appropriate to use AUC and Gini.

  • $\begingroup$ Gini index is often (at least, in decision-tree methods) only evaluated on training data. It measures the deviation between predicted probabilities and the observed response. If one would want to evaluate it on test data, MSE on predicted probabilities (a.k.a. Brier score) would be a good way to quantify, it also measures the deviation between predicted probabilities and observed response. Both Gini and MSE are based on the variance of a binomial variable. $\endgroup$ Feb 28 at 10:18
  • $\begingroup$ The Gini index can be evaluated on either training or testing data. It doesn't have to be exclusively used on the training data. $\endgroup$
    – ralph
    Feb 28 at 23:17
  • $\begingroup$ Yes, indeed. I am curious on how it would be computed. B/c going by this definition: en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity, all class labels could have switched between training and test data, and the impurity would remain exactly the same. This seems quite ambiguous w.r.t. the quality of the predictions. Perhaps a different computation would be used? I'm not sure, would like to know! $\endgroup$ Mar 1 at 10:30

The ROC-AUC has a number of nice statistical properties and is a good metric for binary outcomes. This is what I use most of the time unless there is a huge class imbalance in which case it is the PR curve.

I think you have probabilistic outcomes so people use the Brier score or log likelihood for assessing performance as well. Frank Harrell prefers these approaches because they don’t dichotomise the models output. IE a probabilistic model isn’t the same as a classifier like KNN.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.