0
$\begingroup$

I am using doing a binary classification to classify things 0 or 1 using a set of features with LightGBM and XGBoost. Both models give AUC scores roughly in the 0.85s, which seems good. But the $RMSE$ is around 0.32, which is too high, and a negative $R^2$ score of -0.35 on test data which means the features I'm using are terrible at predicting the label.

I think I don't really understand if $RMSE$/$R^2$ is appropriate for binary classifications. Should I just stick with the AUC score or should I be wary of what $RMSE$/$R^2$ says about the model?

$\endgroup$
  • $\begingroup$ If you are the same person who posted the duplicate, then please visit stats.stackexchange.com/help/merging-accounts to merge your accounts. $\endgroup$ – whuber Nov 1 '19 at 18:28
  • $\begingroup$ I can't merge because I initially asked a question as a guest. $\endgroup$ – pairwiseseq Nov 1 '19 at 18:47
  • $\begingroup$ I believe you can: that's a regular account and it's associated with the same e-mail address. $\endgroup$ – whuber Nov 1 '19 at 18:55
  • $\begingroup$ @whuber sorry, I should've mentioned that I did not use the same email address. $\endgroup$ – pairwiseseq Nov 1 '19 at 20:35
  • $\begingroup$ I deleted the old post for you. $\endgroup$ – whuber Nov 1 '19 at 20:47
1
$\begingroup$

I think AUC is more acceptable for binary classifiers. I personally prefer Gini, which is simply just a restatement of the AUC. Gini goes between 0 and 1, whereas AUC goes between 0.5 and 1. RMSE is more acceptable when the target variable is continuous. For example, if you were validating a linear model in-sample through k - fold cross validation, RMSE would be a more suitable metric to assess model performance.

Think about it like this. Since you're constructing a $\textbf{binary}$ classifier, you're interested in how well you can separate two groups; the group of 0's and the group of 1's. AUC and Gini measure how well you can separate these two groups. So to me at least, it seems more appropriate to use AUC and Gini.

$\endgroup$
0
$\begingroup$

The ROC-AUC has a number of nice statistical properties and is a good metric for binary outcomes. This is what I use most of the time unless there is a huge class imbalance in which case it is the PR curve.

I think you have probabilistic outcomes so people use the Brier score or log likelihood for assessing performance as well. Frank Harrell prefers these approaches because they don’t dichotomise the models output. IE a probabilistic model isn’t the same as a classifier like KNN.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.