# AUC or $R^2$/RMSE for binary classification

I am using doing a binary classification to classify things 0 or 1 using a set of features with LightGBM and XGBoost. Both models give AUC scores roughly in the 0.85s, which seems good. But the $$RMSE$$ is around 0.32, which is too high, and a negative $$R^2$$ score of -0.35 on test data which means the features I'm using are terrible at predicting the label.

I think I don't really understand if $$RMSE$$/$$R^2$$ is appropriate for binary classifications. Should I just stick with the AUC score or should I be wary of what $$RMSE$$/$$R^2$$ says about the model?

• If you are the same person who posted the duplicate, then please visit stats.stackexchange.com/help/merging-accounts to merge your accounts.
– whuber
Nov 1 '19 at 18:28
• I can't merge because I initially asked a question as a guest. Nov 1 '19 at 18:47
• I believe you can: that's a regular account and it's associated with the same e-mail address.
– whuber
Nov 1 '19 at 18:55
• @whuber sorry, I should've mentioned that I did not use the same email address. Nov 1 '19 at 20:35
• I deleted the old post for you.
– whuber
Nov 1 '19 at 20:47

Think about it like this. Since you're constructing a $$\textbf{binary}$$ classifier, you're interested in how well you can separate two groups; the group of 0's and the group of 1's. AUC and Gini measure how well you can separate these two groups. So to me at least, it seems more appropriate to use AUC and Gini.