AUC or $R^2$/RMSE for binary classification I am using doing a binary classification to classify things 0 or 1 using a set of features with LightGBM and XGBoost. Both models give AUC scores roughly in the 0.85s, which seems good. But the $RMSE$ is around 0.32, which is too high, and a negative $R^2$ score of -0.35 on test data which means the features I'm using are terrible at predicting the label.
I think I don't really understand if $RMSE$/$R^2$ is appropriate for binary classifications. Should I just stick with the AUC score or should I be wary of what $RMSE$/$R^2$ says about the model?
 A: I think AUC is more acceptable for binary classifiers. I personally prefer Gini, which is simply just a restatement of the AUC. Gini goes between 0 and 1, whereas AUC goes between 0.5 and 1. RMSE is more acceptable when the target variable is continuous. For example, if you were validating a linear model in-sample through k - fold cross validation, RMSE would be a more suitable metric to assess model performance. 
Think about it like this. Since you're constructing a $\textbf{binary}$ classifier, you're interested in how well you can separate two groups; the group of 0's and the group of 1's. AUC and Gini measure how well you can separate these two groups. So to me at least, it seems more appropriate to use AUC and Gini. 
A: The ROC-AUC has a number of nice statistical properties and is a good metric for binary outcomes. This is what I use most of the time unless there is a huge class imbalance in which case it is the PR curve.
I think you have probabilistic outcomes so people use the Brier score or log likelihood for assessing performance as well. Frank Harrell prefers these approaches because they don’t dichotomise the models output. IE a probabilistic model isn’t the same as a classifier like KNN.
A: $R^2$ only has the usual "proportion of variability explained" in the case of a linear model (and not even all linear models). Further, $R^2$ and $MSE$ give the same information (perhaps depending on how you calculate out-of-sample $R^2$), and $MSE$ and $RMSE$ (obviously) give the same information. Thus, the discussion becomes one of $MSE$ vs $AUC$.
$MSE$ is equivalent to something called Brier score, which is a strictly proper scoring rule. $AUC$ is not a strictly proper scoring rule. Thus, we would prefer the model with the better (lower) Brier score.
