I am using doing a binary classification to classify things 0 or 1 using a set of features with LightGBM and XGBoost. Both models give
AUC scores roughly in the
0.85s, which seems good. But the $RMSE$ is around 0.32, which is too high, and a negative $R^2$ score of
-0.35 on test data which means the features I'm using are terrible at predicting the label.
I think I don't really understand if $RMSE$/$R^2$ is appropriate for binary classifications. Should I just stick with the
AUC score or should I be wary of what $RMSE$/$R^2$ says about the model?