I have been looking for an answer on this website and on Google but I can't seem to find a clear explanation anywhere.
The problem is the following. I built a Random Forest model (using Python's sklearn module) for a binary classification task.
Training, test, everything seems to go well, and I get a relatively high ROC-AUC compared to the previous iteration of my model (I actually calculate the Gini coefficient but use the formule described in this question to convert it to AUC), at around 0.67.
After putting the model in production for a while and that it indeed seems to perform better than the previous one (still with that AUC metric).
At some point I am asked what the R-squared of my model was, that is, as I understand it, the proportion of variance explained. I am a bit puzzled as I usually heard about this metric in a regression task.
"No problem" I think, "I can just estimate the R-squared of my model by checking the labels as 0 and 1 and use the predicted probability as the predicted value". It seems to work relatively well on the training set (R-squared of 0.8), as soon as I try it on a test set it gets really low, even quite often negative !
As far as I understand it, AUC (or Gini) tells me how well I can "separate" my data between each class, and is thus the most important in a classification task, however I am worried a low R-squared tells something about my model. Does it overfit ? Should I instead use the label predicted ?