XGBRegressor score (R2) vs. eval_metric (RMSE) According to the API Reference, XGBRegressor().score() returns R2.
However, according to the XGBoost Paramters page, the default eval_metric for regression is RMSE.
In my understanding, scoring and using an evaluation metric is the same. So what am I missing here? Does XGBoost use different defaults for its native API and the Scikit-Learn API? Or do these two options mean something different?
Thanks a lot!
 A: This is a bit strange, but $R^2$ and $RMSE$ are just functions of each other, so they are, in some sense, conveying the same information.
$$
R^2=1-\dfrac{
n\big(RMSE\big)^2
}{
\sum_{i=1}^n\big(
y_i-\bar y
\big)^2
}
$$
Some people see $R^2$ as easier to interpret. I disagree with the idea that we can liken $R^2$ to grades in school where $0.9$ is an $\text{A}$ that makes us happy and $0.5$ is an $\text{F}$ that makes us sad. Depending on the context, $R^2=0.5$ might be considered spectacular, while another situation might mean that $R^2=0.9$ is rather pedestrian performance.
A: You said:

scoring and using an evaluation metric is the same

But I guess I would question this idea and suggest that, at least in a typical XGBoost workflow, the concepts behind 'scoring' and 'evaluation metric' are slightly different. But only slightly.
The evaluation metric is used to monitor the training process and watch for overtraining (for example). I think it does make sense for this to be something that reflects the magnitude of the error or bias, and RMSE is a good choice because it has the same units as the thing you're trying to predict.
On the other hand, scoring is something we do after the model is fit (it's still an evaluation metric, but in the XGBoost API, I think this is the intended role for 'scoring'). We might use the score to compare between different hyperparameter choices, or between algorithms. Personally I like RMSE for scoring too, but I know lots of people like $R^2$ and perhaps expect it. Either way, I think makes sense to use a variety of metrics in most cases.
Remember not to get fixated on scores (see Anscombe's quartet) and to check the distribution of your residuals! Most people seem to forget this bit...
