2
$\begingroup$

According to the API Reference, XGBRegressor().score() returns R2. However, according to the XGBoost Paramters page, the default eval_metric for regression is RMSE.

In my understanding, scoring and using an evaluation metric is the same. So what am I missing here? Does XGBoost use different defaults for its native API and the Scikit-Learn API? Or do these two options mean something different?

Thanks a lot!

$\endgroup$
1
  • $\begingroup$ Is there any reason at all why XGBoost could not choose a different default metric than sklearn? $\endgroup$
    – Tim
    Jun 28 at 22:04

2 Answers 2

2
$\begingroup$

This is a bit strange, but $R^2$ and $RMSE$ are just functions of each other, so they are, in some sense, conveying the same information.

$$ R^2=1-\dfrac{ n\big(RMSE\big)^2 }{ \sum_{i=1}^n\big( y_i-\bar y \big)^2 } $$

Some people see $R^2$ as easier to interpret. I disagree with the idea that we can liken $R^2$ to grades in school where $0.9$ is an $\text{A}$ that makes us happy and $0.5$ is an $\text{F}$ that makes us sad. Depending on the context, $R^2=0.5$ might be considered spectacular, while another situation might mean that $R^2=0.9$ is rather pedestrian performance.

$\endgroup$
1
$\begingroup$

You said:

scoring and using an evaluation metric is the same

But I guess I would question this idea and suggest that, at least in a typical XGBoost workflow, the concepts behind 'scoring' and 'evaluation metric' are slightly different. But only slightly.

The evaluation metric is used to monitor the training process and watch for overtraining (for example). I think it does make sense for this to be something that reflects the magnitude of the error or bias, and RMSE is a good choice because it has the same units as the thing you're trying to predict.

On the other hand, scoring is something we do after the model is fit (it's still an evaluation metric, but in the XGBoost API, I think this is the intended role for 'scoring'). We might use the score to compare between different hyperparameter choices, or between algorithms. Personally I like RMSE for scoring too, but I know lots of people like $R^2$ and perhaps expect it. Either way, I think makes sense to use a variety of metrics in most cases.

Remember not to get fixated on scores (see Anscombe's quartet) and to check the distribution of your residuals! Most people seem to forget this bit...

$\endgroup$
3
  • $\begingroup$ I thought so too at first (one is training metric, the other validation metric). But the documentation says something different. The section "Learning Task Parameters" explains that the task and the metric to optimize are specified like this: objective = 'reg:squarederror'. And eval_metric is "Evaluation metrics for validation data". $\endgroup$
    – dipetkov
    Jun 29 at 16:16
  • $\begingroup$ Personally, I think this is as deep as two developers having their own preferred choice of validation metric. $\endgroup$
    – dipetkov
    Jun 29 at 16:18
  • $\begingroup$ @dipetkov Sorry, I wasn't trying to say that one of those thing is being used as the cost function during training, only that it's used for in-training validation ('how is training going and am I overfitting?'). As opposed to post-training evaluation ('how did I do?'). It's a pretty subtle distinction, and I agree that there's likely just some personal preference going into the choices of defaults for these things. $\endgroup$
    – kwinkunks
    Jun 29 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.