Why use a scoring rule different from the loss function?

Question

I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still puzzled...

I've been taught that the proper way to perform regression is to find the parameters by maximum likelihood. This holds for linear regression as well as for probabilistic, e.g. logistic regression. Why, then, would I want to quantify the model's performance by anything else but the likelihood (or some monotonous transformation of it)? What would be the justification for using Brier score, spherical score, or whatever else?

This answer explains:

the choice of scoring rule comes down to how much weight you want to place on different portions of the probability scale or, equivalently, at different relative false-positive and false-negative costs.

I don't understand it. From what I've seen, Brier score is symmetric regarding false positives to false negatives. But, even if the explanation above were true and I really wanted to weight 'different portions of the probability scale' differently, shouldn't I then minimise this score in the first place, instead of maximising log-likelihood?

So, the way I see it, I can either

• perform regression using max-likelihood and then stick to likelihood for evaluation; or
• decide on the scoring rule based on my real-world application and then find the regression parameters by optimising that score, which I would later also use for evaluating the model.

(Note that in the latter case, if I'd choose Brier score, this would boil down to least squares optimisation. But least squares is actually max-likelihood in presence of Gaussian noise, which cannot be true for binary data...)

What am I missing?

Answer 1

One place where this is done all the time is when the loss function includes a penalty term. We train the model according to some function that has the penalty term, but then we evaluate without that penalty term. Ridge regression, for instance, does this.

For instance, the loss function for ridge regression is $L(y,\hat y\vert\lambda) = \sum\left(y_i - \hat y_i\right)^2 + \lambda\vert\vert\hat\beta\vert\vert_2 $, but the performance is typically evaluated on just square loss, $\sum\left(y_i - \hat y_i\right)^2$.

Answer 2

In general, I absolutely agree with your point, and usually recommend that the same loss function should be used both in training and in evaluation.

However, there are situations where you might want to evaluate on a different metric than the one used for estimating the model. (Of course, then you would use a point prediction appropriate for your evaluation metric.) For instance, you may need to estimate a single model from which you would subsequently extract different functionals to support different decisions.

As an example, I do forecasting for retail. We are constrained to using a single model from which we extract conditional expectations for promotion planning (you want to run the promotion with the highest expected return), but also quantile forecasts for different cumulative time periods for replenishment decisions. We can then evaluate the expectation forecast using the mean squared error, and the quantile forecasts using a pinball loss, all after having fitted the model using maximum likelihood. Different consumers of our forecasts will care about different forecasts, and therefore about different loss functions.