10
$\begingroup$

I guess my question is related to these ones: Choosing among proper scoring rules, The performance metric used in prediction is different from the objective function to train the model, but I'm still puzzled...

I've been taught that the proper way to perform regression is to find the parameters by maximum likelihood. This holds for linear regression as well as for probabilistic, e.g. logistic regression. Why, then, would I want to quantify the model's performance by anything else but the likelihood (or some monotonic transformation of it)? What would be the justification for using Brier score, spherical score, or whatever else?

This answer explains:

the choice of scoring rule comes down to how much weight you want to place on different portions of the probability scale or, equivalently, at different relative false-positive and false-negative costs.

I don't understand it. From what I've seen, Brier score is symmetric regarding false positives to false negatives. But, even if the explanation above were true and I really wanted to weight 'different portions of the probability scale' differently, shouldn't I then minimise this score in the first place, instead of maximising log-likelihood?

So, the way I see it, I can either

  • perform regression using max-likelihood and then stick to likelihood for evaluation; or
  • decide on the scoring rule based on my real-world application and then find the regression parameters by optimising that score, which I would later also use for evaluating the model.

(Note that in the latter case, if I'd choose Brier score, this would boil down to least squares optimisation. But least squares is actually max-likelihood in presence of Gaussian noise, which cannot be true for binary data...)

What am I missing?

$\endgroup$
4

2 Answers 2

10
$\begingroup$

One place where this is done all the time is when the loss function includes a penalty term. We train the model according to some function that has the penalty term, but then we evaluate without that penalty term. Ridge regression, for instance, does this.

For instance, the loss function for ridge regression is $L(y,\hat y\vert\lambda) = \sum\left(y_i - \hat y_i\right)^2 + \lambda\vert\vert\hat\beta\vert\vert_2 $, but the performance is typically evaluated on just square loss, $\sum\left(y_i - \hat y_i\right)^2$.

$\endgroup$
5
  • $\begingroup$ +1, and funnily enough, we include regularization in our models, which I describe in my answer... $\endgroup$ Commented Sep 6, 2023 at 18:51
  • $\begingroup$ Interesting point, thanks. But, I wonder, where is it relevant? Once the model is fitted, the regularisation term is just a constant. I could include it in the evaluation without it affecting e.g. the ranking (assuming I want to compare different models). $\endgroup$
    – Igor F.
    Commented Sep 6, 2023 at 18:58
  • 2
    $\begingroup$ @IgorF. You might not always be comparing models with the same $\lambda$ term. $\endgroup$
    – Dave
    Commented Sep 6, 2023 at 18:59
  • $\begingroup$ To my understanding, ridge regression optimises two objectives at once: the model of the data and the magnitude of the coefficients. The evaluation function you quote properly quantifies one of the two objectives; it's a part of the total objective. I can live with that. But Brier score and log-loss seem totally independent. $\endgroup$
    – Igor F.
    Commented Sep 7, 2023 at 9:15
  • $\begingroup$ @IgorF. Yes, I can see the ridge loss function as more similar to MSE than log loss is to Broer score. $\endgroup$
    – Dave
    Commented Sep 7, 2023 at 10:34
8
$\begingroup$

In general, I absolutely agree with your point, and usually recommend that the same loss function should be used both in training and in evaluation.

However, there are situations where you might want to evaluate on a different metric than the one used for estimating the model. (Of course, then you would use a point prediction appropriate for your evaluation metric.) For instance, you may need to estimate a single model from which you would subsequently extract different functionals to support different decisions.

As an example, I do forecasting for retail. We are constrained to using a single model from which we extract conditional expectations for promotion planning (you want to run the promotion with the highest expected return), but also quantile forecasts for different cumulative time periods for replenishment decisions. We can then evaluate the expectation forecast using the mean squared error, and the quantile forecasts using a pinball loss, all after having fitted the model using maximum likelihood. Different consumers of our forecasts will care about different forecasts, and therefore about different loss functions.

$\endgroup$
5
  • 2
    $\begingroup$ So it's about an external constraint, right? But ideally, if you could, you would train two different models, each with the loss function optimal for its purpose, and evaluate them using the same loss function? $\endgroup$
    – Igor F.
    Commented Sep 7, 2023 at 9:20
  • 2
    $\begingroup$ @IgorF., are you aware that the best estimator of the median (which corresponds to absolute evaluation loss) is the sample mean (which corresponds to square training loss) when the random variable of interest is known to be normally distributed? Thus, under normality it is optimal to use square loss for training when absolute loss is used for evaluation. This is a counterexample to your "ideal" case. (For terminology, you may check the post I have linked to under your question.) $\endgroup$ Commented Sep 7, 2023 at 15:31
  • $\begingroup$ @RichardHardy Thanks, that's interesting. But isn't it a consequence of median being identical to mean for normal distribution? And still, I cannot intuitively grasp it... $\endgroup$
    – Igor F.
    Commented Sep 20, 2023 at 11:38
  • 1
    $\begingroup$ @IgorF., one thing that could help your intuition is that using an efficient estimator is better than using an inefficient one, and efficiency is determined by the training loss function. This holds even if the evaluation loss function is different from the training loss function. An efficient estimator will tend to produce estimates that are closer to the target. Then you can measure closeness by different evaluation loss functions, but they will tend to rank efficient vs. nonefficient estimators the same (as long as the evaluation loss functions are "sensible" enough). $\endgroup$ Commented Sep 20, 2023 at 11:51
  • 1
    $\begingroup$ @IgorF.: sorry for the late follow-up... here is the problem with training multiple models: when we train the models, we don't know yet which ones we need, because subsequent order optimization (truck fillup, spoilage calculations) may require cumulative quantile forecasts over one day, or over two days, or over ten or twenty. All this for an absolutely enormous number of product-location combinations. Software engineering and cost constrain what we would like to do statistically. $\endgroup$ Commented May 16 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.