# Model evaluation metrics for comparing predicted probability accuracy across different datasets?

I'm working on an online model scoring framework, my goal is to be able to understand if my model's predictive performance is degrading week-over-week. I have a classification model (trained on binary target data) which produces probabilities, and I am most interested in assessing the accuracy of those probabilities. For model selection and tuning, i'm using log loss, but I understand that log loss can't be used to compare performance across different datasets (different weeks of predictions). Is there a best practice metric that I can use to effectively compare the accuracy of predicted probabilities, across different datasets? More explicitly, what are appropriate metrics to monitor predicted probability quality over time?

• but I understand that log loss can't be used to compare performance across different datasets Welcome to Cross Validated! Could you please expand on this?
– Dave
Aug 10 at 19:01
• My understanding is that different datasets can have different class distributions, varying complexity, and shifts in feature distributions between datasets can make comparing log loss across different datasets (or for in my scenario, comparing log loss against predictions made over time) an unreliable measure of performance.
– Ted
Aug 11 at 18:14

I agree with your statement that comparing log losses $$L(p_i) = - \log (p_i)$$ across different datasets is not very meaningful if these datasets exhibit different statistical behaviour. See also this post on 'good' log loss values for more background.

## Looking at differences

I would suggest to modify the metric you are already using, so that it's interpretation becomes (more) meaningful across datasets. The easiest way would be to switch to loss differences instead of pure losses. If $$\bar L$$ is the average loss of your model and $$\bar L_{base}$$ is the average loss of a simple baseline forecast (e.g. frequency of success in last $$n$$ weeks), then $$\bar L_{base} - \bar L$$ should be positive most of the time. If you start to see negative values for several weeks, this suggests that your model is no longer useful. Keep in mind that you'd have to check whether the observed sign changes are significant or just bad luck for your model.

## Forecast Skill

To refine this idea, you could also scale the average loss difference by the average loss of the baseline model (if it is positive, which it should be for the log loss), i.e. via $$\bar L_{skill} := \frac{\bar L_{base} - \bar L}{\bar L_{base}} = 1 - \frac{ \bar L}{\bar L_{base}}$$ Then $$\bar L_{skill}$$ is bounded above by 1 (if your model were perfect) and values below 0 again indicate that your model is no longer useful. Additionally, it quantifies how big the loss difference is compared to the baseline loss and thus gives you a useful interpretation across datasets.

For proper scoring rules (of which the log loss is an example) this quantity is also called skill score, see for instance this paper. In Section 2.3 the authors also state that

If scores [meaning losses in this discussion] for distinct sets of situations are compared, then considerable care must be exercised to separate the confounding effects of intrinsic predictability and predictive performance

which summarizes the reasons why it is probably ill advised to compare losses across different datasets.

## Connection to coefficients of determination such as $$R^2$$

As noted by Dave in the comments, forecast skill relate to McFadden's pseudo $$R^2$$ and might agree with it, depending on the baseline model. Additionally, if $$L$$ is the squared error loss and the baseline model is a pure intercept model (i.e. just the mean of the observations), then the in-sample computation of the skill of a model agrees with the usual $$R^2$$ coefficient of determination from ordinary least squares regression. The same connection arises for the case where $$L$$ is the absolute loss and the coefficient of determination $$R^1$$ which is sometimes used in quantile regression.

• Note that this skill score is related to McFadden’s pseudo $R^2$ and equal to McFadden, depending on the baseline model.
– Dave
Aug 14 at 18:43
• Thank you so much! I have a follow up question - would it be meaningful to use something like brier score or absolute loss (or maybe even log loss) to measure model performance over time (not direct comparison)? For example, if absolute loss is increasing week over week, after a few weeks can you make the deduction that there is some underlying drift in the data you're predicting on vs the data your model was trained on and therefore the model needs to be retrained?
– Ted
Aug 15 at 19:38
• @Ted I suppose you can do this, but why use a score or loss for this job? If you are interested in whether your new data are similar to your training data, then comparing simple means, quantiles or doing something like the Kolmogorov–Smirnov test seem more straightforward. Aug 15 at 19:56
• I will be using those types of metrics to assess drift, but I guess i'm still looking for something to access "if the model is performing well".
– Ted
Aug 15 at 20:04
• picky_porpoise The double negative in the last sentence makes it quite difficult to understand what you mean. Do you mean that everyone sees the usual $R^2$ as having an objective interpretation of the well/poorly spectrum? // @Ted It’s really hard to make a context-independent judgment about model quality.
– Dave
Sep 2 at 16:38