4
$\begingroup$

Let $X_i\sim B(\pi_i), \text{for }i=1,2,\cdots,n$. I have two models and I want to compare which of them forecast better.

Model 1: Estimates the parameters with maximum likelihood.

Model 2: Estimates the parameters with Bayes.

I use the Brier score and the Logarithmic scoring rule for comparison. The results are:

>> Model 1: 0.2505 (Brier), 0.6350 (minus log-score)
>> Model 2: 0.2544 (Brier), 0.6028 (minus log-score)

The smaller the score, the better the model. So, according to Brier Score Model 1 is better, and according to log-score Model 2 is better.

I would like to ask, why there is this difference. Also, is there a paper for learning how to compare the forecasting ability of a frequentist model with a Bayesian one?

|cite|improve this question|||||
$\endgroup$
6
$\begingroup$

Without getting into the Bayesian vs. frequentist part of your question, the two proper accuracy scores are rewarding different things and it's not surprising they behave differently. The logarithmic score rewards more extreme predictions that are in the right direction. This score can be ruined by a single prediction of probability of 0 or 1 that is in the wrong direction, due to taking the log of zero. The logarithmic rule is a rescaling of the gold standard optimization criteria (in the absence of other knowledge that Bayesians would use in the prior distribution) the log likelihood so in a sense it is the best accuracy score to use for binary $Y$.

|cite|improve this answer|||||
$\endgroup$
  • $\begingroup$ Since we don't know which specification is correct we cannot say which scoring rule is better. It makes much more sense to use the domain loss for comparison since the correct specification is never known. I know this is against your motto of separating decision takers from modelers. However having a specific loss in mind during modeling has advantages in terms of getting better results for that particular domain loss in practice. $\endgroup$ – Cagdas Ozgenc Jan 31 at 7:50
  • $\begingroup$ What is domain loss? And can you prove that it is a proper accuracy score? Otherwise beware. $\endgroup$ – Frank Harrell Feb 2 at 12:30
  • $\begingroup$ The argument stems from the fact that what you call proper scoring is actually strictly proper scoring. 0-1 classification is proper, but not strictly proper for example. Potato, potato, I think what you are trying to say is clear from context. $\endgroup$ – Cagdas Ozgenc Feb 8 at 13:46
  • $\begingroup$ Good to know about this. Since classification accuracy can lead to selection of the wrong features and giving them the wrong weights, if the score is still 'proper' I wonder if 'proper' is expansive enough. Or we need another word to describe harm of discontinuous scoring rules. $\endgroup$ – Frank Harrell Feb 11 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.