0
$\begingroup$

The Heidke skill score is a popular measure for quantifying forecasting skill. It follows the general definition of a skill score (SS):

$$SS = \frac{l_m-l_r}{l_p-l_r},$$

with $l_p$ the loss of a perfect model (usually zero), $l_m$ the model loss and $l_r$ the loss of some reference model. Given the following convention for contingency tables:

enter image description here

taken from Appelman 1960, $l_m = \frac{b+c}{T}$ and $l_r = \frac{(a+c)(a+b) + (b+d)(c+d)}{T^2}$. The reference model is thus a random guessing model with the same success probability as the model predictions. This leads to a Heidke skill score of $HSS = \frac{2(ad-bc)}{(a+b)(b+d) +(a+c)(c+d)}$. This choice lets the reference model depend on the prediction model.

As an alternative, Appelman 1960 proposed a deterministic reference model, predicting "Yes" if $X>Y$ and "No" otherwise. This leads to a skill score of $\frac{d-b}{c+d}$.

My question is on a compromise between these two, i.e. a random guessing model with success probability $X/T$, so based on the observations only, without being as skewed as the choice for Appleman skill score. This reference model has as loss $l_r = 2\frac{(b+d)(a+c)}{T^2}$, with skill score $\frac{(a+c)(d-c)+(b+d)(a-b)}{2(b+d)(a+c)}$. I cannot find this skill score in the literature, has it ever been used or named? And why not?

$\endgroup$
4
  • $\begingroup$ Uh gah. So many issues. For one, this is very close to a scaled version of accuracy, which has major problems. Another point: the first reference model the alternative both implicitly presume that all $T$ instances we classify are interchangeable: if they differ, then it makes little sense to add numbers of classified instances. With evaluation metrics like these, I always have an impulse to simulate how they behave in given situations; usually you find some kind of rather unexpected behavior. $\endgroup$ Commented Nov 6 at 15:47
  • $\begingroup$ Are you sure you want to use "hard" classifications, and not change to probabilistic predictions and evaluate these using proper scoring rules? $\endgroup$ Commented Nov 6 at 15:48
  • $\begingroup$ 'Are you sure you want to use "hard" classifications?'. Yes, we statisticians often prefer looking at probabilities, but the rest of the world likes discrete choices $\endgroup$
    – Knarpie
    Commented Nov 7 at 9:16
  • $\begingroup$ Regarding the first comment: I can agree that there are problems, but Heidke skill score is very popular. My question is more: why that one and not the alternative I propose $\endgroup$
    – Knarpie
    Commented Nov 7 at 9:18

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.